CN110633296A - Data query method, device, medium and electronic equipment - Google Patents
Data query method, device, medium and electronic equipment Download PDFInfo
- Publication number
- CN110633296A CN110633296A CN201810571915.6A CN201810571915A CN110633296A CN 110633296 A CN110633296 A CN 110633296A CN 201810571915 A CN201810571915 A CN 201810571915A CN 110633296 A CN110633296 A CN 110633296A
- Authority
- CN
- China
- Prior art keywords
- data
- cache region
- target object
- object data
- data query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The embodiment of the invention provides a data query method, a device, a medium and electronic equipment, wherein the data query method comprises the following steps: responding to a data query request, wherein the data query request comprises: the target object acquires the ID of the target object based on the object ID index stored in the first cache region; extracting target object data from a distributed storage system based on the ID of the target object, and storing the target object data in a second cache region; and realizing data query based on the target object data stored in the second cache region. According to the technical scheme of the embodiment of the invention, all object data IDs can be cached by using the memory to form an index, the frequently accessed objects are stored, the query is quickly responded, the memory consumption is reduced, and the quick analysis and processing of the data are realized at lower cost.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data query method, a data query device, a data query medium and electronic equipment.
Background
As internet applications become widely used, the internet application background has and will continue to accumulate large amounts of data. On the one hand, a large number of internet users can publish content such as ratings, questions and answers through internet applications, such as the kyoto App. On the other hand, many internet users query a large amount of data contents through internet applications, and the magnitude of the data contents often reaches the T byte level. How to store and organize the data and respond to the query request more quickly becomes a problem that needs to be concerned by background developers of internet application
The existing technical scheme mainly comprises: (1) storing data concerned by a user by using a memory cache, and providing query; (2) the relational database is used for storing data and providing query, and database index is increased to accelerate query speed; (3) a large-scale structured storage cluster is built by utilizing a distributed data storage technology to provide data storage service.
The prior technical scheme has the following defects: (1) the cost of computer memory is still relatively high, and if all the data is placed in memory, the cost can be quite high. Some data are not frequently inquired, and if the data occupy the memory for a long time, the memory resource is wasted; (2) the ability of relational databases to support high-concurrency queries is limited; (3) the distributed data storage system is not suitable for traditional transaction processing programs or association analysis and does not support complex queries.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present invention and therefore may include information that does not constitute prior art known to a person of ordinary skill in the art.
Disclosure of Invention
Embodiments of the present invention provide a data query method, apparatus, medium, and electronic device, so as to overcome, at least to some extent, the problem that rapid analysis and processing of data cannot be achieved in the related art.
Additional features and advantages of the invention will be set forth in the detailed description which follows, or may be learned by practice of the invention.
According to a first aspect of the embodiments of the present invention, there is provided a data query implementation method, including:
responding to a data query request, wherein the data query request comprises: the target object acquires the ID of the target object based on the object ID index stored in the first cache region;
extracting target object data from a distributed storage system based on the ID of the target object, and storing the target object data in a second cache region;
and realizing data query based on the target object data stored in the second cache region.
In some embodiments of the present invention, before responding to the data query request, the method further includes:
receiving object data, and storing the object data in a relational database and a distributed storage system;
and writing the ID of the object data into a first cache region according to a preset sorting rule to generate an object ID index, wherein the ID of the object data is an integer ID.
In some embodiments of the present invention, the receiving object data and storing the object data in the relational database and the distributed storage system includes:
storing object data to be counted in the relational database, and counting the stored object data;
and storing the full amount of object data obtained after statistics in the distributed storage system.
In some embodiments of the present invention, the implementing data query based on the target object data stored in the second cache region includes:
analyzing the data query request, and judging whether target object data are acquired from the second cache region or not;
when the target object data are determined to be acquired from the second cache region, calling the target object data in the second cache region, and generating and sending a data query result;
and when determining that the target object data is not acquired from the second cache region, inquiring the ID of the target object in the object ID index, acquiring the target object data from the distributed storage system according to the ID of the target object, and generating and sending a data inquiry result.
In some embodiments of the present invention, the storing the target object data in the second cache area includes:
setting expiration time for the second cache region based on access time after the second cache region is accessed;
and clearing the data stored in the second cache region when the second cache region is not accessed again in the expiration time.
In some embodiments of the invention, the method further comprises:
when the object data stored in the second cache region are updated, writing the object ID in the updated data into the first cache region, and updating the object ID index;
storing the updated data into a distributed storage system, and emptying the second cache region;
and after receiving a data query request, extracting updated target object data from the distributed storage system based on the object ID index, and storing the target object data in a second cache region.
In some embodiments of the present invention, the updating the object data stored in the second cache region includes:
adding update data to the object data stored in the second cache region, and/or
Deleting part of the object data stored in the second cache region, and/or
And replacing the object data stored in the second cache region.
According to a second aspect of the embodiments of the present invention, there is provided a data query apparatus including: the system comprises a response module, an extraction module and a data query module; wherein the content of the first and second substances,
a response module, configured to respond to a data query request, where the data query request includes: the target object acquires the ID of the target object based on the object ID index stored in the first cache region;
the extracting module is used for extracting target object data from the distributed storage system based on the ID of the target object and storing the target object data in a second cache region;
and the data query module is used for realizing data query based on the target object data stored in the second cache region.
In some embodiments of the present invention, the data query apparatus further includes:
the writing module is used for receiving the object data and storing the object data in the relational database and the distributed storage system;
and the distribution module is used for writing the ID of the object data into a first cache region according to a preset sorting rule to generate an object ID index, wherein the ID of the object data is an integer ID.
In some embodiments of the present invention, the writing module is specifically configured to:
storing object data to be counted in the relational database, and counting the stored object data;
and storing the full amount of object data obtained after statistics in the distributed storage system.
In some embodiments of the present invention, the data query apparatus further includes:
the judging module is used for analyzing the data query request and judging whether target object data are acquired from the second cache region or not; when the target object data are determined to be acquired from the second cache region, calling the target object data in the second cache region, and generating and sending a data query result; and when determining that the target object data is not acquired from the second cache region, inquiring the ID of the target object in the object ID index, acquiring the target object data from the distributed storage system according to the ID of the target object, and generating and sending a data inquiry result.
In some embodiments of the present invention, the writing module is further configured to:
setting expiration time for the second cache region after the second cache region is accessed for the first time;
and the second cache region is not accessed again within the expiration time, and the data stored in the second cache region is cleared.
In some embodiments of the present invention, the data query apparatus further includes:
the updating module is used for writing the object ID in the updating data into the first cache region and updating the object ID index when the object data stored in the second cache region is updated; storing the updated data into a distributed storage system, and emptying the second cache region; and after receiving a data query request, extracting updated target object data from the distributed storage system based on the object ID index, and storing the target object data in a second cache region.
In some embodiments of the present invention, the update module is configured to:
adding update data to the object data stored in the second cache region, and/or
Deleting part of the object data stored in the second cache region, and/or
And replacing the object data stored in the second cache region.
According to a third aspect of embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing the data query method of the first aspect as in the above embodiments.
According to a fourth aspect of embodiments of the present invention, there is provided an electronic apparatus, including: one or more processors; a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the data query method of the first aspect as in the above embodiments.
The technical scheme provided by the embodiment of the invention has the following beneficial effects:
the embodiment of the invention provides a data query method, a device, a medium and electronic equipment, wherein the data query method comprises the following steps: responding to a data query request, wherein the data query request comprises: the target object acquires the ID of the target object based on the object ID index stored in the first cache region; extracting target object data from a distributed storage system based on the ID of the target object, and storing the target object data in a second cache region; and realizing data query based on the target object data stored in the second cache region. According to the technical scheme of the embodiment of the invention, all object data IDs can be cached by using the memory to form an index, the frequently accessed objects are stored, the query is quickly responded, the memory consumption is reduced, and the quick analysis and processing of the data are realized at lower cost.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:
FIG. 1 schematically illustrates a flow diagram of a data query method according to one embodiment of the invention;
FIG. 2 schematically illustrates a flow diagram for storing object data and assigning IDs according to one embodiment of the invention;
FIG. 3 schematically illustrates a flow diagram for updating data stored in a first cache region and a second cache region according to one embodiment of the invention;
FIG. 4 schematically shows a block diagram of a data querying device according to one embodiment of the invention;
FIG. 5 illustrates a schematic structural diagram of a computer system suitable for use with the electronic device to implement an embodiment of the invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention may be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations or operations have not been shown or described in detail to avoid obscuring aspects of the invention.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
FIG. 1 schematically shows a flow diagram of a data query method according to one embodiment of the invention.
Referring to fig. 1, a first data query implementation method according to an embodiment of the present invention includes the following steps:
step S110, responding to a data query request, where the data query request includes: the target object acquires the ID of the target object based on the object ID index stored in the first cache region;
in one embodiment of the invention, the object may be data of comments, questions, answers to questions, and the like issued by the user, and after receiving and successfully processing the object data issued by the user, the object data is stored in the relational database and the distributed storage system.
In an embodiment of the present invention, as shown in fig. 2, step S110 may further include, before:
step S210, receiving object data, and storing the object data in a relational database and a distributed storage system;
step S220, writing the ID of the object data into a first cache region according to a preset sorting rule, and generating an object ID index, where the ID of the object data is an integer ID.
In an embodiment of the present invention, the structure of the object ID index may be a data table structure, where the data table structure at least includes the number of pages (cachedPages) for storing the object data and the number of object data IDs (objectCountPerPage) included in each page, and in an actual application, the number of pages in the object ID index and the number of object IDs that can be included in each page may be dynamically configured by using a condition of calling the query interface, so as to ensure that the first cache region does not overflow and crash, or that excessive cache resources are allocated, which results in resource waste.
In one embodiment of the present invention, the maximum number of object data IDs that can be stored by the object ID index can be obtained by multiplying cachedPages by object countperpage.
In an embodiment of the present invention, the preset ordering rule may be flexibly adjusted according to actual requirements, for example: the first cache region can be written into by descending order according to the number of praise issued by the user to issue the problem solutions, or written into by ordering according to the time sequence of issuing the problem solutions by the user.
In one embodiment of the present invention, the storing of the object data in the relational database and the distributed storage system in step S210 may be: storing object data to be counted in a relational database, and counting the quantity of the stored object data; and storing the total object data obtained after statistics in a distributed storage system to realize that the object data needing to participate in statistics is stored in a relational database, counting the object data by using the characteristics of the object data, storing the longer plain text data and the total object data in the distributed storage system, and providing a basis for object ID index by using the characteristics of high concurrent read-write operation support and large storage capacity to realize data query.
In an embodiment of the present invention, based on the foregoing scheme, if the amount of object data is relatively small and a sufficient number of cache regions are available, all object data may be stored in the memory cache, and in response to a query instruction, data may be completely queried from the cache regions. If the operations such as statistical calculation and the like are not needed, the data can be stored and the query function can be provided only by using the cache region and the distributed data storage system, and a relational database is omitted, so that the data query process is simpler and more efficient.
In one embodiment of the present invention, the object data ID may be an integer, such as 13579, so that the cache space occupied by the whole object ID index may be reduced.
Step S120, extracting target object data from the distributed storage system based on the ID of the target object, and storing the target object data in a second cache region;
in an embodiment of the present invention, after the query request is received for the first time, an initialization operation is performed on the second cache region, so as to ensure high accuracy of the object data stored in the second cache region.
In an embodiment of the present invention, the initializing the second cache region specifically includes:
when a query request is received for the first time, or the system finds that the second cache region is empty, the object data IDs of cachedPages object countperpage are read from the object ID index in the first cache region, and at most cachedPages object countperpage are read from the distributed storage system according to the object data IDs, and the object data are written into the second cache region for use in the next query.
In an embodiment of the present invention, after the second cache region is accessed, an expiration time is set for the second cache region based on the access time, for example, 48 hours; and the second cache region is not accessed again within the expiration time, the data stored in the second cache region is cleared, and the initialization operation is carried out on the second cache region again.
In one embodiment of the invention, implementing the data query comprises: analyzing the data query request, and judging whether target object data are acquired from the second cache region; when the target object data are determined to be acquired from the second cache region, calling the target object data in the second cache region, and generating and sending a data query result; and when determining that the target object data is not acquired from the second cache region, inquiring the ID of the target object in the object ID index, acquiring the target object data from the distributed storage system according to the ID of the target object, writing the target object data into the second cache region, acquiring the target object data from the second cache region, generating and sending a data inquiry result.
Step 130, implementing data query based on the target object data stored in the second cache region.
In an embodiment of the present invention, after receiving the query request, it may be checked whether the target object to be queried should be obtained from the second cache region; if the target object data is obtained from the second cache region, the second cache region is ensured to be initialized, and the corresponding target object data in the second cache region is directly returned to the caller; and if the data is determined not to be acquired from the second cache region, finding the ID of the corresponding target object from the object ID index, and acquiring the target object data from the distributed storage system according to the ID of the target object and returning the target object data to the caller.
In an embodiment of the present invention, as shown in fig. 3, the data query method according to the embodiment of the present invention may further include updating data stored in the second cache region and the first cache region, and the specific steps are as follows:
step S310, when the object data stored in the second cache region is updated, the object ID in the updated data is written into the first cache region, and the object ID index is updated;
step S320, storing the updated data into the distributed storage system and emptying a second cache region;
step S330, after receiving the data query request, extracting the updated target object data from the distributed storage system based on the object ID index, and storing the target object data in the second cache region.
In an embodiment of the present invention, updating the data stored in the second cache region and the first cache region includes the following three cases: (1) newly added data needs to be added to the second cache region; (2) some object data already in the second cache region needs to be updated; (3) some object data already in the second cache region needs to be deleted. In the data query method of the embodiment of the invention, under the conditions (1) and (3), the object data ID is added to or cleared from the object ID index respectively; in case (2), the object data ID cache may remain unchanged; subsequently, in all three cases, the second cache region is emptied, and when a new query comes, the second cache region is initialized again.
The following describes embodiments of the apparatus of the present invention, which can be used to perform the above-mentioned data query method of the present invention.
Fig. 4 schematically shows a block diagram of a data querying device according to an embodiment of the present invention.
Referring to fig. 4, a data query apparatus 400 according to an embodiment of the present invention includes: a response module 401, an extraction module 402, a data query module 403; wherein the content of the first and second substances,
a generating module 401, configured to respond to a data query request, where the data query request includes: the target object acquires the ID of the target object based on the object ID index stored in the first cache region;
an extracting module 402, configured to extract target object data from the distributed storage system based on an ID of the target object, and store the target object data in a second cache region;
and a data query module 403, configured to perform data query based on the target object data stored in the second cache region.
In some embodiments of the present invention, the data query apparatus further includes:
a write module 404, configured to receive the object data and store the object data in the relational database and the distributed storage system;
an allocating module 405, configured to write the ID of the object data into a first cache region according to a preset sorting rule, and generate an object ID index, where the ID of the object data is an integer ID.
In some embodiments of the present invention, the writing module 404 is specifically configured to:
storing object data to be counted in the relational database, and counting the stored object data;
and storing the full amount of object data obtained after statistics in the distributed storage system.
In some embodiments of the present invention, the data query apparatus further includes:
a determining module 406, configured to analyze the data query request and determine whether to obtain target object data from the second cache region; when the target object data are determined to be acquired from the second cache region, calling the target object data in the second cache region, and generating and sending a data query result; and when determining that the target object data is not acquired from the second cache region, inquiring the ID of the target object in the object ID index, acquiring the target object data from the distributed storage system according to the ID of the target object, and generating and sending a data inquiry result.
In some embodiments of the present invention, the writing module 404 is further configured to:
setting expiration time for the second cache region after the second cache region is accessed for the first time;
and the second cache region is not accessed again within the expiration time, and the data stored in the second cache region is cleared.
In some embodiments of the present invention, the data query apparatus further includes:
an updating module 407, configured to write an object ID in the update data into the first cache region when the object data stored in the second cache region is updated, and update the object ID index; storing the updated data into a distributed storage system, and emptying the second cache region; and after receiving a data query request, extracting updated target object data from the distributed storage system based on the object ID index, and storing the target object data in a second cache region.
In some embodiments of the present invention, the update module 407 is configured to:
adding update data to the object data stored in the second cache region, and/or
Deleting part of the object data stored in the second cache region, and/or
And replacing the object data stored in the second cache region.
Since each functional module of the data query apparatus according to the exemplary embodiment of the present invention corresponds to the steps of the exemplary embodiment of the data query method according to the first aspect, please refer to the data query method according to the first aspect of the present invention for details that are not disclosed in the embodiment of the apparatus according to the present invention.
Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use in implementing an electronic device of an embodiment of the present invention. The computer system 500 of the electronic device shown in fig. 5 is only an example, and should not bring any limitation to the function and the scope of the use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 1108 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for system operation are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the invention include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The above-described functions defined in the system of the present application are executed when the computer program is executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present invention may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.
As another aspect, the present application also provides a computer-readable medium, which may be contained in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to implement the certificate management method in the above embodiment.
For example, the electronic device described above may implement as shown in fig. 1: step S110, writing the object data ID into a first cache region according to a preset sorting rule, and generating an object ID index; step S210, after receiving the object data, storing the object data in a relational database and a distributed storage system; and step 130, implementing data query based on the object data stored in the second cache region.
As another example, the electronic device described above may implement the steps shown in fig. 2.
As another example, the electronic device described above may implement the steps shown in fig. 3.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the invention. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
Claims (10)
1. A method for querying data, comprising:
responding to a data query request, wherein the data query request comprises: the target object acquires the ID of the target object based on the object ID index stored in the first cache region;
extracting target object data from a distributed storage system based on the ID of the target object, and storing the target object data in a second cache region;
and realizing data query based on the target object data stored in the second cache region.
2. The data query method of claim 1, wherein before responding to the data query request, the method further comprises:
receiving object data, and storing the object data in a relational database and a distributed storage system;
and writing the ID of the object data into a first cache region according to a preset sorting rule to generate an object ID index, wherein the ID of the object data is an integer ID.
3. The data query method of claim 2, wherein the receiving object data and storing the object data in a relational database and a distributed storage system comprises:
storing object data to be counted in the relational database, and counting the stored object data;
and storing the full amount of object data obtained after statistics in the distributed storage system.
4. The data query method of claim 1, wherein the implementing the data query based on the target object data stored in the second cache region comprises:
analyzing the data query request, and judging whether target object data are acquired from the second cache region or not;
when the target object data are determined to be acquired from the second cache region, calling the target object data in the second cache region, and generating and sending a data query result;
and when determining that the target object data is not acquired from the second cache region, inquiring the ID of the target object in the object ID index, acquiring the target object data from the distributed storage system according to the ID of the target object, and generating and sending a data inquiry result.
5. The data query method of claim 1, wherein the storing the target object data in a second cache region comprises:
setting expiration time for the second cache region based on access time after the second cache region is accessed;
and clearing the data stored in the second cache region when the second cache region is not accessed again in the expiration time.
6. The data query method of claim 1, further comprising:
when the object data stored in the second cache region are updated, writing the object ID in the updated data into the first cache region, and updating the object ID index;
storing the updated data into a distributed storage system, and emptying the second cache region;
and after receiving a data query request, extracting updated target object data from the distributed storage system based on the object ID index, and storing the target object data in a second cache region.
7. The data query method according to claim 6, wherein the updating the object data stored in the second cache region includes:
adding update data to the object data stored in the second cache region, and/or
Deleting part of the object data stored in the second cache region, and/or
And replacing the object data stored in the second cache region.
8. A data query apparatus, comprising: the system comprises a response module, an extraction module and a data query module; wherein the content of the first and second substances,
a response module, configured to respond to a data query request, where the data query request includes: the target object acquires the ID of the target object based on the object ID index stored in the first cache region;
the extracting module is used for extracting target object data from the distributed storage system based on the ID of the target object and storing the target object data in a second cache region;
and the data query module is used for realizing data query based on the target object data stored in the second cache region.
9. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the data query method according to any one of claims 1 to 7.
10. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out a data query method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810571915.6A CN110633296A (en) | 2018-05-31 | 2018-05-31 | Data query method, device, medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810571915.6A CN110633296A (en) | 2018-05-31 | 2018-05-31 | Data query method, device, medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110633296A true CN110633296A (en) | 2019-12-31 |
Family
ID=68966142
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810571915.6A Pending CN110633296A (en) | 2018-05-31 | 2018-05-31 | Data query method, device, medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110633296A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111310076A (en) * | 2020-02-25 | 2020-06-19 | 同盾控股有限公司 | Geographic position query method, device, medium and electronic equipment |
| CN111488382A (en) * | 2020-04-16 | 2020-08-04 | 北京思特奇信息技术股份有限公司 | Data calling method and system and electronic equipment |
| CN111597403A (en) * | 2020-05-21 | 2020-08-28 | 北京字节跳动网络技术有限公司 | Method and device for constructing graph index, electronic equipment and storage medium |
| CN111782694A (en) * | 2020-06-28 | 2020-10-16 | 百度在线网络技术（北京）有限公司 | Information writing method and device and electronic equipment |
| CN111797094A (en) * | 2020-05-25 | 2020-10-20 | 北京旷视科技有限公司 | Data processing method and device and electronic equipment |
| CN112632411A (en) * | 2020-12-24 | 2021-04-09 | 武汉旷视金智科技有限公司 | Target object data query method, device, equipment and storage medium |
| CN113726827A (en) * | 2020-05-25 | 2021-11-30 | 北京同邦卓益科技有限公司 | Data packet processing method and device based on distributed cluster |
| CN113779082A (en) * | 2021-01-05 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Method and device for updating data |
| CN115314718A (en) * | 2021-05-07 | 2022-11-08 | 北京字节跳动网络技术有限公司 | Live broadcast data processing method, device, equipment and medium |
| CN117668003A (en) * | 2024-02-01 | 2024-03-08 | 福建省华大数码科技有限公司 | Data processing method and system for realizing aggregate data types in database |
| CN117668003B (en) * | 2024-02-01 | 2024-05-03 | 福建省华大数码科技有限公司 | Data processing method and system for realizing aggregate data types in database |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100332846A1 (en) * | 2009-06-26 | 2010-12-30 | Simplivt Corporation | Scalable indexing |
| CN103853727A (en) * | 2012-11-29 | 2014-06-11 | 深圳中兴力维技术有限公司 | Method and system for improving large data volume query performance |
| CN105354151A (en) * | 2014-08-19 | 2016-02-24 | 阿里巴巴集团控股有限公司 | Cache management method and device |
-
2018
- 2018-05-31 CN CN201810571915.6A patent/CN110633296A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100332846A1 (en) * | 2009-06-26 | 2010-12-30 | Simplivt Corporation | Scalable indexing |
| CN103853727A (en) * | 2012-11-29 | 2014-06-11 | 深圳中兴力维技术有限公司 | Method and system for improving large data volume query performance |
| CN105354151A (en) * | 2014-08-19 | 2016-02-24 | 阿里巴巴集团控股有限公司 | Cache management method and device |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111310076A (en) * | 2020-02-25 | 2020-06-19 | 同盾控股有限公司 | Geographic position query method, device, medium and electronic equipment |
| CN111310076B (en) * | 2020-02-25 | 2023-10-03 | 同盾控股有限公司 | Geographic position query method, geographic position query device, geographic position query medium and electronic equipment |
| CN111488382A (en) * | 2020-04-16 | 2020-08-04 | 北京思特奇信息技术股份有限公司 | Data calling method and system and electronic equipment |
| CN111597403A (en) * | 2020-05-21 | 2020-08-28 | 北京字节跳动网络技术有限公司 | Method and device for constructing graph index, electronic equipment and storage medium |
| CN111597403B (en) * | 2020-05-21 | 2023-06-27 | 抖音视界有限公司 | Method and device for constructing graph index, electronic equipment and storage medium |
| CN113726827A (en) * | 2020-05-25 | 2021-11-30 | 北京同邦卓益科技有限公司 | Data packet processing method and device based on distributed cluster |
| CN111797094A (en) * | 2020-05-25 | 2020-10-20 | 北京旷视科技有限公司 | Data processing method and device and electronic equipment |
| CN111782694A (en) * | 2020-06-28 | 2020-10-16 | 百度在线网络技术（北京）有限公司 | Information writing method and device and electronic equipment |
| CN111782694B (en) * | 2020-06-28 | 2023-10-20 | 百度在线网络技术（北京）有限公司 | Information writing method and device and electronic equipment |
| CN112632411A (en) * | 2020-12-24 | 2021-04-09 | 武汉旷视金智科技有限公司 | Target object data query method, device, equipment and storage medium |
| CN113779082A (en) * | 2021-01-05 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Method and device for updating data |
| CN115314718A (en) * | 2021-05-07 | 2022-11-08 | 北京字节跳动网络技术有限公司 | Live broadcast data processing method, device, equipment and medium |
| CN115314718B (en) * | 2021-05-07 | 2023-07-14 | 北京字节跳动网络技术有限公司 | Live broadcast data processing method, device, equipment and medium |
| CN117668003A (en) * | 2024-02-01 | 2024-03-08 | 福建省华大数码科技有限公司 | Data processing method and system for realizing aggregate data types in database |
| CN117668003B (en) * | 2024-02-01 | 2024-05-03 | 福建省华大数码科技有限公司 | Data processing method and system for realizing aggregate data types in database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110633296A (en) | Data query method, device, medium and electronic equipment | |
| CN109299348B (en) | Data query method and device, electronic equipment and storage medium | |
| CN110708212B (en) | Method and device for tracking call link in distributed system | |
| CN110706093A (en) | Accounting processing method and device | |
| CN112395322A (en) | List data display method and device based on hierarchical cache and terminal equipment | |
| CN111857539A (en) | Method, apparatus and computer program product for managing a storage system | |
| CA3139693A1 (en) | Dynamic allocation of computing resources | |
| CN107748649B (en) | Method and device for caching data | |
| CN110928594A (en) | Service development method and platform | |
| EP2662783A1 (en) | Data archiving approach leveraging database layer functionality | |
| CN111401934A (en) | Distributed advertisement statistical method and device | |
| CN111176641A (en) | Flow node execution method, device, medium and electronic equipment | |
| CN115599787A (en) | Level sub-metering method and device, electronic equipment and storage medium | |
| CN115545639A (en) | Financial business processing method and device, electronic equipment and storage medium | |
| CN115147183A (en) | Chip resource management method, device, equipment and storage medium based on cloud platform | |
| CN112131257B (en) | Data query method and device | |
| CN114661523A (en) | Data backup method, device, program product, medium and electronic equipment | |
| CN113468258A (en) | Heterogeneous data conversion method and device and storage medium | |
| CN113204426A (en) | Task processing method of resource pool and related equipment | |
| CN113094415A (en) | Data extraction method and device, computer readable medium and electronic equipment | |
| CN111382178A (en) | Network task allocation method, system, equipment and storage medium | |
| CN111125112A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN112052330A (en) | Application keyword distribution method and device | |
| CN110716885B (en) | Data management method and device, electronic equipment and storage medium | |
| CN116701384A (en) | Method, device and storage medium for storing database and table aiming at vehicle condition data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |