CN111400322B - Method, apparatus, electronic device and medium for storing data - Google Patents
Method, apparatus, electronic device and medium for storing data Download PDFInfo
- Publication number
- CN111400322B CN111400322B CN202010218079.0A CN202010218079A CN111400322B CN 111400322 B CN111400322 B CN 111400322B CN 202010218079 A CN202010218079 A CN 202010218079A CN 111400322 B CN111400322 B CN 111400322B
- Authority
- CN
- China
- Prior art keywords
- data
- item
- target data
- information
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
Embodiments of the present disclosure disclose methods, apparatuses, electronic devices, and media for storing data. One embodiment of the method comprises the following steps: for each data interval in the target number of data intervals, generating an identification of the data interval; acquiring data related information of each item of target data in at least one item of target data, wherein the data related information comprises time information of the target data; for each item of target data, the target data is stored based on the data-related information of the target data and the identification group. In the embodiment, the data related information of the target data and the identification group are considered first in the data storage, so that the simultaneous storage of a large amount of data is avoided.
Description
Technical Field
Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a method, an apparatus, an electronic device, and a medium for storing data.
Background
Large internet traffic typically generates massive amounts of logs, often requiring the retrieval of the required data by scanning the massive amounts of logs. Large data processing frameworks are typically used for the processing of large amounts of data. Because the data size is too large, the data cannot be stored directly into a conventional relational database.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
It is an aim of some embodiments of the present disclosure to propose an improved method, apparatus, electronic device and medium for storing data, solving the technical problems mentioned in the background section above.
In a first aspect, some embodiments of the present disclosure provide a method for storing data, the method comprising: for each data interval in the target number of data intervals, generating an identification of the data interval; acquiring data related information of each item of target data in at least one item of target data, wherein the data related information comprises time information of the target data; for each item of target data, the target data is stored based on the data-related information of the target data and the identification group.
In a second aspect, some embodiments of the present disclosure provide a method for querying data, the method comprising: acquiring query information corresponding to data based on category information of the data to be queried, wherein the query information comprises: partition main key information and inquiry time range information of the data; determining the identification of a data interval corresponding to the data based on the partition main key information; determining a message file block where the data is based on the inquiry time range information and the identification; and screening target data from the message file block based on the query information.
In a third aspect, some embodiments of the present disclosure provide an apparatus for storing data, the apparatus comprising: a generation unit configured to generate, for each of a target number of data intervals, an identification of the data interval; an acquisition unit configured to acquire data-related information of each item of target data in at least one item of target data, the data-related information including time information of the target data; and a storage unit configured to store the target data based on the data-related information of the target data and the identification group for each item of target data.
In a fourth aspect, some embodiments of the present disclosure provide an apparatus for querying data, the apparatus comprising: an acquisition unit configured to acquire query information corresponding to data based on category information of the data to be queried, wherein the query information includes: partition main key information and inquiry time range information of the data; a first determination unit configured to determine an identification of a data section corresponding to the data based on the partition primary key information; a second determining unit configured to determine a message file block in which the data is located, based on the inquiry time range information and the identification; and a screening unit configured to screen target data from the message file block based on the query information.
In a fifth aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method as in any of the first aspects.
In a sixth aspect, some embodiments of the present disclosure provide a computer readable medium having a computer program stored thereon, wherein the program when executed by a processor implements a method as in any of the first aspects.
One of the above embodiments of the present disclosure has the following advantageous effects: firstly, for each data interval in a target number of data intervals, generating an identification of the data interval; then, acquiring data related information of each item of target data in at least one item of target data, wherein the data related information comprises time information of the target data; finally, for each item of target data, the target data is stored based on the data-related information of the target data and the identification group. The data related information of the target data and the identification group are considered at first when the data is stored, so that the simultaneous storage of a large amount of data is avoided.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.
FIG. 1 is a schematic illustration of one application scenario of a method for storing data according to some embodiments of the present disclosure;
FIG. 2 is a flow chart of some embodiments of a method for storing data according to the present disclosure;
FIG. 3 is a schematic illustration of one application scenario of a method for querying data according to some embodiments of the present disclosure;
FIG. 4 is a flow chart of some embodiments of a method for querying data according to the present disclosure;
FIG. 5 is a schematic structural diagram of some embodiments of an apparatus for a method of storing data according to the present disclosure;
FIG. 6 is a schematic structural diagram of some embodiments of an apparatus for a method of querying data according to the present disclosure;
fig. 7 is a schematic diagram of a computer system suitable for use in implementing embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings. Embodiments of the present disclosure and features of embodiments may be combined with each other without conflict.
The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 is a schematic diagram of one application scenario of a method for storing data according to some embodiments of the present disclosure.
As shown in the application scenario of fig. 1, an execution subject of the method for storing data may be the server 101. First, the server 101 may obtain, based on the target number (3 shown in the figure) of data sections 102-104, an identifier 105 (1 shown in the figure) corresponding to the data section 102, an identifier 106 (2 shown in the figure) corresponding to the data section 103, and an identifier 107 (3 shown in the figure) corresponding to the data section 104. Next, the server 101 may acquire data-related information 111 (shown as "9 hours 40 minutes 00 seconds" in the drawing) of at least one piece (3 pieces shown in the drawing) of target data 108, data-related information 112 (shown as "9 hours 41 minutes 30 seconds" in the drawing) of target data 109, and data-related information 113 (shown as "9 hours 46 minutes 00 seconds" in the drawing) of target data 110. Wherein the data-related information includes: and the time information corresponding to each item of target data in the at least one item of target data. Here, the above-described time information may be a time when the server 101 acquired the target data. Finally, the server 101 may store the at least one item of target data 102-105 based on the data-related information 111-113 and the identifiers 105-107.
It will be appreciated that the method for generating a picture may be performed by the server 101, or by other devices, or by various software programs. The server 101 may be, for example, various electronic devices with a display screen including, but not limited to, smartphones, tablet computers, electronic book readers, laptop and desktop computers, and the like. The execution body may be embodied as a server, software, or the like. When the execution subject is software, the execution subject can be installed in the electronic device enumerated above. It may be implemented as a plurality of software or software modules, for example, for providing distributed services, or as a single software or software module. The present invention is not particularly limited herein.
It should be understood that the number of servers in fig. 1 is merely illustrative. There may be any number of servers, as desired for implementation.
With continued reference to fig. 2, a flow 200 of some embodiments of a method for storing data according to the present disclosure is shown. The method for storing data comprises the following steps:
step 201, for each data interval in the target number of data intervals, generating an identification of the data interval.
In some embodiments, an execution body of a method for storing data (e.g., server 101 shown in fig. 1) may generate, for each of a target number of data intervals, an identification of the data interval. The identification of the target number of data intervals may be sequentially generated in a natural number order. The target number may be predetermined or manually set. The data section may be a section for storing a part of data. The identification may be a partition number added to the data section. The partition number may be an integer value, for example, 0,1,2,3, … n. As an example, when the target number is 3, the corresponding identifier of each data section may be "1", "2", or "3", respectively.
Step 202, obtaining data related information of each item of target data in at least one item of target data, wherein the data related information comprises time information of the target data.
In some embodiments, the executing entity may obtain data-related information of each item of target data in the at least one item of target data, where the data-related information includes time information of the target data. As an example, the at least one item of target data may be data transmitted by a user watching live broadcast in a live broadcast scene, or may be data transmitted by a user being live broadcast. The time information may be a time when the user transmits data, and for example, the time information may be 19 hours 29 minutes 32 seconds.
In some optional implementations of some embodiments, the data-related information further includes: and category information corresponding to each item of target data in the at least one item of target data. The category information includes: and issuing the message class, and pulling out the message class. As an example, the target data in the above-mentioned download message class may be data transmitted by a user who is being live in a live scene. The pull message class may be data viewed by a user viewing a live in a live scene.
In some optional implementations of some embodiments, for each data interval in the target number of data intervals, generating the identifier of the data interval includes: obtaining a partition main key corresponding to each item of target data in the at least one item of target data based on the category information corresponding to each item of target data in the at least one item of target data; as an example, when the category information of the target data is a down message category, the partition main key of the target data may be a room number or a device number. The room number may be a number of a live room, for example, the number may be "0001". The device number may be a device number of a device viewing live broadcast, for example, the device number may be "a1001". And obtaining the identification corresponding to each data interval based on the target number and the partition primary key corresponding to each item of target data in the at least one item of target data. As an example, the above-mentioned identification may be calculated by the following formula: partitionid=hashfunc (key, N). Wherein, "PartitionId" represents a corresponding identifier for each data interval. "key" indicates the partition main key corresponding to the target data. "N" represents the above-mentioned target number. "HashFunc" represents a division function for generating a corresponding identification for each data interval based on given "keys" and "N".
Step 203, for each item of target data, storing the target data based on the data-related information of the target data and the identification group.
In some embodiments, the execution body may store the target data based on the data-related information of the target data and the identification group for each item of target data. As an example, the target data may be stored in a data section corresponding to the identifier in the identifier group based on the data-related information of each item of the target data and the identifier group.
In some optional implementations of some embodiments, for each item of target data, storing the target data based on the data-related information of the target data and the identification group includes: according to the time information, at least one message file block respectively storing at least one item of label data is obtained; the time information may be a time when the user transmits data, and for example, the time information may be 19 hours 29 minutes 32 seconds. The message file block may be for storing target data. Here, one message file block may be used to store 5 minutes of target data. And adding a file name to the at least one message file block based on the time information and the identification, respectively. As an example, when the category information of the target data is a down message class, the above file names may be named in the following manner: msg_ [ time ] _ [ part ]. Log. Where "msg" may indicate that the class information of the target data is a down message class. "time" means the time when the message file block holds the target data, for example, the message file block holds the target data for 5 minutes, and then the "time" may be "19 hours 30 minutes". "part" indicates the identification of the data section corresponding to the target data.
In some optional implementations of some embodiments, for each item of target data, storing the target data based on the data-related information of the target data and the identification group includes: and storing the message primary key of the at least one item of target data based on the data related information and the identification in response to the category information of the at least one item of target data being a pull message category. The message primary key may be a primary key corresponding to the target data, and for example, the primary key may be "0,1,2, …, n". As an example, when the category information of the target data is a pull message category, the message primary key of the at least one item of target data is stored in at least one message file block based on the time information of the target data. The file names of the above-mentioned message file blocks may be named in the following manner: ack_ [ time ] _ [ part ]. Log. Where "ack" may indicate that the class information of the target data is a pull message class. "time" means the time of the message primary key of the message file block holding the target data, for example, the message file block holding the target data for 5 minutes, and then the "time" may be "19 hours 30 minutes". "part" indicates the identification of the data section corresponding to the target data.
As can be seen from the above example, first, for each of a target number of data intervals, an identification of the data interval is generated; then, acquiring data related information of each item of target data in at least one item of target data, wherein the data related information comprises time information of the target data; finally, for each item of target data, the target data is stored based on the data-related information of the target data and the identification group. The data related information of the target data and the identification group are considered at first when the data is stored, so that the simultaneous storage of a large amount of data is avoided.
Fig. 3 is a schematic diagram of one application scenario of a method for querying data according to some embodiments of the present disclosure.
As shown in the application scenario of fig. 3, an execution subject of the method for querying data may be the server 301. First, the server 301 may obtain, based on the category information 302 of the data to be queried, query information 303 corresponding to the data, where the query information 303 includes: partition primary key information 304 and inquiry time range information 305 of the above data; the category information 302 includes: and issuing the message class, and pulling out the message class. The inquiry time range information 305 includes: the start time and the end time may be, for example, ""19 hours 28 minutes ","19 hours 31 minutes ". Next, based on the partition primary key information 304, determining an identifier 306 of a data section corresponding to the data; then, based on the inquiry time range information 305 and the identifier 306, a message file block 307 in which the data is located is determined; finally, the target data 308 is screened from the message file block 307 based on the query information 303.
It will be appreciated that the method for generating a picture may be performed by the server 301, or by other devices, or by various software programs. The server 301 may be, for example, various electronic devices with a display screen, including but not limited to a smart phone, a tablet computer, an electronic book reader, a laptop portable computer, a desktop computer, and the like. The execution body may be embodied as a server, software, or the like. When the execution subject is software, the execution subject can be installed in the electronic device enumerated above. It may be implemented as a plurality of software or software modules, for example, for providing distributed services, or as a single software or software module. The present invention is not particularly limited herein.
It should be understood that the number of servers in fig. 3 is merely illustrative. There may be any number of servers, as desired for implementation.
With continued reference to fig. 4, a flow 400 of some embodiments of a method for querying data according to the present disclosure is shown. The method for querying data comprises the following steps:
step 401, acquiring query information corresponding to data based on category information of the data to be queried.
In some embodiments, an executing body (e.g., the server 301 shown in fig. 3) may first acquire query information corresponding to data to be queried based on category information of the data. The category information includes: and issuing the message class, and pulling out the message class. The query information may include at least one of: partition primary key information and inquiry time range information of the data. As an example, when the category information of the above data is a down message category, the corresponding query information includes: partition primary key information and inquiry time range information of the data. The inquiry time range information includes: start time and end time. For example, the inquiry time range information may be "19 hours 28 minutes", "19 hours 31 minutes".
Step 402, determining the identification of the data section corresponding to the data based on the partition primary key information.
In some embodiments, the executing body may determine the identifier of the data section corresponding to the data based on the partition primary key information. As an example, the identification of the data interval may be obtained by the following formula: partitionid=hashfunc (RoomId, N). Wherein, "PartitionId" represents the identification of the above-mentioned data interval. "RoomId" indicates partition primary key information of the above data. "N" represents the number of data intervals.
Step 403, determining a message file block where the data is located based on the inquiry time range information and the identification.
In some embodiments, the executing entity may first determine a message file block in which the data is located based on the query time range information obtained in step 401 and based on the identification obtained in step 402. The message file block may be for storing target data. As an example, the file name of the message file block may be obtained by the following formula: msg_ [ time ] _ partitionid.log. Where "msg" may indicate that the class information of the target data is a down message class. "time" means the time when the message file block holds the target data, for example, the message file block holds the target data for 5 minutes, and then the "time" may be "19 hours 30 minutes". "part" indicates the identification of the data section corresponding to the target data. As an example, the above-described inquiry time range information may be "19 hours 28 minutes", "19 hours 31 minutes". The time indicated by "time" is required to be after the start time and before the end time in the inquiry time range information.
And step 404, screening target data from the message file block based on the query information.
In some embodiments, the executing entity may screen the target data from the message file block based on the query information obtained in step 401.
The method for querying data disclosed in some embodiments of the present disclosure includes first, acquiring query information corresponding to data to be queried based on category information of the data to be queried; secondly, determining the identification of a data interval corresponding to the data based on the partition primary key information; then, based on the inquiry time range information and the identification, determining a message file block in which the data is located; and finally, screening target data from the message file block based on the query information. When the data is queried, the corresponding message file block can be found first, and when the data is queried from the message file block, the size of the file to be scanned is reduced, and the query speed is increased.
With further reference to fig. 5, as an implementation of the method described above for the various figures, the present disclosure provides some embodiments of an apparatus for storing data, which apparatus embodiments correspond to those described above for fig. 2, which apparatus is particularly applicable in a variety of electronic devices.
As shown in fig. 5, an apparatus 500 for storing data of some embodiments includes: a generation unit 501, an acquisition unit 502, and a storage unit 503. Wherein the generating unit 501 is configured to generate, for each of the target number of data intervals, an identification of the data interval; an obtaining unit 502 configured to obtain data-related information of each item of target data in at least one item of target data, the data-related information including time information of the target data; the storage unit 503 is configured to store the target data based on the data-related information of the target data and the identification group for each item of target data.
In some optional implementations of some embodiments, the data-related information further includes: the category information corresponding to each item of target data in the at least one item of target data includes: and issuing the message class, and pulling out the message class.
In some optional implementations of some embodiments, the generating unit 501 in the apparatus 500 for storing data is further configured to: obtaining a partition main key corresponding to each item of target data in the at least one item of target data based on the category information corresponding to each item of target data in the at least one item of target data; and obtaining the identification corresponding to each data interval based on the target number and the partition primary key corresponding to each item of target data in the at least one item of target data.
In some alternative implementations of some embodiments, the storage unit 503 in the apparatus 500 for storing data is further configured to: according to the time information, at least one message file block respectively storing at least one item of label data is obtained; and adding a file name to the at least one message file block based on the time information and the identification, respectively.
In some alternative implementations of some embodiments, the storage unit 503 in the apparatus 500 for storing data is further configured to: and storing the message primary key of the at least one item of target data based on the data related information and the identification in response to the category information of the at least one item of target data being a pull message category.
It will be appreciated that the elements described in the apparatus 500 correspond to the various steps in the method described with reference to fig. 2. Thus, the operations, features and resulting benefits described above with respect to the method are equally applicable to the apparatus 500 and the units contained therein, and are not described in detail herein.
With further reference to fig. 6, as an implementation of the method described above for the various figures, the present disclosure provides some embodiments of an apparatus for querying data, which apparatus embodiments correspond to those described above for fig. 4, and which apparatus is particularly applicable in a variety of electronic devices.
As shown in fig. 6, an apparatus 600 for querying data of some embodiments includes: an acquisition unit 601, a first determination unit 602, a second determination unit 603, and a screening unit 604. The acquiring unit 601 is configured to acquire query information corresponding to data to be queried based on category information of the data, where the query information includes: partition main key information and inquiry time range information of the data; a first determining unit 602 configured to determine, based on the partition primary key information, an identification of a data section corresponding to the data; a second determining unit 603 configured to determine a message file block in which the data is located, based on the inquiry time range information and the identification; and a screening unit 604 configured to screen the target data from the message file block based on the query information.
It will be appreciated that the elements described in the apparatus 600 correspond to the various steps in the method described with reference to fig. 4. Thus, the operations, features and resulting benefits described above with respect to the method are equally applicable to the apparatus 600 and the units contained therein, and are not described in detail herein.
Referring now to fig. 7, a schematic diagram of an electronic device (e.g., server in fig. 1) 700 suitable for use in implementing some embodiments of the present disclosure is shown. Terminal devices in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), car terminals (e.g., car navigation terminals), and the like, as well as stationary terminals such as digital TVs, desktop computers, and the like. The terminal device shown in fig. 7 is only one example, and should not impose any limitation on the functions and scope of use of the embodiments of the present disclosure.
As shown in fig. 7, the electronic device 700 may include a processing means (e.g., a central processor, a graphics processor, etc.) 701, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage means 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
In general, the following devices may be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, and the like; an output device 707 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 708 including, for example, a memory card; and a communication device 709. The communication means 709 may allow the electronic device 700 to communicate wirelessly or by wire with other devices to exchange data. While fig. 7 shows an electronic device 700 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead. Each block shown in fig. 7 may represent one device or a plurality of devices as needed.
In particular, according to some embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via communications device 709, or from storage 708, or from ROM 702. The above-described functions defined in the methods of some embodiments of the present disclosure are performed when the computer program is executed by the processing means 701.
It should be noted that, in some embodiments of the present disclosure, the computer readable medium may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, the computer-readable signal medium may comprise a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: for each data interval in the target number of data intervals, generating an identification of the data interval; acquiring data related information of each item of target data in at least one item of target data, wherein the data related information comprises time information of the target data; for each item of target data, the target data is stored based on the data-related information of the target data and the identification group.
Computer program code for carrying out operations for some embodiments of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in some embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. The described units may also be provided in a processor, for example, described as: a processor includes a generation unit, an acquisition unit, and a storage unit. The names of these units do not in some cases limit the unit itself, and the generating unit may also be described as "a unit that generates an identification of a target number of data sections for each data section of the data sections", for example.
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
According to one or more embodiments of the present disclosure, there is provided a method for storing data, comprising: for each data interval in the target number of data intervals, generating an identification of the data interval; acquiring data related information of each item of target data in at least one item of target data, wherein the data related information comprises time information of the target data; for each item of target data, the target data is stored based on the data-related information of the target data and the identification group.
According to one or more embodiments of the present disclosure, the data-related information further includes category information of the target data, wherein the category information includes: and issuing the message class, and pulling out the message class.
According to one or more embodiments of the present disclosure, the generating, for each data interval of the target number of data intervals, an identifier of the data interval includes: obtaining a partition main key corresponding to each item of target data in the at least one item of target data based on the category information corresponding to each item of target data in the at least one item of target data; and obtaining the identification corresponding to each data interval based on the target number and the partition primary key corresponding to each item of target data in the at least one item of target data.
According to one or more embodiments of the present disclosure, the storing the target data based on the data-related information of the target data and the identification group for each item of target data includes: according to the time information, at least one message file block respectively storing at least one item of label data is obtained; and adding a file name to the at least one message file block based on the time information and the identification, respectively.
According to one or more embodiments of the present disclosure, the storing the target data based on the data-related information of the target data and the identification group for each item of target data includes: and storing the message primary key of the at least one item of target data based on the data related information and the identification in response to the category information of the at least one item of target data being a pull message category.
In accordance with one or more embodiments of the present disclosure, there is provided a method for querying data, comprising: acquiring query information corresponding to data based on category information of the data to be queried, wherein the query information comprises: partition main key information and inquiry time range information of the data; determining the identification of a data interval corresponding to the data based on the partition main key information; determining a message file block where the data is based on the inquiry time range information and the identification; and screening target data from the message file block based on the query information.
According to one or more embodiments of the present disclosure, there is provided an apparatus for storing data, comprising: a generation unit configured to generate, for each of a target number of data intervals, an identification of the data interval; an acquisition unit configured to acquire data-related information of each item of target data in at least one item of target data, the data-related information including time information of the target data; and a storage unit configured to store the target data based on the data-related information of the target data and the identification group for each item of target data.
According to one or more embodiments of the present disclosure, the data-related information further includes category information of the target data, wherein the category information includes: and issuing the message class, and pulling out the message class.
According to one or more embodiments of the present disclosure, the generating unit in the above apparatus is further configured to: obtaining a partition main key corresponding to each item of target data in the at least one item of target data based on the category information corresponding to each item of target data in the at least one item of target data; and obtaining the identification corresponding to each data interval based on the target number and the partition primary key corresponding to each item of target data in the at least one item of target data.
According to one or more embodiments of the present disclosure, the storage unit in the above apparatus is further configured to: according to the time information, at least one message file block respectively storing at least one item of label data is obtained; and adding a file name to the at least one message file block based on the time information and the identification, respectively.
According to one or more embodiments of the present disclosure, the storage unit in the above apparatus is further configured to: and storing the message primary key of the at least one item of target data based on the data related information and the identification in response to the category information of the at least one item of target data being a pull message category.
According to one or more embodiments of the present disclosure, there is provided an apparatus for querying data, comprising: an acquisition unit configured to acquire query information corresponding to data based on category information of the data to be queried, wherein the query information includes: partition main key information and inquiry time range information of the data; a first determination unit configured to determine an identification of a data section corresponding to the data based on the partition primary key information; a second determining unit configured to determine a message file block in which the data is located, based on the inquiry time range information and the identification; and a screening unit configured to screen target data from the message file block based on the query information.
According to one or more embodiments of the present disclosure, there is provided an electronic device including: one or more processors; and a storage device having one or more programs stored thereon, which when executed by the one or more processors, cause the one or more processors to implement the method as described in any of the embodiments above.
According to one or more embodiments of the present disclosure, there is provided a computer readable medium having stored thereon a computer program, wherein the program, when executed by a processor, implements a method as described in any of the embodiments above.
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above technical features, but encompasses other technical features formed by any combination of the above technical features or their equivalents without departing from the spirit of the invention. Such as the above-described features, are mutually substituted with (but not limited to) the features having similar functions disclosed in the embodiments of the present disclosure.
Claims (9)
1. A method for storing data, comprising:
for each data interval in the target number of data intervals, generating an identification of the data interval; the identification of the data interval is obtained based on the target number and partition main keys corresponding to each item of target data in at least one item of target data; the partition main key corresponding to each item of target data in the at least one item of target data is obtained based on the category information corresponding to each item of target data in the at least one item of target data;
acquiring data related information of each item of target data in at least one item of target data, wherein the data related information comprises time information of the target data;
For each item of target data, storing the target data based on the data related information of the target data and the identification.
2. The method of claim 1, wherein the data-related information further comprises category information of the target data, wherein the category information comprises: and issuing the message class, and pulling out the message class.
3. The method of claim 1, wherein the storing the target data based on the data-related information and the identification of the target data for each item of target data comprises:
according to the time information, at least one message file block respectively storing at least one item of label data is obtained;
and adding file names to the at least one message file block respectively based on the time information and the identification.
4. The method of claim 1, wherein the storing the at least one item of tag data based on the data-related information and the identification comprises:
and storing the message main key of the at least one item of target data based on the data related information and the identification in response to the category information of the at least one item of target data being a pull message class.
5. A method for querying data, comprising:
acquiring query information corresponding to data based on category information of the data to be queried, wherein the query information comprises: partition main key information and inquiry time range information of the data;
determining the identification of a data interval corresponding to the data based on the partition primary key information;
the identification of the data interval is obtained based on the target number and partition main keys corresponding to each item of target data in at least one item of target data; the partition main key corresponding to each item of target data in the at least one item of target data is obtained based on the category information corresponding to each item of target data in the at least one item of target data;
determining a message file block where the data is based on the inquiry time range information and the identification;
and screening target data from the message file block based on the query information.
6. An apparatus for storing data, comprising:
a generation unit configured to generate, for each of a target number of data intervals, an identification of the data interval; the identification of the data interval is obtained based on the target number and partition main keys corresponding to each item of target data in at least one item of target data; the partition main key corresponding to each item of target data in the at least one item of target data is obtained based on the category information corresponding to each item of target data in the at least one item of target data;
An acquisition unit configured to acquire data-related information of each item of target data in at least one item of target data, the data-related information including time information of the target data;
and a storage unit configured to store, for each item of target data, the target data based on the data-related information of the target data and the identification.
7. An apparatus for querying data, comprising:
an acquisition unit configured to acquire query information corresponding to data based on category information of the data to be queried, wherein the query information includes: partition main key information and inquiry time range information of the data;
a first determining unit configured to determine an identification of a data section corresponding to the data based on the partition primary key information; the identification of the data interval is obtained based on the target number and partition main keys corresponding to each item of target data in at least one item of target data; the partition main key corresponding to each item of target data in the at least one item of target data is obtained based on the category information corresponding to each item of target data in the at least one item of target data;
a second determining unit configured to determine a message file block in which the data is located, based on the inquiry time range information and the identification;
And a screening unit configured to screen target data from the message file block based on the query information.
8. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1 to 5.
9. A computer readable medium having stored thereon a computer program, wherein the program when executed by a processor implements the method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010218079.0A CN111400322B (en) | 2020-03-25 | 2020-03-25 | Method, apparatus, electronic device and medium for storing data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010218079.0A CN111400322B (en) | 2020-03-25 | 2020-03-25 | Method, apparatus, electronic device and medium for storing data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400322A CN111400322A (en) | 2020-07-10 |
| CN111400322B true CN111400322B (en) | 2023-10-03 |
Family
ID=71429255
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010218079.0A Active CN111400322B (en) | 2020-03-25 | 2020-03-25 | Method, apparatus, electronic device and medium for storing data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400322B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105260404A (en) * | 2015-09-22 | 2016-01-20 | 北京百度网讯科技有限公司 | Time sequence data storage method and apparatus |
| CN106648446A (en) * | 2015-10-30 | 2017-05-10 | 阿里巴巴集团控股有限公司 | Time series data storage method and apparatus, and electronic device |
| CN107315533A (en) * | 2016-04-26 | 2017-11-03 | 杭州海康威视数字技术股份有限公司 | A kind of date storage method and device |
| CN108256088A (en) * | 2018-01-23 | 2018-07-06 | 清华大学 | A kind of storage method and system of the time series data based on key value database |
| CN109344162A (en) * | 2018-09-14 | 2019-02-15 | 北京京东金融科技控股有限公司 | Business data processing method and device |
| CN109542330A (en) * | 2017-09-21 | 2019-03-29 | 杭州海康威视系统技术有限公司 | Date storage method, data query method and device |
| CN110908967A (en) * | 2019-11-13 | 2020-03-24 | 北京云途腾科技有限责任公司 | Method, device, equipment and computer readable medium for storing log |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10614132B2 (en) * | 2013-04-30 | 2020-04-07 | Splunk Inc. | GUI-triggered processing of performance data and log data from an information technology environment |
| US20160063021A1 (en) * | 2014-08-28 | 2016-03-03 | Futurewei Technologies, Inc. | Metadata Index Search in a File System |
| US11030176B2 (en) * | 2016-07-01 | 2021-06-08 | Ebay Inc. | Distributed storage of metadata for large binary data |
| US10713271B2 (en) * | 2017-02-10 | 2020-07-14 | Oracle International Corporation | Querying distributed log data using virtual fields defined in query strings |
| US10872075B2 (en) * | 2017-07-20 | 2020-12-22 | Sap Se | Indexing flexible multi-representation storages for time series data |
-
2020
- 2020-03-25 CN CN202010218079.0A patent/CN111400322B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105260404A (en) * | 2015-09-22 | 2016-01-20 | 北京百度网讯科技有限公司 | Time sequence data storage method and apparatus |
| CN106648446A (en) * | 2015-10-30 | 2017-05-10 | 阿里巴巴集团控股有限公司 | Time series data storage method and apparatus, and electronic device |
| CN107315533A (en) * | 2016-04-26 | 2017-11-03 | 杭州海康威视数字技术股份有限公司 | A kind of date storage method and device |
| CN109542330A (en) * | 2017-09-21 | 2019-03-29 | 杭州海康威视系统技术有限公司 | Date storage method, data query method and device |
| CN108256088A (en) * | 2018-01-23 | 2018-07-06 | 清华大学 | A kind of storage method and system of the time series data based on key value database |
| CN109344162A (en) * | 2018-09-14 | 2019-02-15 | 北京京东金融科技控股有限公司 | Business data processing method and device |
| CN110908967A (en) * | 2019-11-13 | 2020-03-24 | 北京云途腾科技有限责任公司 | Method, device, equipment and computer readable medium for storing log |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111400322A (en) | 2020-07-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110781373B (en) | List updating method and device, readable medium and electronic equipment | |
| CN111679990B (en) | Test data generation method and device, readable medium and electronic equipment | |
| CN111309747A (en) | Data synchronization method, system and device | |
| CN111596991A (en) | Interactive operation execution method and device and electronic equipment | |
| CN111432001B (en) | Method, apparatus, electronic device and computer readable medium for jumping scenes | |
| CN111813685B (en) | Automatic test method and device | |
| CN111311358B (en) | Information processing method and device and electronic equipment | |
| CN113220281A (en) | Information generation method and device, terminal equipment and storage medium | |
| CN111798251A (en) | Verification method and device of house source data and electronic equipment | |
| CN110619101B (en) | Method and apparatus for processing information | |
| CN111596992A (en) | Navigation bar display method and device and electronic equipment | |
| CN111787041A (en) | Method and apparatus for processing data | |
| CN111400322B (en) | Method, apparatus, electronic device and medium for storing data | |
| CN111460020B (en) | Method, device, electronic equipment and medium for resolving message | |
| CN115292635A (en) | Page floor data display method, device, equipment and computer readable medium | |
| CN114116247A (en) | Redis-based message processing method, device, system, server and medium | |
| CN112685075A (en) | Gray scale distribution method and device, electronic equipment and computer readable medium | |
| CN111597439A (en) | Information processing method and device and electronic equipment | |
| CN111338621B (en) | Method, apparatus, electronic device and computer readable medium for displaying data | |
| CN111581305B (en) | Feature processing method, device, electronic equipment and medium | |
| CN113472565B (en) | Method, apparatus, device and computer readable medium for expanding server function | |
| CN112084440B (en) | Data verification method, device, electronic equipment and computer readable medium | |
| CN111399902B (en) | Client source file processing method and device, readable medium and electronic equipment | |
| CN112468849B (en) | Method, apparatus, electronic device and medium for video information transmission | |
| CN111291199B (en) | Information query method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Douyin Vision Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: Tiktok vision (Beijing) Co.,Ltd. Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Tiktok vision (Beijing) Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: BEIJING BYTEDANCE NETWORK TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |