CN111400322A - Method, apparatus, electronic device, and medium for storing data - Google Patents
Method, apparatus, electronic device, and medium for storing data Download PDFInfo
- Publication number
- CN111400322A CN111400322A CN202010218079.0A CN202010218079A CN111400322A CN 111400322 A CN111400322 A CN 111400322A CN 202010218079 A CN202010218079 A CN 202010218079A CN 111400322 A CN111400322 A CN 111400322A
- Authority
- CN
- China
- Prior art keywords
- data
- information
- target data
- piece
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2477—Temporal data queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Abstract
Embodiments of the present disclosure disclose methods, apparatuses, electronic devices, and media for storing data. One embodiment of the method comprises: for each data interval in the target number of data intervals, generating an identifier of the data interval; acquiring data related information of each piece of target data in at least one piece of target data, wherein the data related information comprises time information of the target data; and for each piece of target data, storing the target data based on the data-related information of the target data and the identification group. The embodiment avoids the simultaneous storage of a large amount of data because the data-related information of the target data and the identification group are considered first when the data is stored.
Description
Technical Field
Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a method, an apparatus, an electronic device, and a medium for storing data.
Background
Large internet services usually generate huge amounts of logs, and there is often a need to retrieve the required data by scanning the huge amounts of logs. For processing mass data, a large data processing framework is generally used. Because the amount of data is too large, the data cannot be directly stored in a traditional relational database.
Disclosure of Invention
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
It is an object of some embodiments of the present disclosure to propose an improved method, apparatus, electronic device and medium for storing data to solve the technical problems mentioned in the background section above.
In a first aspect, some embodiments of the present disclosure provide a method for storing data, the method comprising: for each data interval in the target number of data intervals, generating an identifier of the data interval; acquiring data related information of each piece of target data in at least one piece of target data, wherein the data related information comprises time information of the target data; and for each piece of target data, storing the target data based on the data related information of the target data and the identification group.
In a second aspect, some embodiments of the present disclosure provide a method for querying data, the method comprising: acquiring query information corresponding to data to be queried based on category information of the data, wherein the query information comprises: partition main key information and query time range information of the data; determining the identifier of a data interval corresponding to the data based on the partition primary key information; determining a message file block where the data is located based on the query time range information and the identifier; and screening target data from the message file block based on the query information.
In a third aspect, some embodiments of the present disclosure provide an apparatus for storing data, the apparatus comprising: a generation unit configured to generate, for each of a target number of data intervals, an identification of the data interval; an acquisition unit configured to acquire data-related information of each of at least one piece of target data, the data-related information including time information of the target data; and a storage unit configured to store, for each piece of target data, the target data based on the data-related information of the target data and the identification group.
In a fourth aspect, some embodiments of the present disclosure provide an apparatus for querying data, the apparatus comprising: an obtaining unit configured to obtain query information corresponding to data to be queried based on category information of the data, wherein the query information includes: partition main key information and query time range information of the data; a first determining unit configured to determine an identifier of a data section corresponding to the data based on the partition primary key information; a second determining unit configured to determine a message file block in which the data is located based on the query time range information and the identifier; and the screening unit is configured to screen target data from the message file block based on the query information.
In a fifth aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon which, when executed by one or more processors, cause the one or more processors to implement a method as in any one of the first aspects.
In a sixth aspect, some embodiments of the disclosure provide a computer readable medium having a computer program stored thereon, wherein the program when executed by a processor implements the method as in any one of the first aspect.
One of the above-described various embodiments of the present disclosure has the following advantageous effects: firstly, generating an identifier of each data interval in a target number of data intervals; then, acquiring data related information of each piece of target data in at least one piece of target data, wherein the data related information comprises time information of the target data; and finally, storing each piece of target data based on the data related information of the target data and the identification group. Since the data related information of the target data and the identification group are considered first during data storage, simultaneous storage of a large amount of data is avoided.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale.
FIG. 1 is a schematic diagram of one application scenario of a method for storing data, in accordance with some embodiments of the present disclosure;
FIG. 2 is a flow diagram of some embodiments of a method for storing data according to the present disclosure;
FIG. 3 is a schematic diagram of one application scenario of a method for querying data, in accordance with some embodiments of the present disclosure;
FIG. 4 is a flow diagram of some embodiments of a method for querying data according to the present disclosure;
FIG. 5 is a schematic block diagram of some embodiments of an apparatus for a method of storing data according to the present disclosure;
FIG. 6 is a schematic block diagram of some embodiments of an apparatus for a method of querying data according to the present disclosure;
FIG. 7 is a schematic block diagram of a computer system suitable for use in implementing an electronic device of an embodiment of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings. The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 is a schematic diagram of one application scenario of a method for storing data according to some embodiments of the present disclosure.
As shown in the application scenario of fig. 1, the executing agent of the method for storing data may be the server 101. First, the server 101 can obtain the identifier 105 (shown as "1" in the figure), the identifier 106 (shown as "2" in the figure) corresponding to the data interval 103, and the identifier 107 (shown as "3" in the figure) corresponding to the data interval 104 based on the target number (shown as 3) of the data intervals 102 and 104. Next, the server 101 can acquire at least one piece (3 pieces shown in the figure) of data-related information 111 (shown in the figure as "9 hours 40 minutes 00 seconds"), data-related information 112 (shown in the figure as "9 hours 41 minutes 30 seconds") of the target data 109, and data-related information 113 (shown in the figure as "9 hours 46 minutes 00 seconds") of the target data 110. Wherein the data-related information includes: and time information corresponding to each piece of target data in the at least one piece of target data. Here, the time information may be a time when the server 101 acquires the target data. Finally, the server 101 may store the at least one piece of target data 102 and 105 based on the data-related information 111 and 113 and the identifier 105 and 107.
It will be appreciated that the method for generating pictures may be performed by the server 101, or by other devices, or by various software programs. The server 101 may be, for example, various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, desktop computers, and the like. Furthermore, the execution body may also be embodied as a server, software, or the like. When the execution subject is software, the software can be installed in the electronic device listed above. It may be implemented, for example, as multiple software or software modules to provide distributed services, or as a single software or software module. And is not particularly limited herein.
It should be understood that the number of servers in fig. 1 is merely illustrative. There may be any number of servers, as desired for implementation.
With continued reference to fig. 2, a flow 200 of some embodiments of a method for storing data in accordance with the present disclosure is shown. The method for storing data comprises the following steps:
In some embodiments, the executing agent of the method for storing data (e.g., server 101 shown in fig. 1) may generate, for each of a target number of data intervals, an identification of the data interval. The identifiers of the target number of data intervals may be sequentially generated according to a natural number order. The target number may be predetermined or may be set manually. The data section may be a section for storing a part of data. The identifier may be a partition number added to the data section. The partition number may be an integer value, for example, 0,1,2,3 … n. As an example, when the target number is 3, the corresponding identifiers of each data interval may be "1", "2", "3", respectively.
In some embodiments, the execution subject may obtain data-related information of each of at least one piece of target data, where the data-related information includes time information of the target data. As an example, the at least one piece of target data may be data sent by a user watching a live broadcast in a live broadcast scene, or may be data sent by a user who is watching a live broadcast. The time information may be a time when the user transmits data, and for example, the time information may be 19 hours, 29 minutes, and 32 seconds.
In some optional implementations of some embodiments, the data related information further includes: and the category information corresponds to each piece of target data in the at least one piece of target data. The category information includes: and sending the message class and pulling the message class. As an example, the target data in the delivered message class may be data sent by a user who is live in a live scene. The pull message class may be data viewed by a user watching a live broadcast in a live broadcast scene.
In some optional implementations of some embodiments, the generating, for each data interval of the target number of data intervals, an identifier of the data interval includes: obtaining a partition main key corresponding to each piece of target data in the at least one piece of target data based on the category information corresponding to each piece of target data in the at least one piece of target data; as an example, when the category information of the target data is the down message category, the partition primary key of the target data may be a room number or a device number. The room number may be a number of a live room, for example, the number may be "0001". The device number may be a device number of a device viewing a live broadcast, and for example, the device number may be "a 1001". And obtaining the identifier corresponding to each data interval based on the target number and the partition main key corresponding to each piece of target data in the at least one piece of target data. As an example, the above-mentioned identification may be calculated by the following formula: PartitionId HashFunc (key, N). Wherein "PartitionId" represents the identity corresponding to each data interval. "key" represents the partition primary key corresponding to the target data. "N" represents the above target number. "HashFunc" represents a score function for generating a corresponding identification for each data interval based on a given "key" and "N".
In some embodiments, the execution subject may store, for each piece of target data, the target data based on the data-related information of the target data and the identification group. As an example, the target data may be stored in a data interval corresponding to the identifier in the identifier group based on the data related information of each piece of target data and the identifier group.
In some optional implementations of some embodiments, the storing, for each piece of target data, the target data based on the data-related information of the target data and the identification group includes: obtaining at least one message file block respectively storing at least one piece of target data according to the time information; the time information may be a time when the user transmits data, and for example, the time information may be 19 hours, 29 minutes, and 32 seconds. The message file block may be for storing target data. Here, one message file block may be used to store 5 minutes of target data. And respectively adding file names to the at least one message file block based on the time information and the identification. As an example, when the category information of the target data is the down message category, the above-mentioned file name may be named in the following manner: msg _ time _ partition _ log. Wherein, the "msg" may indicate that the category information of the target data is the down message category. "time" indicates the time when the message file block holds the target data, for example, if the message file block holds the target data for 5 minutes, the "time" may be "19 minutes and 30 minutes". "partition" indicates the identification of the data section corresponding to the target data.
In some optional implementations of some embodiments, the storing, for each piece of target data, the target data based on the data-related information of the target data and the identification group includes: and in response to the fact that the category information of the at least one piece of target data is the pull message category, storing the message primary key of the at least one piece of target data based on the data related information and the identification. The message primary key may be a primary key corresponding to the target data, for example, the primary key may be "0, 1,2, …, n". As an example, when the category information of the target data is a pull message category, the message primary key of the at least one piece of target data is stored in at least one message file block based on the time information of the target data. The file names of the above message file blocks may be named in the following manner: ack _ time _ partition _ log. Where "ack" may indicate that the category information of the target data is a pull message class. "time" indicates the time when the message file block holds the message primary key of the target data, for example, if the message file block holds target data for 5 minutes, the "time" may be "19 hours and 30 minutes". "partition" indicates the identification of the data section corresponding to the target data.
As can be seen from the above example, first, for each data interval of the target number of data intervals, an identifier of the data interval is generated; then, acquiring data related information of each piece of target data in at least one piece of target data, wherein the data related information comprises time information of the target data; and finally, storing each piece of target data based on the data related information of the target data and the identification group. Since the data related information of the target data and the identification group are considered first during data storage, simultaneous storage of a large amount of data is avoided.
FIG. 3 is a schematic diagram of one application scenario of a method for querying data, in accordance with some embodiments of the present disclosure.
As shown in the application scenario of fig. 3, the execution subject of the method for querying data may be the server 301. First, the server 301 may obtain query information 303 corresponding to data to be queried based on category information 302 of the data, where the query information 303 includes: partition primary key information 304 and inquiry time range information 305 of the above data; the category information 302 includes: and sending the message class and pulling the message class. The query time range information 305 includes: the start time and the end time may be, for example, "" 19 hours 28 minutes "," 19 hours 31 minutes ". Next, based on the partition primary key information 304, determining an identifier 306 of a data section corresponding to the data; then, based on the query time range information 305 and the identifier 306, determining a message file block 307 in which the data is located; finally, target data 308 is screened from the message file block 307 based on the query information 303.
It will be appreciated that the method for generating pictures may be performed by the server 301, or may be performed by other devices, or may be performed by various software programs. The server 301 may be, for example, various electronic devices having a display screen, including but not limited to a smart phone, a tablet computer, an e-book reader, a laptop portable computer, a desktop computer, and the like. Furthermore, the execution body may also be embodied as a server, software, or the like. When the execution subject is software, the software can be installed in the electronic device listed above. It may be implemented, for example, as multiple software or software modules to provide distributed services, or as a single software or software module. And is not particularly limited herein.
It should be understood that the number of servers in fig. 3 is merely illustrative. There may be any number of servers, as desired for implementation.
With continued reference to fig. 4, a flow 400 of some embodiments of a method for querying data in accordance with the present disclosure is shown. The method for querying data comprises the following steps:
In some embodiments, an execution subject (e.g., the server 301 shown in fig. 3) may first obtain query information corresponding to data to be queried based on category information of the data. The category information includes: and sending the message class and pulling the message class. The query information may include at least one of: the partition main key information and the query time range information of the data. As an example, when the category information of the data is a down message category, the corresponding query information includes: the partition main key information and the query time range information of the data. The query time range information includes: a start time and an end time. For example, the query time range information may be "19 hours 28 minutes", "19 hours 31 minutes".
In some embodiments, the execution main body may determine, based on the partition primary key information, an identifier of a data interval corresponding to the data. As an example, the identification of the data interval may be obtained by the following formula: PartitionId ═ HashFunc (roommd, N). Wherein "PartitionId" represents the identity of the data interval described above. "RoomId" indicates partition primary key information of the above data. "N" indicates the number of data intervals.
In some embodiments, the execution subject may first determine a message file block in which the data is located based on the query time range information obtained in step 401 and the identifier obtained in step 402. The message file block may be for storing target data. As an example, the file name of the message file block may be obtained by the following formula: msg _ [ time ] _ partitionid. Wherein, the "msg" may indicate that the category information of the target data is the down message category. "time" indicates the time when the message file block holds the target data, for example, if the message file block holds the target data for 5 minutes, the "time" may be "19 minutes and 30 minutes". "partition" indicates the identification of the data section corresponding to the target data. As an example, the above-mentioned inquiry time range information may be "19 hours 28 minutes" and "19 hours 31 minutes". The time represented by "time" needs to be after the start time and before the end time in the query time range information.
And step 404, screening target data from the message file block based on the query information.
In some embodiments, the execution body may screen the message file block for target data based on the query information obtained in step 401.
Some embodiments of the present disclosure disclose a method for querying data, first, based on category information of data to be queried, obtaining query information corresponding to the data; secondly, determining the identification of a data interval corresponding to the data based on the partition main key information; then, based on the query time range information and the identification, determining a message file block where the data is located; and finally, screening target data from the message file block based on the query information. The corresponding message file block can be found firstly when the data is inquired, and the data is inquired from the message file block, so that the size of the file to be scanned is reduced, and the inquiry speed is accelerated.
With further reference to fig. 5, as an implementation of the above-described method for the above-described figures, the present disclosure provides some embodiments of an apparatus for storing data, which correspond to those of the method embodiments described above for fig. 2, and which may be applied in particular to various electronic devices.
As shown in fig. 5, an apparatus 500 for storing data of some embodiments includes: a generating unit 501, an acquiring unit 502, and a storing unit 503. Wherein, the generating unit 501 is configured to generate, for each data interval of the target number of data intervals, an identifier of the data interval; an obtaining unit 502 configured to obtain data-related information of each of at least one piece of target data, the data-related information including time information of the target data; a storage unit 503 configured to store the target data for each piece of target data based on the data-related information of the target data and the identification group.
In some optional implementations of some embodiments, the data related information further includes: the category information corresponding to each piece of target data in the at least one piece of target data includes: and sending the message class and pulling the message class.
In some optional implementations of some embodiments, the generating unit 501 in the apparatus 500 for storing data is further configured to: obtaining a partition main key corresponding to each piece of target data in the at least one piece of target data based on the category information corresponding to each piece of target data in the at least one piece of target data; and obtaining the identifier corresponding to each data interval based on the target number and the partition main key corresponding to each piece of target data in the at least one piece of target data.
In some optional implementations of some embodiments, the storage unit 503 in the apparatus for storing data 500 is further configured to: obtaining at least one message file block respectively storing at least one piece of target data according to the time information; and respectively adding file names to the at least one message file block based on the time information and the identification.
In some optional implementations of some embodiments, the storage unit 503 in the apparatus for storing data 500 is further configured to: and in response to the fact that the category information of the at least one piece of target data is the pull message category, storing the message primary key of the at least one piece of target data based on the data related information and the identification.
It will be understood that the elements described in the apparatus 500 correspond to various steps in the method described with reference to fig. 2. Thus, the operations, features and resulting advantages described above with respect to the method are also applicable to the apparatus 500 and the units included therein, and are not described herein again.
With further reference to fig. 6, as an implementation of the above-described methods for the above-described figures, the present disclosure provides some embodiments of an apparatus for querying data, which correspond to those of the method embodiments described above for fig. 4, and which may be applied in various electronic devices in particular.
As shown in fig. 6, an apparatus 600 for querying data of some embodiments includes: an acquisition unit 601, a first determination unit 602, a second determination unit 603, and a filtering unit 604. The obtaining unit 601 is configured to obtain query information corresponding to data to be queried based on category information of the data, where the query information includes: partition main key information and query time range information of the data; a first determining unit 602 configured to determine an identifier of a data section corresponding to the data based on the partition primary key information; a second determining unit 603 configured to determine a message file block in which the data is located based on the query time range information and the identifier; a screening unit 604 configured to screen the target data from the message file block based on the query information.
It will be understood that the elements described in the apparatus 600 correspond to various steps in the method described with reference to fig. 4. Thus, the operations, features and resulting advantages described above with respect to the method are also applicable to the apparatus 600 and the units included therein, and are not described herein again.
Referring now to fig. 7, a schematic diagram of an electronic device (e.g., the server of fig. 1) 700 suitable for use in implementing some embodiments of the present disclosure is shown. The terminal device in some embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. The terminal device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the use range of the embodiments of the present disclosure.
As shown in fig. 7, electronic device 700 may include a processing means (e.g., central processing unit, graphics processor, etc.) 701 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from storage 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data necessary for the operation of the electronic apparatus 700 are also stored. The processing device 701, the ROM 702, and the RAM703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Generally, input devices 706 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc., output devices 707 including, for example, a liquid crystal display (L CD), speaker, vibrator, etc., storage devices 708 including, for example, a memory card, etc., and communication devices 709. communication devices 709 may allow electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data although FIG. 7 illustrates electronic device 700 with various means, it is to be understood that not all of the illustrated means are required to be implemented or provided, more or less means may be implemented instead, or provided, each block illustrated in FIG. 7 may represent one means, or a plurality of means as desired.
In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In some such embodiments, the computer program may be downloaded and installed from a network via communications means 709, or may be installed from storage 708, or may be installed from ROM 702. The computer program, when executed by the processing device 701, performs the above-described functions defined in the methods of some embodiments of the present disclosure.
It should be noted that the computer readable medium described above in some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText transfer protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communications network). examples of communications networks include local area networks ("L AN"), wide area networks ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: for each data interval in the target number of data intervals, generating an identifier of the data interval; acquiring data related information of each piece of target data in at least one piece of target data, wherein the data related information comprises time information of the target data; and for each piece of target data, storing the target data based on the data related information of the target data and the identification group.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including AN object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language, or similar programming languages.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in some embodiments of the present disclosure may be implemented by software, and may also be implemented by hardware. The described units may also be provided in a processor, and may be described as: a processor includes a generation unit, an acquisition unit, and a storage unit. Where the names of these units do not in some cases constitute a limitation on the unit itself, for example, the generating unit may also be described as "a unit that generates an identification of the data interval for each of a target number of data intervals".
For example, without limitation, exemplary types of hardware logic that may be used include Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CP L D), and so forth.
In accordance with one or more embodiments of the present disclosure, there is provided a method for storing data, including: for each data interval in the target number of data intervals, generating an identifier of the data interval; acquiring data related information of each piece of target data in at least one piece of target data, wherein the data related information comprises time information of the target data; and for each piece of target data, storing the target data based on the data related information of the target data and the identification group.
According to one or more embodiments of the present disclosure, the data related information further includes category information of the target data, where the category information includes: and sending the message class and pulling the message class.
According to one or more embodiments of the present disclosure, the generating, for each data interval of the target number of data intervals, an identifier of the data interval includes: obtaining a partition main key corresponding to each piece of target data in the at least one piece of target data based on the category information corresponding to each piece of target data in the at least one piece of target data; and obtaining the identifier corresponding to each data interval based on the target number and the partition main key corresponding to each piece of target data in the at least one piece of target data.
According to one or more embodiments of the present disclosure, the storing, for each piece of target data, the target data based on the data-related information of the target data and the identification group includes: obtaining at least one message file block respectively storing at least one piece of target data according to the time information; and respectively adding file names to the at least one message file block based on the time information and the identification.
According to one or more embodiments of the present disclosure, the storing, for each piece of target data, the target data based on the data-related information of the target data and the identification group includes: and in response to the fact that the category information of the at least one piece of target data is the pull message category, storing the message primary key of the at least one piece of target data based on the data related information and the identification.
In accordance with one or more embodiments of the present disclosure, there is provided a method for querying data, including: acquiring query information corresponding to data to be queried based on category information of the data, wherein the query information comprises: partition main key information and query time range information of the data; determining the identifier of a data interval corresponding to the data based on the partition primary key information; determining a message file block where the data is located based on the query time range information and the identifier; and screening target data from the message file block based on the query information.
According to one or more embodiments of the present disclosure, there is provided an apparatus for storing data, including: a generation unit configured to generate, for each of a target number of data intervals, an identification of the data interval; an acquisition unit configured to acquire data-related information of each of at least one piece of target data, the data-related information including time information of the target data; and a storage unit configured to store, for each piece of target data, the target data based on the data-related information of the target data and the identification group.
According to one or more embodiments of the present disclosure, the data related information further includes category information of the target data, where the category information includes: and sending the message class and pulling the message class.
According to one or more embodiments of the present disclosure, the generating unit in the above apparatus is further configured to: obtaining a partition main key corresponding to each piece of target data in the at least one piece of target data based on the category information corresponding to each piece of target data in the at least one piece of target data; and obtaining the identifier corresponding to each data interval based on the target number and the partition main key corresponding to each piece of target data in the at least one piece of target data.
According to one or more embodiments of the present disclosure, the storage unit in the above apparatus is further configured to: obtaining at least one message file block respectively storing at least one piece of target data according to the time information; and respectively adding file names to the at least one message file block based on the time information and the identification.
According to one or more embodiments of the present disclosure, the storage unit in the above apparatus is further configured to: and in response to the fact that the category information of the at least one piece of target data is the pull message category, storing the message primary key of the at least one piece of target data based on the data related information and the identification.
According to one or more embodiments of the present disclosure, there is provided an apparatus for querying data, including: an obtaining unit configured to obtain query information corresponding to data to be queried based on category information of the data, wherein the query information includes: partition main key information and query time range information of the data; a first determining unit configured to determine an identifier of a data section corresponding to the data based on the partition primary key information; a second determining unit configured to determine a message file block in which the data is located based on the query time range information and the identifier; and the screening unit is configured to screen target data from the message file block based on the query information.
According to one or more embodiments of the present disclosure, there is provided an electronic device including: one or more processors; a storage device having one or more programs stored thereon which, when executed by one or more processors, cause the one or more processors to implement a method as described in any of the embodiments above.
According to one or more embodiments of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the method as described in any of the embodiments above.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.
Claims (10)
1. A method for storing data, comprising:
for each data interval in the target number of data intervals, generating an identifier of the data interval;
acquiring data related information of each piece of target data in at least one piece of target data, wherein the data related information comprises time information of the target data;
and for each piece of target data, storing the target data based on the data related information of the target data and the identification group.
2. The method of claim 1, wherein the data-related information further comprises category information of the target data, wherein the category information comprises: and sending the message class and pulling the message class.
3. The method of claim 2, wherein said generating, for each of a target number of data intervals, an identification of the data interval comprises:
obtaining a partition main key corresponding to each piece of target data in the at least one piece of target data based on the category information corresponding to each piece of target data in the at least one piece of target data;
and obtaining the identifier corresponding to each data interval based on the target number and the partition main key corresponding to each piece of target data in the at least one piece of target data.
4. The method of claim 1, wherein the storing, for each piece of target data, the target data based on the data-related information of the target data and the identification group comprises:
obtaining at least one message file block respectively storing at least one piece of target data according to the time information;
and respectively adding file names to the at least one message file block based on the time information and the identification.
5. The method of claim 3, wherein said storing the at least one piece of target data based on the data-related information and the identification comprises:
and in response to the category information of the at least one piece of target data being the pull message category, storing the message primary key of the at least one piece of target data based on the data related information and the identification.
6. A method for querying data, comprising:
acquiring query information corresponding to data to be queried based on category information of the data, wherein the query information comprises: partition main key information and query time range information of the data;
determining the identifier of a data interval corresponding to the data based on the partition primary key information;
determining a message file block where the data is located based on the query time range information and the identification;
and screening target data from the message file block based on the query information.
7. An apparatus for storing data, comprising:
a generation unit configured to generate, for each of a target number of data intervals, an identification of the data interval;
an acquisition unit configured to acquire data-related information of each of at least one piece of target data, the data-related information including time information of the target data;
a storage unit configured to store, for each piece of target data, the target data based on the data-related information of the target data and the identification group.
8. An apparatus for querying data, comprising:
an obtaining unit configured to obtain query information corresponding to data to be queried based on category information of the data, wherein the query information includes: partition main key information and query time range information of the data;
a first determination unit configured to determine an identifier of a data section to which the data corresponds based on the partition primary key information;
a second determining unit configured to determine a message file block in which the data is located based on the query time range information and the identifier;
a screening unit configured to screen target data from the message file block based on the query information.
9. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
10. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010218079.0A CN111400322B (en) | 2020-03-25 | 2020-03-25 | Method, apparatus, electronic device and medium for storing data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010218079.0A CN111400322B (en) | 2020-03-25 | 2020-03-25 | Method, apparatus, electronic device and medium for storing data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400322A true CN111400322A (en) | 2020-07-10 |
| CN111400322B CN111400322B (en) | 2023-10-03 |
Family
ID=71429255
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010218079.0A Active CN111400322B (en) | 2020-03-25 | 2020-03-25 | Method, apparatus, electronic device and medium for storing data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400322B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105260404A (en) * | 2015-09-22 | 2016-01-20 | 北京百度网讯科技有限公司 | Time sequence data storage method and apparatus |
| US20160063021A1 (en) * | 2014-08-28 | 2016-03-03 | Futurewei Technologies, Inc. | Metadata Index Search in a File System |
| CN106648446A (en) * | 2015-10-30 | 2017-05-10 | 阿里巴巴集团控股有限公司 | Time series data storage method and apparatus, and electronic device |
| US20170169134A1 (en) * | 2013-04-30 | 2017-06-15 | Splunk Inc. | Gui-triggered processing of performance data and log data from an information technology environment |
| CN107315533A (en) * | 2016-04-26 | 2017-11-03 | 杭州海康威视数字技术股份有限公司 | A kind of date storage method and device |
| US20180004745A1 (en) * | 2016-07-01 | 2018-01-04 | Ebay Inc. | Distributed storage of metadata for large binary data |
| CN108256088A (en) * | 2018-01-23 | 2018-07-06 | 清华大学 | A kind of storage method and system of the time series data based on key value database |
| US20180232425A1 (en) * | 2017-02-10 | 2018-08-16 | Oracle International Corporation | Systems and methods for distributed log data querying using virtual fields defined in query strings |
| US20190026329A1 (en) * | 2017-07-20 | 2019-01-24 | Sap Se | Indexing flexible multi-representation storages for time series data |
| CN109344162A (en) * | 2018-09-14 | 2019-02-15 | 北京京东金融科技控股有限公司 | Business data processing method and device |
| CN109542330A (en) * | 2017-09-21 | 2019-03-29 | 杭州海康威视系统技术有限公司 | Date storage method, data query method and device |
| CN110908967A (en) * | 2019-11-13 | 2020-03-24 | 北京云途腾科技有限责任公司 | Method, device, equipment and computer readable medium for storing log |
-
2020
- 2020-03-25 CN CN202010218079.0A patent/CN111400322B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170169134A1 (en) * | 2013-04-30 | 2017-06-15 | Splunk Inc. | Gui-triggered processing of performance data and log data from an information technology environment |
| US20160063021A1 (en) * | 2014-08-28 | 2016-03-03 | Futurewei Technologies, Inc. | Metadata Index Search in a File System |
| CN105260404A (en) * | 2015-09-22 | 2016-01-20 | 北京百度网讯科技有限公司 | Time sequence data storage method and apparatus |
| CN106648446A (en) * | 2015-10-30 | 2017-05-10 | 阿里巴巴集团控股有限公司 | Time series data storage method and apparatus, and electronic device |
| CN107315533A (en) * | 2016-04-26 | 2017-11-03 | 杭州海康威视数字技术股份有限公司 | A kind of date storage method and device |
| US20180004745A1 (en) * | 2016-07-01 | 2018-01-04 | Ebay Inc. | Distributed storage of metadata for large binary data |
| US20180232425A1 (en) * | 2017-02-10 | 2018-08-16 | Oracle International Corporation | Systems and methods for distributed log data querying using virtual fields defined in query strings |
| US20190026329A1 (en) * | 2017-07-20 | 2019-01-24 | Sap Se | Indexing flexible multi-representation storages for time series data |
| CN109542330A (en) * | 2017-09-21 | 2019-03-29 | 杭州海康威视系统技术有限公司 | Date storage method, data query method and device |
| CN108256088A (en) * | 2018-01-23 | 2018-07-06 | 清华大学 | A kind of storage method and system of the time series data based on key value database |
| CN109344162A (en) * | 2018-09-14 | 2019-02-15 | 北京京东金融科技控股有限公司 | Business data processing method and device |
| CN110908967A (en) * | 2019-11-13 | 2020-03-24 | 北京云途腾科技有限责任公司 | Method, device, equipment and computer readable medium for storing log |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111400322B (en) | 2023-10-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110781373B (en) | List updating method and device, readable medium and electronic equipment | |
| CN110909521B (en) | Online document information synchronous processing method and device and electronic equipment | |
| CN112311656B (en) | Message aggregation and display method and device, electronic equipment and computer readable medium | |
| CN111679990A (en) | Test data generation method and device, readable medium and electronic equipment | |
| CN111309747A (en) | Data synchronization method, system and device | |
| CN111460049A (en) | Content sharing method and device, electronic equipment and computer readable storage medium | |
| CN111163336A (en) | Video resource pushing method and device, electronic equipment and computer readable medium | |
| CN111596992B (en) | Navigation bar display method and device and electronic equipment | |
| CN112367241A (en) | Message generation and message transmission method, device, equipment and computer readable medium | |
| CN112148744A (en) | Page display method and device, electronic equipment and computer readable medium | |
| CN110619101B (en) | Method and apparatus for processing information | |
| CN115292635A (en) | Page floor data display method, device, equipment and computer readable medium | |
| CN114490718A (en) | Data output method, data output device, electronic equipment and computer readable medium | |
| CN114428925A (en) | Page rendering method and device, electronic equipment and computer readable medium | |
| CN111400322B (en) | Method, apparatus, electronic device and medium for storing data | |
| CN111460020A (en) | Method, apparatus, electronic device and medium for parsing message | |
| CN111294657A (en) | Information processing method and device | |
| CN111367592A (en) | Information processing method and device | |
| CN111294321B (en) | Information processing method and device | |
| CN113472565B (en) | Method, apparatus, device and computer readable medium for expanding server function | |
| CN111930229B (en) | Man-machine interaction method and device and electronic equipment | |
| CN110633324B (en) | Method, apparatus, electronic device and computer readable medium for synchronizing data | |
| CN111291199B (en) | Information query method and device | |
| CN113626664A (en) | House resource screening information processing method, device, equipment and computer readable medium | |
| CN115391605A (en) | Data query method, device, equipment, computer readable medium and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Douyin Vision Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: Tiktok vision (Beijing) Co.,Ltd. Address after: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant after: Tiktok vision (Beijing) Co.,Ltd. Address before: 100041 B-0035, 2 floor, 3 building, 30 Shixing street, Shijingshan District, Beijing. Applicant before: BEIJING BYTEDANCE NETWORK TECHNOLOGY Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |