CN113568936B - Real-time stream data storage method, device and terminal equipment - Google Patents
Real-time stream data storage method, device and terminal equipment Download PDFInfo
- Publication number
- CN113568936B CN113568936B CN202110868989.8A CN202110868989A CN113568936B CN 113568936 B CN113568936 B CN 113568936B CN 202110868989 A CN202110868989 A CN 202110868989A CN 113568936 B CN113568936 B CN 113568936B
- Authority
- CN
- China
- Prior art keywords
- message
- output
- message group
- target
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the invention discloses a real-time stream data storage method, a device and terminal equipment. One embodiment of the method comprises the following steps: receiving a message set sent by target terminal equipment, wherein in response to sending the message set, the target terminal equipment deletes the message set, and the message set comprises a real-time consumer group and a re-running consumer group; setting each real-time message data switch identifier and the re-run data switch identifier in the message set to represent an on state; generating a target message group set according to the message set; generating an output message group set based on the target message group set; updating the target message group set; and sending the output message group set to the target terminal equipment, wherein the target terminal equipment stores and displays the output message group set. According to the embodiment, the real-time stream data is updated according to the data switch identifier, invalid data is deleted in time, the data storage space is saved, and the storage efficiency of the real-time stream data is improved.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to a real-time stream data storage method, a device and terminal equipment.
Background
With the continuous development of big data technology, the importance of the cleaning, processing and processing of massive data for big data storage and subsequent big data application is increasingly developed. The data in the database is extracted, cleaned and converted to integrate scattered, scattered and non-uniform data, and the query performance can be improved by modeling the data in the analysis database. And merging data from multiple sources, constructing complex connections and aggregations to create visual icons of the data so that users can more intuitively obtain the value of the data. Real-time streaming data processing is an important application scenario for large data processing, where the data source is uninterrupted in real-time, and the response time of the user is required to be real-time. The real-time stream data processing includes a data generation and collection phase, a transmission and analysis processing phase, and a storage pair out-of-service phase.
However, in the process of processing and storing real-time stream data, there are often the following technical problems:
in the prior art, when the real-time stream calculation repair problem is processed, if the historical data and the data after real-time repair cannot be effectively isolated, the phenomenon of data reversing covered by the repeatedly calculated historical data exists in the current data. The method of constructing two sets of data systems simultaneously is adopted, the consistency between the two sets of data systems is very difficult to maintain, the storage resources of the server are wasted, and the storage efficiency of the real-time stream data is affected.
Disclosure of Invention
The disclosure is in part intended to introduce concepts in a simplified form that are further described below in the detailed description. The disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
Some embodiments of the present disclosure propose a real-time stream data storage method, apparatus, terminal device to solve one or more of the technical problems mentioned in the background section above.
In a first aspect, some embodiments of the present disclosure provide a real-time streaming data storage method, the method comprising: receiving a message set sent by target terminal equipment, wherein in response to sending the message set, the target terminal equipment deletes the message set, the message set comprises a real-time consumer group and a rerun consumer group, the real-time consumer group comprises a first number of real-time message sets, the rerun consumer group comprises a second number of rerun message sets, the real-time message sets comprise a real-time message information set and a real-time message data switch identifier, and the rerun message sets comprise a rerun message information set and a rerun message data switch identifier; setting each real-time message data switch identifier and the re-run data switch identifier in the message set to represent an on state; generating a target message group set according to the message set, wherein the target message group set comprises a third number of target message groups, and the target message groups comprise attribute tags, data switch identifiers, target data, target messages, target message water lines and object unique keys; generating an output message group set based on the target message group set; updating the target message group set; and sending the output message group set to the target terminal equipment, wherein the target terminal equipment stores and displays the output message group set.
In a second aspect, some embodiments of the present disclosure provide a real-time streaming data storage device, the device comprising: a receiving unit configured to receive a set of message sets sent from a target terminal device, wherein in response to sending the set of message sets, the target terminal device deletes the set of message sets, the set of message sets including a real-time consumer group and a re-run consumer group, the real-time consumer group including a first number of real-time message sets, the re-run consumer group including a second number of re-run message sets, the real-time message sets including a real-time message information set and a real-time message data switch identifier, the re-run message sets including a re-run message information set and a re-run message data switch identifier; a setting unit configured to set each of the real-time message data switch identifier and the rewashing data switch identifier in the message set to characterize an on state; a first generating unit configured to generate a set of target message groups according to the set of message sets, wherein the set of target message groups includes a third number of target message groups, and the target message groups include an attribute tag, a data switch identifier, target data, a target message water line, and an object unique key; a second generation unit configured to generate an output message group set based on the target message group set; an updating unit configured to update the target message group set; and a transmitting unit configured to transmit the output message group set to the target terminal device, wherein the target terminal device stores and displays the output message group set.
In a third aspect, some embodiments of the present disclosure provide a terminal device, including: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method as in any of the first aspects.
The above embodiments of the present disclosure have the following advantages: according to the real-time stream data storage method of some embodiments of the present disclosure, real-time stream data can be updated according to the data switch identifier, invalid data can be deleted in time, the data storage space is saved, and the storage efficiency of the real-time stream data is improved. In particular, the inventors have found that the reason for the waste and inefficiency of current real-time streaming data storage resources is: in the prior art, when the real-time stream calculation repair problem is processed, if the historical data and the data after real-time repair cannot be effectively isolated, the phenomenon of data reversing covered by the repeatedly calculated historical data exists in the current data. The method of constructing two sets of data systems simultaneously is adopted, the consistency between the two sets of data systems is very difficult to maintain, the storage resources of the server are wasted, and the storage efficiency of the real-time stream data is affected. Based on this, first, some embodiments of the present disclosure receive a set of message sets sent by a target terminal device. Wherein, in response to sending the message set, the target terminal device deletes the message set. The message set includes a real-time consumer group and a re-run consumer group, the real-time consumer group includes a first number of real-time message sets, the re-run consumer group includes a second number of re-run message sets, the real-time message sets include a real-time message information set and a real-time message data switch identifier, and the re-run message sets include a re-run message information set and a re-run message data switch identifier. Second, each of the real-time message data switch identifiers and the rewash data switch identifiers in the message set collection are set to characterize an on state. Again, a set of target message groups is generated from the set of message sets. Wherein the set of target message groups comprises a third number of target message groups, the target message groups comprising an attribute tag, a data switch identifier, a target message water line, an object unique key. Then, based on the set of target message groups, an output set of message groups is generated, and the set of target message groups is updated. And finally, sending the output message group set to the target terminal equipment. Wherein the target terminal device stores and displays the set of output message groups. The method divides the message set into a real-time consumer group and a rerun consumer group, thereby distinguishing historical real-time data and real-time data needing rerun repair number, needing no building two sets of systems to store respectively, and saving data storage resources. In addition, the data switch identifier is used for controlling the data modification state, and the data is updated according to the unique key of the object, so that the phenomenon of data reversing is avoided, and the storage efficiency of real-time stream data is improved.
Drawings
The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.
FIG. 1 is an architecture diagram of an exemplary system in which some embodiments of the present disclosure may be applied;
FIG. 2 is a flow chart of some embodiments of a real-time streaming data storage method according to the present disclosure;
FIG. 3 is a flow chart of some embodiments of a real-time streaming data storage device according to the present disclosure;
fig. 4 is a schematic structural diagram of a terminal device suitable for use in implementing some embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete. It should be understood that the drawings and embodiments of the present disclosure are for illustration purposes only and are not intended to limit the scope of the present disclosure.
It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings. Embodiments of the present disclosure and features of embodiments may be combined with each other without conflict.
It should be noted that the terms "first," "second," and the like in this disclosure are merely used to distinguish between different devices, modules, or units and are not used to define an order or interdependence of functions performed by the devices, modules, or units.
It should be noted that references to "one", "a plurality" and "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be understood as "one or more" unless the context clearly indicates otherwise.
The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates an exemplary system architecture 100 to which embodiments of the real-time streaming data storage methods of the present disclosure may be applied.
As shown in fig. 1, a system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as an information processing application, a data storage application, a data analysis application, and the like, may be installed on the terminal devices 101, 102, 103.
The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various terminal devices with display screens including, but not limited to, smartphones, tablets, laptop and desktop computers, and the like. When the terminal apparatuses 101, 102, 103 are software, they can be installed in the above-listed terminal apparatuses. Which may be implemented as multiple software or software modules (e.g., to provide for collection of inputs as a message set, etc.), or as a single software or software module. The present invention is not particularly limited herein.
The server 105 may be a server that provides various services, such as a server that stores a set of message sets input by the terminal devices 101, 102, 103, and the like. The server may process the received set of message sets and feed back the processing result (e.g., the set of output message sets) to the terminal device.
It should be noted that, the real-time stream data storage method provided by the embodiment of the present disclosure may be executed by the server 105 or the terminal device.
It should be noted that, the local server 105 may also directly store the message set, and the server 105 may directly extract the local message set to obtain the similarity index after processing, where the exemplary system architecture 100 may not include the terminal devices 101, 102, 103 and the network 104.
It should also be noted that the real-time stream data storage application may also be installed in the terminal devices 101, 102, 103, in which case the processing method may also be performed by the terminal devices 101, 102, 103. At this point, the exemplary system architecture 100 may also not include the server 105 and the network 104.
The server 105 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster formed by a plurality of servers, or as a single server. When the server is software, it may be implemented as a plurality of software or software modules (e.g., to provide a real-time streaming data storage service), or as a single software or software module. The present invention is not particularly limited herein.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to fig. 2, a flow 200 of some embodiments of a real-time streaming data storage method according to the present disclosure is shown. The real-time stream data storage method comprises the following steps:
In some embodiments, an executing body of the real-time streaming data storage method (e.g., a server shown in fig. 1) receives a set of message sets sent from a target terminal device. The target terminal device may be a "mobile phone" or a "computer". The target terminal device stores a message set in advance. In response to sending the set of message sets, the target terminal device deletes the locally stored set of message sets. The message set collection includes a real-time consumer group and a rerun consumer group. The real-time consumer group includes a first number of real-time message sets and the re-run consumer group includes a second number of re-run message sets. The real-time message set includes a real-time message information set and a real-time message data switch identifier, and the re-run message set includes a re-run message information set and a re-run message data switch identifier. In particular, a real-time message data switch identifier and a rewinds message data switch identifier may be used to characterize the validity of the data.
In some embodiments, the execution body sets each of the real-time message data switch identifier and the rewash data switch identifier in the message set to characterize an on state. Specifically, the real-time message information set corresponding to the real-time message data switch identifier in the on state is valid data. And the information set of the re-running message corresponding to the re-running message data switch identifier in the on state is effective data.
In some embodiments, the executing body generates the target message group set from the message set. Wherein the set of target message groups comprises a third number of target message groups. The target message group includes an attribute tag, a data switch identifier, target data, a target message water line, and an object unique key. The attribute tags include real-time attribute tags and rerun attribute tags. In particular, attribute tags may be used to characterize the attributes of the target data. The real-time attribute tag may be a tag that characterizes an attribute of the real-time data, and the rerun attribute tag may be a tag that characterizes an attribute of the data of the rerun wait number. Specifically, real-time data and rerun data can be distinguished by attribute tags. In particular, a real-time attribute tag may be determined for each set of real-time message information in the real-time consumer group. The heavy race attribute tags may be determined for respective sets of real-time message information in the heavy race consumer group. Specifically, an object unique key is a field that can be used to uniquely identify a target message in a database. The target message may be a message object to be processed in a database. In particular, messages refer to a way in which software objects interact and communicate with each other.
In some embodiments, the execution body generates the set of output message groups based on the set of target message groups. Optionally, the executing body divides the target message group set into a real-time target message group set and a rerun target message group set according to the attribute tag. Specifically, a target message group with an attribute tag being a real-time attribute tag is determined as a real-time target message group, so as to obtain a real-time target message group set. And determining the target message group of the attribute tag for the heavy running attribute tag as a heavy running target message group so as to obtain a heavy running target message group set. And carrying out aggregation processing on the rerun target message group set to obtain an output rerun message group set. Specifically, the output rerun message group in the output rerun message group set is a message group obtained after aggregation processing. Specifically, the aggregation process may be a statistical analysis process performed on the data. An output message group set is generated based on the output re-run message group set. Optionally, for each output rerun message group in the output rerun message group set, searching the output rerun message group at the back end of the predetermined state according to the object unique key of the output rerun message group. Wherein the predetermined state backend is a data structure that stores and manages a predetermined set of message groups. Specifically, the object unique key of the output re-running message group is searched in the predetermined state back end.
And in response to finding the output rerun message group, determining the water line of the output rerun message group found in the back end of the predetermined state as a standard water line. In particular, the water line may be a data storage capacity threshold in a database. And generating a filtering index of each output rerun message group in the output rerun message group set according to the standard water line so as to obtain a filtering index set. Optionally, for each output re-running message group in the set of output re-running message groups, in response to the target message water line of the output re-running message group not being less than the standard water line, setting a filter index of the output re-running message group to be token 1, and saving the output re-running message group to a predetermined state backend. For each output re-run message group in the set of output re-run message groups, determining the output re-run message group as an output message group in response to the filter indicator representation 1 of the output re-run message group.
Optionally, in response to not finding the output re-run message group, determining the output re-run message group as an output message group.
In some embodiments, the executing entity updates the set of target message groups. Optionally, the data switch identifier corresponding to each real-time attribute tag in the target message group set is set to represent the closed state. And setting each re-running attribute tag in the target message group set as a real-time attribute tag. Specifically, the attribute tag value of each rerun target message group in the rerun target message group set in the target message group set is changed from the rerun attribute tag to the real-time attribute tag,
and step 206, transmitting the output message group set to the target terminal equipment.
In some embodiments, the executing body sends the set of output message groups to the target terminal device. Wherein the target terminal device stores and displays the set of output message groups. Specifically, the target terminal device stores the output message group set locally. The target terminal device may display the set of outgoing message groups for viewing and recall by the user.
One embodiment, as illustrated in fig. 2, has the following beneficial effects: receiving a message set sent by target terminal equipment, wherein in response to sending the message set, the target terminal equipment deletes the message set, and the message set comprises a real-time consumer group and a re-running consumer group; setting each real-time message data switch identifier and the re-run data switch identifier in the message set to represent an on state; generating a target message group set according to the message set; generating an output message group set based on the target message group set; updating the target message group set; and sending the output message group set to the target terminal equipment, wherein the target terminal equipment stores and displays the output message group set. According to the embodiment, the real-time stream data is updated according to the data switch identifier, invalid data is deleted in time, the data storage space is saved, and the storage efficiency of the real-time stream data is improved.
With further reference to fig. 3, as an implementation of the method described above for each of the above figures, the present disclosure provides embodiments of a real-time streaming data storage device, which correspond to those described above for fig. 2, and which are particularly applicable in various terminal devices.
As shown in fig. 3, some embodiments provide a real-time streaming data storage device 300, the device comprising: a receiving unit 301, a setting unit 302, a first generating unit 303, a second generating unit 304, an updating unit 305, and a transmitting unit 306. Wherein the receiving unit 301 is configured to receive a set of message sets sent by a target terminal device. Wherein, in response to sending the message set, the target terminal device deletes the message set, the message set including a real-time consumer group and a rerun consumer group, the real-time consumer group including a first number of real-time message sets, the rerun consumer group including a second number of rerun message sets. The real-time message set includes a real-time message information set and a real-time message data switch identifier, and the re-run message set includes a re-run message information set and a re-run message data switch identifier. A setting unit 302 is configured to set each of the real-time message data switch identifier and the re-run data switch identifier in the message set to characterize an on state. The first generating unit 303 is configured to generate a set of target message groups from the set of message groups. Wherein the set of target message groups comprises a third number of target message groups, the target message groups comprising an attribute tag, a data switch identifier, target data, a target message water line, an object unique key. The second generating unit 304 is configured to generate an output message group set based on the target message group set. An updating unit 305 configured to update the set of target message groups. A sending unit 306 configured to send the set of output message groups to the target terminal device. Wherein the target terminal device stores and displays the set of output message groups.
It will be appreciated that the elements described in the apparatus 300 correspond to the various steps in the method described with reference to fig. 2. Thus, the operations, features and resulting benefits described above with respect to the method are equally applicable to the apparatus 300 and the units contained therein, and are not described in detail herein.
Referring now to FIG. 4, there is illustrated a schematic diagram of a computer system 400 suitable for use in implementing the terminal device of an embodiment of the present disclosure. The terminal device shown in fig. 4 is only one example, and should not impose any limitation on the functions and scope of use of the embodiments of the present disclosure.
As shown in fig. 4, the computer system 400 includes a central processing unit (CPU, central Processing Unit) 401, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 402 or a program loaded from a storage section 406 into a random access Memory (RAM, random Access Memory) 403. In RAM403, various programs and data required for the operation of system 400 are also stored. The CPU 401, ROM 402, and RAM403 are connected to each other by a bus 404. An Input/Output (I/O) interface 405 is also connected to bus 404.
The following components are connected to the I/O interface 405: a storage section 406 including a hard disk and the like; and a communication section 407 including a network interface card such as a LAN (local area network ) card, a modem, or the like. The communication section 407 performs communication processing via a network such as the internet. The driver 408 is also connected to the I/O interface 405 as needed. Removable media 409, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is installed as needed in drive 408, so that a computer program read therefrom is installed as needed in storage section 406.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 407, and/or installed from the removable medium 409. The above-described functions defined in the method of the present disclosure are performed when the computer program is executed by a Central Processing Unit (CPU) 401. It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the C-language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by those skilled in the art that the scope of the invention referred to in this disclosure is not limited to the specific combination of features described above, but encompasses other embodiments in which features described above or their equivalents may be combined in any way without departing from the spirit of the invention. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).
Claims (6)
1. A real-time streaming data storage method, comprising:
receiving a message set sent by a target terminal device, wherein in response to sending the message set, the target terminal device deletes the message set, the message set comprises a real-time consumer group and a rerun consumer group, the real-time consumer group comprises a first number of real-time message sets, the rerun consumer group comprises a second number of rerun message sets, the real-time message sets comprise a real-time message information set and a real-time message data switch identifier, and the rerun message sets comprise a rerun message information set and a rerun message data switch identifier;
setting each real-time message data switch identifier and the rewashing data switch identifier in the message set to represent an on state;
generating a target message group set according to the message set, wherein the target message group set comprises a third number of target message groups, the target message groups comprise attribute tags, data switch identifiers, target data, target messages, target message water level lines and object unique keys, the water level lines are data storage capacity limit values in a database, and the attribute tags comprise real-time attribute tags and re-running attribute tags;
generating an output message group set based on the target message group set, wherein the generating the output message group set based on the target message group set includes:
dividing the target message group set into a real-time target message group set and a rerun target message group set according to the attribute tags;
the re-running target message group set is subjected to aggregation treatment to obtain an output re-running message group set;
generating an output message group set based on the output rerun message group set, wherein the generating the output message group set based on the output rerun message group set includes:
searching the output rerun message group at the back end of a predetermined state according to the object unique key of the output rerun message group for each output rerun message group in the output rerun message group set;
in response to finding the output rerun message group, determining a water line of the output rerun message group found in a predetermined state back end as a standard water line;
generating a filtering index of each output rerun message group in the output rerun message group set according to the standard water line so as to obtain a filtering index set;
determining, for each output re-run message group in the set of output re-run message groups, the output re-run message group as an output message group in response to a preset characterization of a filtering indicator of the output re-run message group;
updating the set of target message groups, wherein the updating the set of target message groups comprises:
setting a data switch identifier corresponding to each real-time attribute tag in the target message group set to be in a representation closing state;
setting each re-running attribute tag in the target message group set as a real-time attribute tag;
and sending the output message group set to the target terminal equipment, wherein the target terminal equipment stores and displays the output message group set.
2. The method of claim 1, wherein the generating an output message group set based on the output re-run message group set further comprises:
and in response to not finding the output rerun message group, determining the output rerun message group as an output message group.
3. The method of claim 2, wherein the generating a filter indicator of the set of output rerun messages from the standard water line comprises:
setting a filtering index of the output rerun message group as a preset representation in response to the target message water line of the output rerun message group being not less than a standard water line;
the output re-run message group is saved to a predetermined status back end.
4. A method according to claim 3, wherein the predetermined status backend is a data structure storing and managing a predetermined set of message groups.
5. A real-time streaming data storage device, comprising:
a receiving unit configured to receive a set of message sets sent from a target terminal device, wherein in response to sending the set of message sets, the target terminal device deletes the set of message sets, the set of message sets including a real-time consumer group and a re-run consumer group, the real-time consumer group including a first number of real-time message sets, the re-run consumer group including a second number of re-run message sets, the real-time message sets including a real-time message information set and a real-time message data switch identifier, the re-run message sets including a re-run message information set and a re-run message data switch identifier;
a setting unit configured to set each of the real-time message data switch identifier and the rewinds data switch identifier in the message set to characterize an on state;
a first generating unit configured to generate a set of target message groups according to the set of message sets, wherein the set of target message groups includes a third number of target message groups, the target message groups include an attribute tag, a data switch identifier, target data, a target message water line, an object unique key, the water line is a data storage capacity limit value in a database, and the attribute tag includes a real-time attribute tag and a re-run attribute tag;
a second generating unit configured to generate an output message group set based on the target message group set, wherein the generating the output message group set based on the target message group set includes:
dividing the target message group set into a real-time target message group set and a rerun target message group set according to the attribute tags;
the re-running target message group set is subjected to aggregation treatment to obtain an output re-running message group set;
generating an output message group set based on the output rerun message group set, wherein the generating the output message group set based on the output rerun message group set includes:
searching the output rerun message group at the back end of a predetermined state according to the object unique key of the output rerun message group for each output rerun message group in the output rerun message group set;
in response to finding the output rerun message group, determining a water line of the output rerun message group found in a predetermined state back end as a standard water line;
generating a filtering index of each output rerun message group in the output rerun message group set according to the standard water line so as to obtain a filtering index set;
determining, for each output re-run message group in the set of output re-run message groups, the output re-run message group as an output message group in response to a preset characterization of a filtering indicator of the output re-run message group;
an updating unit configured to update the target message group set, wherein the updating the target message group set includes:
setting a data switch identifier corresponding to each real-time attribute tag in the target message group set to be in a representation closing state;
setting each re-running attribute tag in the target message group set as a real-time attribute tag;
and a transmitting unit configured to transmit the output message group set to the target terminal device, wherein the target terminal device stores and displays the output message group set.
6. A terminal device, comprising:
one or more processors;
a storage device having one or more programs stored thereon;
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110868989.8A CN113568936B (en) | 2021-07-30 | 2021-07-30 | Real-time stream data storage method, device and terminal equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110868989.8A CN113568936B (en) | 2021-07-30 | 2021-07-30 | Real-time stream data storage method, device and terminal equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113568936A CN113568936A (en) | 2021-10-29 |
| CN113568936B true CN113568936B (en) | 2023-06-13 |
Family
ID=78169278
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110868989.8A Active CN113568936B (en) | 2021-07-30 | 2021-07-30 | Real-time stream data storage method, device and terminal equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113568936B (en) |
Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2000045555A1 (en) * | 1999-01-28 | 2000-08-03 | Data Race, Inc. | Escape sequence protocol for multiplexing real-time data with non-real-time data |
| JP2006272000A (en) * | 2006-07-10 | 2006-10-12 | Nintendo Co Ltd | Additional data distribution system and upload system |
| CN101807073A (en) * | 2010-04-20 | 2010-08-18 | 杭州和利时自动化有限公司 | Historical data processing method and device of distributed control system |
| CN102571933A (en) * | 2011-12-22 | 2012-07-11 | 中国电子科技集团公司第十五研究所 | Reliable message transmission method |
| CN105337973A (en) * | 2015-10-27 | 2016-02-17 | 上海钢富电子商务有限公司 | Message exchange method and system |
| CN105554262A (en) * | 2015-12-11 | 2016-05-04 | 小米科技有限责任公司 | Method and device for operation mode switching |
| CN107133342A (en) * | 2017-05-16 | 2017-09-05 | 广州舜飞信息科技有限公司 | A kind of IndexR real-time data analysis storehouse |
| CN108508850A (en) * | 2017-02-28 | 2018-09-07 | Sap欧洲公司 | Manufacturing process data collects and analyzes |
| WO2018223800A1 (en) * | 2017-06-08 | 2018-12-13 | 腾讯科技（深圳）有限公司 | Data processing method and device, storage medium, and electronic device |
| CN109542873A (en) * | 2018-10-26 | 2019-03-29 | 深圳点猫科技有限公司 | A kind of language based on programming realizes the method and electronic equipment of race historical data again |
| CN110019262A (en) * | 2017-10-10 | 2019-07-16 | 北京国双科技有限公司 | Data-updating method and device |
| CN110673807A (en) * | 2019-08-20 | 2020-01-10 | 华为技术有限公司 | Data display method, terminal and data display system |
| WO2020018421A1 (en) * | 2018-07-15 | 2020-01-23 | Shyft Power Solutions, Inc. | Apparatuses, methods and systems for intelligent and flexible transfer switches |
| CN111614546A (en) * | 2020-04-30 | 2020-09-01 | 南京南瑞信息通信科技有限公司 | Real-time information interaction method and device for OA system and storage medium |
| CN111787603A (en) * | 2020-07-10 | 2020-10-16 | 深圳传音通讯有限公司 | Data transmission method of mobile terminal, mobile terminal and storage medium |
| CN111984826A (en) * | 2020-09-02 | 2020-11-24 | 河北幸福消费金融股份有限公司 | XML-based data automatic storage method, system, device and storage medium |
| CN112527484A (en) * | 2020-12-17 | 2021-03-19 | 平安银行股份有限公司 | Workflow breakpoint continuous running method and device, computer equipment and readable storage medium |
| CN112612775A (en) * | 2020-12-17 | 2021-04-06 | 平安消费金融有限公司 | Data storage method and device, computer equipment and storage medium |
| CN112632127A (en) * | 2020-12-29 | 2021-04-09 | 国华卫星数据科技有限公司 | Data processing method for real-time data acquisition and time sequence of equipment operation |
| CN112954407A (en) * | 2021-04-23 | 2021-06-11 | 中国建设银行股份有限公司 | Bullet screen interaction method, device, equipment and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9621473B2 (en) * | 2004-08-18 | 2017-04-11 | Open Text Sa Ulc | Method and system for sending data |
| US9081501B2 (en) * | 2010-01-08 | 2015-07-14 | International Business Machines Corporation | Multi-petascale highly efficient parallel supercomputer |
| US8819156B2 (en) * | 2011-03-11 | 2014-08-26 | James Robert Miner | Systems and methods for message collection |
-
2021
- 2021-07-30 CN CN202110868989.8A patent/CN113568936B/en active Active
Patent Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2000045555A1 (en) * | 1999-01-28 | 2000-08-03 | Data Race, Inc. | Escape sequence protocol for multiplexing real-time data with non-real-time data |
| JP2006272000A (en) * | 2006-07-10 | 2006-10-12 | Nintendo Co Ltd | Additional data distribution system and upload system |
| CN101807073A (en) * | 2010-04-20 | 2010-08-18 | 杭州和利时自动化有限公司 | Historical data processing method and device of distributed control system |
| CN102571933A (en) * | 2011-12-22 | 2012-07-11 | 中国电子科技集团公司第十五研究所 | Reliable message transmission method |
| CN105337973A (en) * | 2015-10-27 | 2016-02-17 | 上海钢富电子商务有限公司 | Message exchange method and system |
| CN105554262A (en) * | 2015-12-11 | 2016-05-04 | 小米科技有限责任公司 | Method and device for operation mode switching |
| CN108508850A (en) * | 2017-02-28 | 2018-09-07 | Sap欧洲公司 | Manufacturing process data collects and analyzes |
| CN107133342A (en) * | 2017-05-16 | 2017-09-05 | 广州舜飞信息科技有限公司 | A kind of IndexR real-time data analysis storehouse |
| WO2018223800A1 (en) * | 2017-06-08 | 2018-12-13 | 腾讯科技（深圳）有限公司 | Data processing method and device, storage medium, and electronic device |
| CN110019262A (en) * | 2017-10-10 | 2019-07-16 | 北京国双科技有限公司 | Data-updating method and device |
| WO2020018421A1 (en) * | 2018-07-15 | 2020-01-23 | Shyft Power Solutions, Inc. | Apparatuses, methods and systems for intelligent and flexible transfer switches |
| CN109542873A (en) * | 2018-10-26 | 2019-03-29 | 深圳点猫科技有限公司 | A kind of language based on programming realizes the method and electronic equipment of race historical data again |
| CN110673807A (en) * | 2019-08-20 | 2020-01-10 | 华为技术有限公司 | Data display method, terminal and data display system |
| CN111614546A (en) * | 2020-04-30 | 2020-09-01 | 南京南瑞信息通信科技有限公司 | Real-time information interaction method and device for OA system and storage medium |
| CN111787603A (en) * | 2020-07-10 | 2020-10-16 | 深圳传音通讯有限公司 | Data transmission method of mobile terminal, mobile terminal and storage medium |
| CN111984826A (en) * | 2020-09-02 | 2020-11-24 | 河北幸福消费金融股份有限公司 | XML-based data automatic storage method, system, device and storage medium |
| CN112527484A (en) * | 2020-12-17 | 2021-03-19 | 平安银行股份有限公司 | Workflow breakpoint continuous running method and device, computer equipment and readable storage medium |
| CN112612775A (en) * | 2020-12-17 | 2021-04-06 | 平安消费金融有限公司 | Data storage method and device, computer equipment and storage medium |
| CN112632127A (en) * | 2020-12-29 | 2021-04-09 | 国华卫星数据科技有限公司 | Data processing method for real-time data acquisition and time sequence of equipment operation |
| CN112954407A (en) * | 2021-04-23 | 2021-06-11 | 中国建设银行股份有限公司 | Bullet screen interaction method, device, equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| PETRENKO S.A.BIG DATA TECHNOLOGIES FOR CYBERSECURITY.《SECURE INFORMATION TECHNOLOGIES: 2017》.2017,107-111. * |
| 周瑞卿.面向商业银行数据仓库的数据质量管理关键技术研究.《中国优秀硕士学位论文全文数据库 信息科技》.2016,I138-909. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113568936A (en) | 2021-10-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109145023B (en) | Method and apparatus for processing data | |
| CN110390493B (en) | Task management method and device, storage medium and electronic equipment | |
| CN110019263B (en) | Information storage method and device | |
| CN110909521A (en) | Synchronous processing method and device for online document information and electronic equipment | |
| CN109597810B (en) | Task segmentation method, device, medium and electronic equipment | |
| CN113760982A (en) | Data processing method and device | |
| CN113568936B (en) | Real-time stream data storage method, device and terminal equipment | |
| CN112948138A (en) | Method and device for processing message | |
| CN115086347B (en) | Data synchronization method, device, electronic equipment and storage medium | |
| CN106933449B (en) | Icon processing method and device | |
| CN112887426B (en) | Information stream pushing method and device, electronic equipment and storage medium | |
| CN112507676B (en) | Method and device for generating energy report, electronic equipment and computer readable medium | |
| CN114490718A (en) | Data output method, data output device, electronic equipment and computer readable medium | |
| CN114691684A (en) | Data display method, device and system | |
| CN114612212A (en) | Business processing method, device and system based on risk control | |
| CN113282455A (en) | Monitoring processing method and device | |
| CN112115154A (en) | Data processing and data query method, device, equipment and computer readable medium | |
| CN110262756B (en) | Method and device for caching data | |
| CN112988857A (en) | Service data processing method and device | |
| CN117170821B (en) | Service processing method, device, electronic equipment and computer readable medium | |
| CN115705326A (en) | Real-time data storage method and device and terminal equipment | |
| CN115378991B (en) | Message processing method, device, equipment and computer storage medium | |
| CN117692672B (en) | Snapshot-based video information sending method and device, electronic equipment and medium | |
| CN116760717B (en) | Communication path query method, device, electronic equipment and computer readable medium | |
| CN115495518B (en) | Method and device for generating chart |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |