CN110851435B

CN110851435B - Data storage method and device

Info

Publication number: CN110851435B
Application number: CN201810878972.9A
Authority: CN
Inventors: 阮彬彬; 郭峰; 李森
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2018-08-03
Filing date: 2018-08-03
Publication date: 2022-02-11
Anticipated expiration: 2038-08-03
Also published as: CN110851435A

Abstract

The application relates to a method and a device for storing data, and belongs to the field of monitoring. The method comprises the following steps: acquiring data to be stored from Kafka, wherein the data to be stored comprises user identification; obtaining corresponding Dag from a corresponding relation between a user identifier and a directed acyclic graph Dag according to the user identifier, where Dag is established according to a writing order of multiple databases, the writing order of the multiple databases is configured by a user corresponding to the user identifier, a node in Dag is used for storing an identifier of a database, and at least one child node of the node is used for storing an identifier of a database whose writing order is located behind the database; and storing the data to be stored into the databases according to the writing sequence of the databases included in the Dag. The method and the device can store the data into the database according to the database storage sequence required by the user.

Description

Data storage method and device

Technical Field

The present application relates to the field of computer technologies, and in particular, to a method and an apparatus for storing data.

Background

Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action flow data in a consumer-scale website. Kafka receives a large amount of data, so it is necessary to store the data in Kafka in a database.

There is a need to store data in Kafka into multiple databases, such as Hbase, ES (elastic search) system, and Structured Query Language (SQL) databases. Currently, the data in Kafka are stored in each database respectively according to a preset storage sequence of each database. For example, the preset storage sequence of each database is Hbase, the ES system and the SQL database, the data is stored in the Hbase first when the data is stored, the data is stored in the ES system after the data is stored, and the data is stored in the SQL database again after the data is stored.

The preset storage sequence of each database may not be the storage sequence required by the user, so that the current storage method cannot realize the storage of the data in the database according to the storage sequence required by the user.

Disclosure of Invention

In order to store data into a database according to a database storage sequence required by a user, the embodiment of the application provides a method and a device for storing data. The technical scheme is as follows:

in a first aspect, an embodiment of the present application provides a method for storing data, where the method includes:

acquiring data to be stored from Kafka, wherein the data to be stored comprises user identification;

obtaining corresponding Dag from a corresponding relation between a user identifier and a directed acyclic graph Dag according to the user identifier, where Dag is established according to a writing order of multiple databases, the writing order of the multiple databases is configured by a user corresponding to the user identifier, a node in Dag is used for storing an identifier of a database, and at least one child node of the node is used for storing an identifier of a database whose writing order is located behind the database;

and storing the data to be stored into the databases according to the writing sequence of the databases included in the Dag.

Optionally, the acquiring data to be stored from Kafka includes:

converting the format of the data in the Kafka into data in a preset format, and caching each converted data in a cache queue;

and dequeuing data from the tail of the buffer queue as the data to be stored.

Optionally, the data to be stored is written according to the writing sequence of the databases included in the Dag

Storing data into each database, including:

reading an identification of a database from a root node of the Dag;

storing the data to be stored in a database corresponding to the identifier of the database;

reading identifiers of n databases from n child nodes of the root node, wherein n is an integer greater than or equal to 1;

and storing the data to be stored in the n databases corresponding to the identifiers of the n databases.

Optionally, after the storing the data to be stored in the n databases corresponding to the identifiers of the n databases, the method further includes:

when m child nodes exist in a target node, the target node is any one of the n child nodes, m is an integer greater than or equal to 1, and the identifiers of m databases are read from the m child nodes;

and storing the data to be stored in the m databases corresponding to the identifiers of the m databases.

Optionally, before obtaining Dag corresponding to the corresponding relationship between the user identifier and the directed acyclic graph Dag according to the user identifier, the method further includes:

according to the writing sequence establishment Dag of the multiple databases configured by the user, the user identification of the user and the Dag correspondence are stored in the correspondence relationship between the user identification and Dag.

In a second aspect, an embodiment of the present application provides an apparatus for storing data, where the apparatus includes:

the device comprises a first acquisition module, a second acquisition module and a storage module, wherein the first acquisition module is used for acquiring data to be stored from Kafka, and the data to be stored comprises user identification;

a second obtaining module, configured to obtain Dag corresponding to the user identifier from a corresponding relationship between the user identifier and a directed acyclic graph Dag according to the user identifier, where the Dag is established according to a writing order of multiple databases, the writing order of the multiple databases is configured by a user corresponding to the user identifier, a node in the Dag is used to store an identifier of a database, and at least one child node of the node is used to store an identifier of a database whose writing order is located after the database;

and the storage module is used for storing the data to be stored into each database according to the writing sequence of each database included in the Dag.

Optionally, the first obtaining module is configured to:

and dequeuing data from the tail of the buffer queue as the data to be stored.

Optionally, the storage module is configured to:

reading an identification of a database from a root node of the Dag;

Optionally, the storage module is further configured to:

Optionally, the apparatus further comprises:

and the establishing module is used for establishing Dag according to the writing sequence of the plurality of databases configured by the user, and correspondingly storing the user identifier of the user and the Dag in the corresponding relation between the user identifier and Dag.

In a third aspect, embodiments of the present application provide a non-transitory computer-readable storage medium for storing a computer program, where the computer program is loaded by a processor to execute the first aspect or any optional implementation manner of the first aspect.

The technical scheme provided by the embodiment of the application can have the following beneficial effects:

and acquiring Dag corresponding to the data to be stored from the corresponding relation between the user identifier and Dag through the user identifier included in the data to be stored, and storing the data to be stored in each database according to the writing sequence of each database included in Dag. Since Dag is established according to the writing order of the user configured database corresponding to the user identifier, the data is stored in the database according to the database storage order required by the user.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

FIG. 1 is a schematic diagram of a system provided in an embodiment of the present application;

FIG. 2 is a flow chart of a method for storing data according to an embodiment of the present disclosure;

FIG. 3 is a flow chart of another method for storing data provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of a display interface provided by an embodiment of the present application;

FIG. 5 is a schematic view Dag of an embodiment of the present application;

FIG. 6 is a schematic structural diagram of an apparatus for storing data according to an embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of a terminal provided in an embodiment of the present application.

With the above figures, there are shown specific embodiments of the present application, which will be described in more detail below. These drawings and written description are not intended to limit the scope of the inventive concepts in any manner, but rather to illustrate the inventive concepts to those skilled in the art by reference to specific embodiments.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

Referring to fig. 1, an embodiment of the present application provides a storage system, including:

kafka, a control device and a plurality of storage devices, wherein each storage device comprises at least one database, and the database can be Hbase, ES, SQL and the like.

A network connection is established between the control device and the Kafka, and a network connection is established between the control device and each storage device, wherein the network connection can be a wired connection or a wireless connection.

The Kafka receives data sent by a terminal, and the terminal is a terminal corresponding to a user. The control device may retrieve each data from kafka and save the data to a database of a plurality of storage devices.

The control device may be a terminal device such as a desktop computer or a notebook computer.

Referring to fig. 2, an embodiment of the present application provides a method for storing data, where the method includes:

step 201: and acquiring data to be stored from the Kafka, wherein the data to be stored comprises user identification.

Step 202: according to the user identifier, corresponding Dag is obtained from a corresponding relationship between the user identifier and a Directed Acyclic Graph (Dag), the Dag is established according to a writing sequence of a plurality of databases, the writing sequence of the plurality of databases is configured by a user corresponding to the user identifier, a node in the Dag is used for storing the identifier of the database, and at least one child node of the node is used for storing the identifier of a database of which the writing sequence is located behind the database.

Step 203: the data to be stored is stored in the databases according to the writing order of the databases included in Dag.

In the embodiment of the present application, according to the user identifier included in the data to be stored, corresponding Dag is obtained from the correspondence between the user identifier and Dag, and according to the writing sequence of each database included in Dag, the data to be stored is stored in each database. Since Dag is established according to the writing order of the user configured database corresponding to the user identifier, the data is stored in the database according to the database storage order required by the user.

Referring to fig. 3, an embodiment of the present application provides a method for storing data, where the method may be applied to a system as shown in fig. 1, and an execution subject of the method may be a control device in the system, and the method includes:

step 301: according to the writing sequence establishment Dag of the multiple databases configured by the user, the user identification of the user and the Dag correspondence are stored in the correspondence relationship between the user identification and Dag.

The data sent by the user to Kafka is stored in multiple databases, and the user can set which databases are used to store their data, and the writing order of each database. The user may enter in their corresponding terminal the identity of the plurality of databases used to store their data and the writing order of each database.

Optionally, the terminal corresponding to the user may define a display interface as shown in fig. 4, where the display interface includes at least one row of input boxes, and each row includes two input boxes, which are a first input box and a second input box respectively; the first input box is used for inputting the identification of a database for storing data, and the second input box is used for storing the writing sequence of the database.

Optionally, the first input box may include a drop-down list including the identities of the plurality of databases, and the user may select the identity of one of the databases in the drop-down list. And when the terminal detects the selection operation of the user, acquiring the identifier of the selected database from the pull-down list, and filling the identifier of the database in the first input box. After entering the identity of the database into the first input box, the user may enter the writing order of the database into the second input box.

Optionally, when the terminal detects that the two input boxes in the last row in the display interface are both input with content, one or more rows of input boxes may be additionally displayed after the last row of input boxes on the display interface for the user to continue inputting.

The identifier of the database may be a name of the database, for example, for the Hbase database, the identifier may be a name of the database Hbase. There may be at least two databases among the plurality of databases selected by the user, which have the same writing order, that is, the user wants to store their data in parallel to the at least two databases, so as to improve the data storage efficiency.

Optionally, after the user selects the plurality of databases of which the data needs to be stored, the user may submit the confirmation command to the terminal by clicking a confirmation button in the display interface. After detecting a confirmation command triggered by clicking the confirmation button, the terminal acquires the identifier and the writing sequence of each database selected by the user from each row of input boxes of the display interface, and sends a configuration request message to the control device, wherein the configuration request message comprises the user identifier of the user and the identifier and the writing sequence of each database.

For example, referring to fig. 4, the terminal obtains, from the display interface shown in fig. 4, the identifier and writing order of the database selected by the user, which are identifier Hbase and writing order 1 of the Hbase database, identifier ES and writing order 2 of the ES database, identifier SQL and writing order 3 of the SQL database, identifier Sysbase and writing order 3 of the Sysbase database, and identifier Access and writing order 4 of the Access database, respectively; and then sending a configuration request message to the control device, wherein the configuration request message comprises a user identification UseID1 of the user, an identification Hbase and writing sequence 1 of a Hbase database, an identification ES and writing sequence 2 of the ES database, an identification SQL and writing sequence 3 of the SQL database, an identification Sysbase and writing sequence 3 of the Sysbase database and an identification Access and writing sequence 4 of the Access database.

Dag is a directed acyclic graph, wherein Dag nodes are used to store database identifiers, and for a node in Dag, the order of writing the database corresponding to the database identifier stored in the node precedes the order of writing the database corresponding to the database identifier stored in the child node of the node.

In this step, Dag may be established based on the user-configured writing order of the databases. For example, the control device receives a configuration request message sent by the terminal, and according to the writing sequence 1 of the Hbase database, the writing sequence 2 of the ES database, the writing sequence 3 of the SQL database, the writing sequence 3 of the Sysbase database, and the writing sequence 4 of the Access database included in the configuration request message, establishes a root node of Dag, the root node is used to store the identifier Hbase of the Hbase database with the writing sequence 1, establishes a child node of the root node to obtain a second level node of Dag, the second level node is used to store the identifier ES of the ES database with the writing sequence 2, establishes two child nodes of the second level node to obtain two third level nodes of Dag, one of the third level nodes is used to store the identifier SQL of the SQL database with the writing sequence 3, the other third level node is used to store the identifier Sysbase of the Sysbase database, and establishes a child node of one of the third level node to obtain a fourth level node of Dag, the fourth layer of nodes are configured to store the identifier Access of the Access database with a write order of 4, and Dag is obtained as shown in fig. 5. Then, the user identifier UserID1 included in the configuration request message and the corresponding relation between the user identifier stored correspondingly to Dag as shown in fig. 5 and Dag are included in the configuration request message.

For each user, each user may request from the control device a plurality of databases configured to store data and a writing order of the databases.

Step 302: and acquiring data to be stored from the Kafka, wherein the data to be stored comprises user identification.

For each user, the user may send data to Kafka through its corresponding terminal, where the data includes the user identification of the user. Kafka receives the data and stores the data locally.

In the step, reading data from Kafka, converting the format of the data into data in a preset format, and caching the converted data in a cache queue; then reading next data from Kafka, converting the format of the data into data with a preset format, and caching the converted data in a cache queue; the above process is repeated until all the data in Kafka is stored in the buffer queue. And then dequeuing one data from the tail of the buffering queue as data to be stored, and executing the following data storage operation.

Step 303: and acquiring corresponding Dag from the corresponding relation between the user identification and Dag according to the user identification in the data to be stored.

For example, assume that the user identifier included in the data to be stored is user id1, a corresponding Dag is obtained from the correspondence between the user identifiers and Dag according to the user identifier user id1, and the obtained Dag is Dag as shown in fig. 5.

Step 304: the data to be stored is stored in the databases according to the writing order of the databases included in Dag.

This step can be achieved by the following steps 3041 to 3044, respectively:

3041: the identity of the database is read from the root node at Dag.

3042: and storing the data to be stored in the database corresponding to the identifier of the database.

In this embodiment, for each database, the storage device in which the database is located establishes a processing unit for the database in advance, and the processing unit is responsible for storing data into the database. The processing unit corresponds to a piece of mark information, when the processing unit stores data into the database, the mark information corresponding to the processing unit is the mark information being processed, and when the processing unit finishes storing the data, the mark information corresponding to the processing unit is the mark information being processed.

In this step, a database corresponding to the identifier of the database is determined, and the flag information corresponding to the processing unit of the database is acquired from the storage device in which the database is located; if the flag information is the processing flag information, waiting and detecting the flag information in real time, when the flag information is detected to be changed into the processed flag information, sending the data to be stored to the processing unit and setting the flag information of the processing unit as the processing flag information, and storing the data to be stored to the database by the processing unit; if the flag information is the processed flag information, the data to be stored is sent to the processing unit, the flag information of the processing unit is set as the flag information being processed, and the processing unit stores the data to be stored to the database.

Optionally, when the processing unit finishes storing the data to be stored, the processing unit may set the flag information corresponding to the processing unit as the processed flag information.

Optionally, the flag information of the processing unit is detected in real time, and when it is detected that the flag information of the processing unit becomes the processed flag information, it is determined that the data to be stored has been stored in the database, and then the following steps are performed.

3043: reading identifiers of n databases from n child nodes of a root node, wherein n is an integer greater than or equal to 1; and storing the data to be stored in the n databases corresponding to the identifiers of the n databases.

For each child node in the n child nodes, reading the identifier of a database from the child node, determining the database corresponding to the identifier of the database, and acquiring the mark information corresponding to the processing unit of the database from the storage device where the database is located; if the flag information is the processing flag information, waiting and detecting the flag information in real time, when the flag information is detected to be changed into the processed flag information, sending the data to be stored to the processing unit and setting the flag information of the processing unit as the processing flag information, and storing the data to be stored to the database by the processing unit; if the flag information is the processed flag information, the data to be stored is sent to the processing unit, the flag information of the processing unit is set as the flag information being processed, and the processing unit stores the data to be stored to the database.

The above-described process of storing data is performed for each of the n child nodes. For any of the n child nodes, it is referred to as a target node for ease of explanation.

Optionally, the flag information corresponding to the processing unit of the database is detected in real time corresponding to the database corresponding to the data identifier in the target node, and when the flag information is detected to become the processed flag information, it is determined that the data to be stored has been stored in the database, and then the following steps are performed.

3044: when m child nodes exist in the target node, the target node is any one of the n child nodes, m is an integer greater than or equal to 1, the identifiers of the m databases are read from the m child nodes, and the data to be stored are stored in the m databases corresponding to the identifiers of the m databases.

For each child node in the m child nodes, reading the identifier of a database from the child node, determining the database corresponding to the identifier of the database, and acquiring the mark information corresponding to the processing unit of the database from the storage device where the database is located; if the flag information is the processing flag information, waiting and detecting the flag information in real time, when the flag information is detected to be changed into the processed flag information, sending the data to be stored to the processing unit and setting the flag information of the processing unit as the processing flag information, and storing the data to be stored to the database by the processing unit; if the flag information is the processed flag information, the data to be stored is sent to the processing unit, the flag information of the processing unit is set as the flag information being processed, and the processing unit stores the data to be stored to the database.

Optionally, for any child node in the m child nodes, determining a database corresponding to an identifier of the database stored in the child node, detecting flag information corresponding to a processing unit of the database in real time, determining that data to be stored has been stored in the database after detecting that the flag information becomes processed flag information, and then repeatedly executing 3044 with the child node as a target node.

For example, for Dag shown in fig. 5, the identifier Hbase of the database is read from the root node of Dag, the Hbase database corresponding to the identifier Hbase is determined, the flag information corresponding to the Hbase processing unit of the Hbase database is obtained, and assuming that the identifier information is the processed flag information, the data to be stored is sent to the Hbase processing unit and the flag information of the Hbase processing unit is set as the flag information being processed, and the Hbase processing unit stores the data to be stored in the Hbase database.

Detecting the mark information of the Hbase processing unit in real time, when the mark information is detected to be changed into processed mark information, acquiring a child node of the root node, namely a second layer node, reading an identification ES of a database from the second layer node, determining an ES database corresponding to the identification ES, acquiring the mark information corresponding to an ES processing unit of the ES database, and if the identification information is the processed mark information, sending the data to be stored to the ES processing unit, setting the mark information of the ES processing unit as the processing mark information, and storing the data to be stored to the ES database by the ES processing unit.

Detecting the mark information of the ES processing unit in real time, when the mark information is detected to be changed into processed mark information, acquiring two child nodes of a second layer node, namely two third layer nodes, reading an identification SQL of a database from one of the third layer nodes, determining the SQL database corresponding to the identification SQL, acquiring the mark information corresponding to the SQL processing unit of the SQL database, and if the identification information is the processed mark information, sending the data to be stored to the SQL processing unit and setting the mark information of the SQL processing unit as the flag information being processed, storing the data to be stored to the SQL database by the SQL processing unit, and detecting the mark information of the SQL processing unit in real time. And reading the identifier Sysbase of the database from another third-layer node, determining the Sysbase database corresponding to the identifier Sysbase, acquiring the flag information corresponding to the Sysbase processing unit of the Sysbase database, and if the identifier information is the processed flag information, sending the data to be stored to the Sysbase processing unit and setting the flag information of the Sysbase processing unit as the flag information being processed, so that the Sysbase processing unit stores the data to be stored to the Sysbase database, and detecting the flag information of the Sysbase processing unit in real time.

When the flag information of the SQL processing unit is detected to be changed into the processed flag information, acquiring a child node of the third-layer node, namely a fourth-layer node, reading the identifier Access of the database from the fourth-layer node, determining the Access database corresponding to the identifier Access, acquiring the flag information corresponding to the Access processing unit of the Access database, assuming that the identifier information is the processed flag information, sending the data to be stored to the Access processing unit, setting the flag information of the Access processing unit as the processing flag information, storing the data to be stored to the Access database by the Access processing unit, and detecting the flag information of the Access processing unit in real time.

In the embodiment of the present application, according to the user identifier included in the data to be stored, corresponding Dag is obtained from the correspondence between the user identifier and Dag, and according to the writing sequence of each database included in Dag, the data to be stored is stored in each database. Since Dag is established according to the writing order of the user configured database corresponding to the user identifier, the data is stored in the database according to the database storage order required by the user. In addition, when configuring the writing sequence of a plurality of databases, the user can configure the writing sequence of partial databases, so that data can be stored in the partial databases in parallel, and the data storage efficiency is improved.

The following are embodiments of the apparatus of the present application that may be used to perform embodiments of the method of the present application. For details which are not disclosed in the embodiments of the apparatus of the present application, reference is made to the embodiments of the method of the present application.

Referring to fig. 6, an embodiment of the present application provides a data storage apparatus 400, where the apparatus 400 includes:

a first obtaining module 401, configured to obtain data to be stored from Kafka, where the data to be stored includes a user identifier;

a second obtaining module 402, configured to obtain Dag corresponding to a user identifier from a corresponding relationship between the user identifier and a directed acyclic graph Dag according to the user identifier, where the Dag is established according to a writing order of multiple databases, the writing order of the multiple databases is configured by a user corresponding to the user identifier, a node in the Dag is used to store an identifier of a database, and at least one child node of the node is used to store an identifier of a database whose writing order is located after the database;

a storage module 403, configured to store the data to be stored in each database according to the writing sequence of each database included in Dag.

Optionally, the first obtaining module 401 is configured to:

and dequeuing data from the tail of the buffer queue as the data to be stored.

Optionally, the storage module 403 is configured to:

reading an identification of a database from a root node of the Dag;

Optionally, the storage module 403 is further configured to:

Optionally, the apparatus 400 further includes:

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Fig. 7 is a block diagram illustrating a terminal 500 according to an exemplary embodiment of the present invention. The terminal 500 may be a control device in the system shown in fig. 1, such as: a tablet, laptop, or desktop computer. Terminal 500 may also be referred to by other names such as user equipment, portable terminal, laptop terminal, desktop terminal, and the like.

In general, the terminal 500 includes: a processor 501 and a memory 502.

The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor 501 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, processor 501 may also include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.

Memory 502 may include one or more computer-readable storage media, which may be non-transitory. Memory 502 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 502 is used to store at least one instruction for execution by processor 501 to implement the methods provided by the method embodiments herein.

In some embodiments, the terminal 500 may further optionally include: a peripheral interface 503 and at least one peripheral. The processor 501, memory 502 and peripheral interface 503 may be connected by a bus or signal lines. Each peripheral may be connected to the peripheral interface 503 by a bus, signal line, or circuit board. Specifically, the peripheral device includes: at least one of radio frequency circuitry 504, touch screen display 505, camera 506, audio circuitry 507, positioning components 508, and power supply 509.

The peripheral interface 503 may be used to connect at least one peripheral related to I/O (Input/Output) to the processor 501 and the memory 502. In some embodiments, the processor 501, memory 502, and peripheral interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 501, the memory 502, and the peripheral interface 503 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The Radio Frequency circuit 504 is used for receiving and transmitting RF (Radio Frequency) signals, also called electromagnetic signals. The radio frequency circuitry 504 communicates with communication networks and other communication devices via electromagnetic signals. The rf circuit 504 converts an electrical signal into an electromagnetic signal to transmit, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so forth. The radio frequency circuitry 504 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocols include, but are not limited to: the world wide web, metropolitan area networks, intranets, generations of mobile communication networks (2G, 3G, 4G, and 5G), Wireless local area networks, and/or WiFi (Wireless Fidelity) networks. In some embodiments, the rf circuit 504 may further include NFC (Near Field Communication) related circuits, which are not limited in this application.

The display screen 505 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 505 is a touch display screen, the display screen 505 also has the ability to capture touch signals on or over the surface of the display screen 505. The touch signal may be input to the processor 501 as a control signal for processing. At this point, the display screen 505 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display screen 505 may be one, providing the front panel of the terminal 500; in other embodiments, the display screens 505 may be at least two, respectively disposed on different surfaces of the terminal 500 or in a folded design; in still other embodiments, the display 505 may be a flexible display disposed on a curved surface or on a folded surface of the terminal 500. Even more, the display screen 505 can be arranged in a non-rectangular irregular figure, i.e. a shaped screen. The Display screen 505 may be made of LCD (Liquid Crystal Display), OLED (organic light-Emitting Diode), and other materials.

The camera assembly 506 is used to capture images or video. Optionally, camera assembly 506 includes a front camera and a rear camera. Generally, a front camera is disposed at a front panel of the terminal, and a rear camera is disposed at a rear surface of the terminal. In some embodiments, the number of the rear cameras is at least two, and each rear camera is any one of a main camera, a depth-of-field camera, a wide-angle camera and a telephoto camera, so that the main camera and the depth-of-field camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize panoramic shooting and VR (Virtual Reality) shooting functions or other fusion shooting functions. In some embodiments, camera assembly 506 may also include a flash. The flash lamp can be a monochrome temperature flash lamp or a bicolor temperature flash lamp. The double-color-temperature flash lamp is a combination of a warm-light flash lamp and a cold-light flash lamp, and can be used for light compensation at different color temperatures.

Audio circuitry 507 may include a microphone and a speaker. The microphone is used for collecting sound waves of a user and the environment, converting the sound waves into electric signals, and inputting the electric signals to the processor 501 for processing, or inputting the electric signals to the radio frequency circuit 504 to realize voice communication. For the purpose of stereo sound collection or noise reduction, a plurality of microphones may be provided at different portions of the terminal 500. The microphone may also be an array microphone or an omni-directional pick-up microphone. The speaker is used to convert electrical signals from the processor 501 or the radio frequency circuit 504 into sound waves. The loudspeaker can be a traditional film loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, the speaker can be used for purposes such as converting an electric signal into a sound wave audible to a human being, or converting an electric signal into a sound wave inaudible to a human being to measure a distance. In some embodiments, audio circuitry 507 may also include a headphone jack.

The positioning component 508 is used for positioning the current geographic Location of the terminal 500 for navigation or LBS (Location Based Service). The Positioning component 508 may be a Positioning component based on the Global Positioning System (GPS) in the united states, the beidou System in china, or the galileo System in russia.

Power supply 509 is used to power the various components in terminal 500. The power source 509 may be alternating current, direct current, disposable or rechargeable. When power supply 509 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. The wired rechargeable battery is a battery charged through a wired line, and the wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery may also be used to support fast charge technology.

In some embodiments, terminal 500 also includes one or more sensors 510. The one or more sensors 510 include, but are not limited to: acceleration sensor 511, gyro sensor 512, pressure sensor 513, fingerprint sensor 514, optical sensor 515, and proximity sensor 516.

The acceleration sensor 511 may detect the magnitude of acceleration on three coordinate axes of the coordinate system established with the terminal 500. For example, the acceleration sensor 511 may be used to detect components of the gravitational acceleration in three coordinate axes. The processor 501 may control the touch screen 505 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 511. The acceleration sensor 511 may also be used for acquisition of motion data of a game or a user.

The gyro sensor 512 may detect a body direction and a rotation angle of the terminal 500, and the gyro sensor 512 may cooperate with the acceleration sensor 511 to acquire a 3D motion of the user on the terminal 500. The processor 501 may implement the following functions according to the data collected by the gyro sensor 512: motion sensing (such as changing the UI according to a user's tilting operation), image stabilization at the time of photographing, game control, and inertial navigation.

The pressure sensor 513 may be disposed on a side bezel of the terminal 500 and/or an underlying layer of the touch display screen 505. When the pressure sensor 513 is disposed on the side frame of the terminal 500, a user's holding signal of the terminal 500 may be detected, and the processor 501 performs left-right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 513. When the pressure sensor 513 is disposed at the lower layer of the touch display screen 505, the processor 501 controls the operability control on the UI interface according to the pressure operation of the user on the touch display screen 505. The operability control comprises at least one of a button control, a scroll bar control, an icon control and a menu control.

The fingerprint sensor 514 is used for collecting a fingerprint of the user, and the processor 501 identifies the identity of the user according to the fingerprint collected by the fingerprint sensor 514, or the fingerprint sensor 514 identifies the identity of the user according to the collected fingerprint. Upon recognizing that the user's identity is a trusted identity, the processor 501 authorizes the user to perform relevant sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings, etc. The fingerprint sensor 514 may be provided on the front, back, or side of the terminal 500. When a physical button or a vendor Logo is provided on the terminal 500, the fingerprint sensor 514 may be integrated with the physical button or the vendor Logo.

The optical sensor 515 is used to collect the ambient light intensity. In one embodiment, the processor 501 may control the display brightness of the touch display screen 505 based on the ambient light intensity collected by the optical sensor 515. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 505 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 505 is turned down. In another embodiment, processor 501 may also dynamically adjust the shooting parameters of camera head assembly 506 based on the ambient light intensity collected by optical sensor 515.

A proximity sensor 516, also referred to as a distance sensor, is typically disposed on the front panel of the terminal 500. The proximity sensor 516 is used to collect the distance between the user and the front surface of the terminal 500. In one embodiment, when the proximity sensor 516 detects that the distance between the user and the front surface of the terminal 500 gradually decreases, the processor 501 controls the touch display screen 505 to switch from the bright screen state to the dark screen state; when the proximity sensor 516 detects that the distance between the user and the front surface of the terminal 500 becomes gradually larger, the processor 501 controls the touch display screen 505 to switch from the screen-rest state to the screen-on state.

Those skilled in the art will appreciate that the configuration shown in fig. 5 is not intended to be limiting of terminal 500 and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components may be used.

Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A method of storing data, the method comprising:

2. The method of claim 1, wherein said retrieving data to be stored from Kafka comprises:

and dequeuing data from the tail of the buffer queue as the data to be stored.

3. The method as claimed in claim 1, wherein said storing said data to be stored in said databases according to a writing order of said databases included in said Dag comprises:

reading an identification of a database from a root node of the Dag;

4. The method of claim 3, wherein after storing the data to be stored in the n databases corresponding to the identities of the n databases, further comprising:

5. The method of claim 1, wherein before obtaining corresponding Dag from the correspondence of the user id with the directed acyclic graph Dag according to the user id, further comprising:

6. An apparatus for storing data, the apparatus comprising:

7. The apparatus of claim 6, wherein the first obtaining module is to:

and dequeuing data from the tail of the buffer queue as the data to be stored.

8. The apparatus of claim 6, wherein the storage module is to:

reading an identification of a database from a root node of the Dag;

9. The apparatus of claim 8, wherein the storage module is further configured to:

10. The apparatus of claim 6, wherein the apparatus further comprises: