CN110598059B - Database operation method and device - Google Patents
Database operation method and device Download PDFInfo
- Publication number
- CN110598059B CN110598059B CN201910874903.5A CN201910874903A CN110598059B CN 110598059 B CN110598059 B CN 110598059B CN 201910874903 A CN201910874903 A CN 201910874903A CN 110598059 B CN110598059 B CN 110598059B
- Authority
- CN
- China
- Prior art keywords
- data
- source file
- database
- configuration parameters
- cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9024—Graphs; Linked lists
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a database operation method and device, relates to the field of computers, and particularly relates to the field of databases. The specific implementation scheme is as follows: reading a source file according to the set configuration parameters; the source file comprises updated content of map data; and updating the corresponding brief map data of the source file in the cache and the map data in the distributed map database according to the source file and the configuration parameters. According to the embodiment of the application, the expandability of the database can be improved, and the maintenance cost is reduced.
Description
Technical Field
The present application relates to the field of computers, and more particularly to the field of databases.
Background
The graph database is a database for storing relationship information between entities by applying graph theory, and graph data is stored in the database. The graph database stores relational data to achieve a good effect. The bearing capacity and scale of the graph database are correspondingly different according to different scenes or different application requirements.
As the demand for graph databases increases, graph databases of corresponding sizes need to be designed for different levels of graph data.
Disclosure of Invention
In order to solve at least one problem of the prior art, embodiments of the present application provide a database operation method and apparatus.
In a first aspect, embodiments of the present application provide reading a source file according to set configuration parameters; the database operation source file comprises the update content of the map data;
and updating brief map data corresponding to the database operation source file in the cache and corresponding map data in the distributed map database according to the database operation source file and the database operation configuration parameters.
In the embodiment of the application, the source file used for updating the map data can be read according to the configuration parameters, and the map data is stored in the distributed map database, so that the map division processing is not needed when the data amount is increased, the consistency of the data during data change is favorably maintained, and meanwhile, the data communication is facilitated.
In one embodiment, the database operation profile data comprises vertices; according to the database operation source file and the configuration parameters, the brief map data corresponding to the database operation source file in the cache and the map data corresponding to the distributed map database are updated, and the method comprises the following steps:
performing duplicate removal on point data in a database operation source file;
judging whether the data of the residual points after the duplication removal is consistent with the existing vertexes in the cache;
if so, updating the corresponding complete data of the existing vertex of the database operation in the database operation distributed graph database according to the residual point data and the configuration parameters of the database operation;
if not, establishing a new vertex in the database operation cache according to the residual point data and configuration parameters of the database operation, and storing complete data corresponding to the new vertex of the database operation in the database operation distributed graph database.
In the embodiment of the application, reading and writing of data in the distributed graph database are different from Hadoop modes, only the cache and the data in the distributed graph database need to be updated according to the existing brief graph data, the establishment of the database does not depend on various components, and the development cost is low.
In one embodiment, the database operation profile data includes vertices and edge data corresponding to the vertices; according to the database operation source file and the configuration parameters, the brief map data corresponding to the database operation source file in the cache and the map data in the distributed map database are updated, and the method comprises the following steps:
generating edge processing and loading parameters according to a database operation source file;
screening the side data in the database operation source file by using the database operation side processing and loading parameters;
inquiring whether the vertexes corresponding to the screened residual edge data are consistent with the existing vertexes in the cache;
if so, updating the corresponding complete data of the existing vertex of the database operation in the database operation distributed graph database according to the database operation side data and the configuration parameters;
if not, establishing a new edge in the database operation cache according to the database operation edge data and the configuration parameters, and storing complete data corresponding to the new edge of the database operation in the database operation distributed graph database.
In the embodiment of the application, reading and writing of data in the distributed graph database are different from Hadoop modes, only the cache and the data in the distributed graph database need to be updated according to the existing brief graph data, the establishment of the database does not depend on various components, and the development cost is low.
In one embodiment, after updating the brief map data corresponding to the database operation source file in the cache and the corresponding map data in the distributed map database according to the database operation source file and the configuration parameters, the method further includes:
and when the time interval from the last execution of the step of reading the source file according to the set configuration parameters reaches a set time threshold, returning to the step of reading the source file according to the set configuration parameters.
In the embodiment of the application, the data in the cache and the distributed graph database can be updated at a certain period, and the validity of the data is ensured.
In one embodiment, the configuration parameters include data source configuration information and target source configuration information;
the database operation data source configuration information comprises data source paths and the meaning of fields of a database operation source file; the database operation data source path is used for indicating a reading path of a database operation source file;
the database operation target source configuration information comprises a target path, identification information and attribute information of a vertex corresponding to point data in a database operation source file, and attribute information of an edge corresponding to edge data in the database operation source file; the database operation target path is used for indicating a path for updating the brief map data in the database operation cache according to the database operation source file and/or a path for updating the picture data in the database operation distributed map database according to the database operation source file.
According to the embodiment of the application, the configuration information is set, so that reading of the source file and updating of data can be carried out according to the configuration information, the number of system components is reduced, and development and maintenance cost is reduced.
In a second aspect, an embodiment of the present application provides a database processing apparatus, including:
a configuration file reading module: the device is used for reading the source file according to the set configuration parameters; the database operation source file comprises the update content of the map data;
an update module: and the map updating module is used for updating the brief map data corresponding to the database operation source file in the cache and the map data corresponding to the distributed map database according to the database operation source file and the configuration parameters.
In one embodiment, the database operation profile data includes vertices; the database operation updating module comprises:
a point data deduplication unit: the database operation source file is used for carrying out deduplication on point data in the database operation source file;
a point data judgment unit: the vertex data processing module is used for judging whether the residual point data after the duplication removal is consistent with the existing vertexes in the cache or not;
a first point data operation unit: the database operation point data judging unit is used for judging whether the data in the database operation distributed graph database corresponds to the vertex data in the database operation distributed graph database or not according to the configuration parameters and the data in the database operation distributed graph database;
a second dot data operation unit: and the data processing unit is used for establishing a new vertex in the database operation cache according to the residual point data and configuration parameters of the database operation under the condition that the judgment result of the database operation point data judgment unit is negative, and storing complete data corresponding to the new vertex of the database operation in the database operation distributed database.
In one embodiment, the database operation profile data includes vertices and edge data corresponding to the vertices; the database operation updating module comprises:
a side loading unit: the system comprises a database, a processing module, a loading module and a processing module, wherein the database is used for operating a source file according to a database and generating edge processing and loading parameters;
side data screening unit: the system comprises a database operation source file, a database operation side processing and loading parameter selection module, a database operation module and a database operation module, wherein the database operation side processing and loading parameter selection module is used for selecting side data in the database operation source file by using the database operation side processing and loading parameter;
an edge query unit: the vertex corresponding to the screened residual edge data is inquired whether to be consistent with the existing vertex in the cache or not;
a first side data operation unit: the database operation side query unit is used for updating the corresponding complete data of the existing vertex of the database operation in the database operation distributed database according to the database operation side data and the configuration parameters under the condition that the judgment result of the database operation side query unit is yes;
a second side data operation unit: and the data processing unit is used for establishing a new side in the database operation cache according to the database operation side data and the configuration parameters under the condition that the judgment result of the database operation side inquiry unit is negative, and storing complete data corresponding to the new side of the database operation in the database operation distributed database.
In one embodiment, the database operating apparatus further includes:
a circulation module: and returning to the step of reading the source file according to the set configuration parameters when the time interval from the last execution of the step of reading the source file according to the set configuration parameters reaches a set time threshold.
In one embodiment, the configuration parameters include data source configuration information and target source configuration information;
the database operation data source configuration information comprises data source paths and the meaning of fields of a database operation source file; the database operation data source path is used for indicating a reading path of a database operation source file;
the database operation target source configuration information comprises a target path, identification information and attribute information of a vertex corresponding to point data in a database operation source file, and attribute information of an edge corresponding to edge data in the database operation source file; the database operation target path is used for indicating a path for updating the brief map data in the database operation cache according to the database operation source file and/or a path for updating the picture data in the database operation distributed map database according to the database operation source file.
One embodiment in the above application has the following advantages or benefits: a database is provided that is capable of applying atlas data on the order of tens of millions of scales or less. Because the technical means of combining the cache with the distributed graph database is adopted, the technical problems of difficult development and maintenance and the like are overcome, and the technical effects of no need of graph splitting processing, support of horizontal expansion and easiness in development and maintenance when the data amount is increased are further achieved.
Other effects of the above-described alternative will be described below with reference to specific embodiments.
Drawings
The drawings are included to provide a better understanding of the present solution and are not to be considered limiting of the present application. Wherein:
FIG. 1 is a schematic diagram of a database operation method according to a first embodiment of the present application;
FIG. 2 is a schematic diagram of a database operation method according to a second embodiment of the present application;
FIG. 3 is a schematic diagram of a database operation method according to a third embodiment of the present application;
FIG. 4 is a schematic diagram of a database operation method according to a fourth embodiment of the present application;
FIG. 5 is a schematic diagram of a database operation method according to a fifth embodiment of the present application;
FIG. 6 is a schematic diagram of a database operating apparatus according to a sixth embodiment of the present application;
FIG. 7 is a schematic diagram of a database operating apparatus according to a sixth embodiment of the present application;
FIG. 8 is a schematic diagram of a database operating apparatus according to an eighth embodiment of the present application;
FIG. 9 is a schematic diagram of a database operating apparatus according to a ninth embodiment of the present application;
FIG. 10 is a schematic diagram of a database operating apparatus according to a tenth embodiment of the present application;
fig. 11 is a block diagram of an electronic device for implementing a database operation method according to an embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
An embodiment of the present application first provides a database operation method, as shown in fig. 1, including:
step S11: reading a source file according to the set configuration parameters; the database operation source file comprises the updated content of the map data.
Step S12: and updating brief map data corresponding to the database operation source file in the cache and corresponding map data in the distributed map database according to the database operation source file and the configuration parameters.
In the embodiment of the application, the source file used for updating the map data can be read according to the configuration parameters, and the map data is stored in the distributed map database, so that the map division processing is not needed when the data amount is increased, the consistency of the data during data change is favorably maintained, and the data communication is facilitated.
In the embodiment of the present application, the source file read according to the set configuration parameters may be stored on a disk or other storage device.
In the embodiment of the application, the brief map data corresponding to the source file in the cache and the map data in the distributed map database have a corresponding relation. According to the database operation source file and the configuration parameters, updating the brief map data corresponding to the database operation source file in the cache and the map data in the distributed map database, which may include: according to the source file and the configuration data, whether brief data corresponding to the source file exists or not is searched in the cache, and a search result is obtained; and then, according to the search result and the configuration data, updating the brief map data corresponding to the source file in the cache and the map data in the distributed map database.
In this embodiment of the present application, updating the brief map data corresponding to the database operation source file in the cache and the map data in the distributed map database may include: updating the complete data of the existing brief map data in the cache in the distributed map database; and adding new brief map data in the cache, and adding complete data corresponding to the new brief map data in the distributed map database.
In an example of the present application, a data reading method is shown in fig. 2, and includes:
step S21: setting configuration parameters, wherein the configuration parameters comprise a reading path and other parameters of a source file. In the example of the application, reading files from a multi-directory can be supported by setting configuration parameters.
Step S22: and reading the source file in a multithread mode according to the configuration parameters. And batch data acquisition is supported by reading data in multiple threads.
Step S23: batch (batch) data is returned from multi-threaded read file operations.
Step S24: and judging whether the reading is finished.
Step S25: and if so, using the obtained batch data as a source file.
Step S26: if not, the process returns to step S22.
In one embodiment, the database operation profile data includes vertices; according to the database operation source file and the configuration parameters, the brief map data corresponding to the database operation source file in the cache and the map data corresponding to the distributed map database are updated, and the method comprises the following steps as shown in fig. 3:
step S31: and carrying out deduplication on the point data in the database operation source file. And then the point data remaining after the deduplication is loaded.
Step S32: and judging whether the data of the data remained after the duplication removal is consistent with the existing vertex in the cache.
Step S33: if yes, updating the corresponding complete data of the existing vertex of the database operation in the database operation distributed graph database according to the remaining point data and the configuration parameters of the database operation.
Step S34: if not, establishing a new vertex in the database operation cache according to the residual point data and configuration parameters of the database operation, and storing complete data corresponding to the new vertex of the database operation in the database operation distributed graph database.
In the embodiment of the application, reading and writing of data in the distributed graph database are different from Hadoop modes, only the cache and the data in the distributed graph database need to be updated according to the existing brief graph data, the establishment of the database does not depend on various components, and the development cost is low.
In the embodiment of the application, the point data in the database operation source file is subjected to deduplication, that is, redundant data corresponding to the same point data in the source file is removed.
In the embodiment of the application, the database operation brief atlas data comprises a vertex and edge data corresponding to the vertex; according to the database operation source file and the configuration parameters, the brief map data corresponding to the database operation source file in the cache and the map data corresponding to the distributed map database are updated, as shown in fig. 4, including:
step S41: and operating the source file according to the database to generate edge processing and loading parameters.
Step S42: and screening the side data in the database operation source file by using the database operation side processing and loading parameters.
Step S43: and inquiring whether the vertexes corresponding to the screened residual edge data are consistent with the existing vertexes in the cache.
Step S44: if so, updating the corresponding complete data of the existing vertex of the database operation in the database operation distributed graph database according to the database operation side data and the configuration parameters.
Step S45: if not, establishing a new edge in the database operation cache according to the database operation edge data and the configuration parameters, and storing complete data corresponding to the new edge of the database operation in the database operation distributed graph database.
In the embodiment of the application, reading and writing of data in the distributed graph database are different from Hadoop modes, only the cache and the data in the distributed graph database need to be updated according to the existing brief graph data, the establishment of the database does not depend on various components, and the development cost is low.
In one embodiment, as shown in fig. 5, after updating the corresponding brief map data of the database operation source file in the cache and the map data in the distributed map database according to the database operation source file and the configuration parameters, the method further includes:
step S51: and when the time interval from the last execution of the step of reading the source file according to the set configuration parameters reaches a set time threshold, returning to the step of reading the source file according to the set configuration parameters.
In the embodiment of the application, the data in the cache and the distributed graph database can be updated at a certain period, and the validity of the data is ensured.
In one embodiment, the configuration parameters include data source configuration information and target source configuration information;
the database operation data source configuration information comprises data source paths and the meaning of fields of a database operation source file; the database operation data source path is used for indicating a reading path of a database operation source file;
the database operation target source configuration information comprises a target path, identification information and attribute information of a vertex corresponding to point data in a database operation source file, and attribute information of an edge corresponding to edge data in the database operation source file; the database operation target path is used for indicating a path for updating the brief map data in the database operation cache according to the database operation source file and/or a path for updating the picture data in the database operation distributed map database according to the database operation source file.
According to the embodiment of the application, the configuration information is set, so that reading of the source file and updating of data can be carried out according to the configuration information, the number of system components is reduced, and development and maintenance cost is reduced.
In the embodiment of the present application, the configuration information may further include a delimiter.
An embodiment of the present application further provides a database operating apparatus, as shown in fig. 6, including:
the configuration file reading module 61: the device is used for reading the source file according to the set configuration parameters; the database operation source file comprises the update content of the map data;
the update module 62: and the map updating module is used for updating the brief map data corresponding to the database operation source file in the cache and the map data corresponding to the distributed map database according to the database operation source file and the configuration parameters.
In one embodiment, the database operation profile data includes vertices; referring to fig. 7, the database operation update module includes:
the point-data deduplication unit 71: the database operation source file is used for carrying out deduplication on point data in the database operation source file;
the dot data judgment unit 72: the vertex data processing module is used for judging whether the residual point data after the duplication removal is consistent with the existing vertexes in the cache or not;
first point data operation unit 73: the database operation point data judging unit is used for judging whether the data in the database operation distributed graph database corresponds to the vertex data in the database operation distributed graph database or not according to the configuration parameters and the data in the database operation distributed graph database;
the second point-data operating unit 74: and the data processing unit is used for establishing a new vertex in the database operation cache according to the data and the configuration parameters of the data and the configuration parameters which are left in the database operation under the condition that the judgment result of the data judgment unit of the database operation point is negative, and storing the complete data corresponding to the new vertex in the database operation distributed database.
In one embodiment, the database operation profile data includes vertices and edge data corresponding to the vertices; referring to fig. 8, the database operation update module includes:
side loading unit 81: the system comprises a database, a processing module, a loading module and a processing module, wherein the database is used for operating a source file according to a database and generating edge processing and loading parameters;
the side data filtering unit 82: the system comprises a database operation source file, a database operation processing and loading parameter and a database operation processing and loading parameter, wherein the database operation processing and loading parameter is used for processing the operation of the database operation source file;
the side query unit 83: the vertex corresponding to the screened residual edge data is inquired whether to be consistent with the existing vertex in the cache or not;
first side data operation unit 84: the database operation side query unit is used for updating the corresponding complete data of the existing vertex of the database operation in the database operation distributed database according to the database operation side data and the configuration parameters under the condition that the judgment result of the database operation side query unit is yes;
the second side data operation unit 85: and the data processing unit is used for establishing a new side in the database operation cache according to the database operation side data and the configuration parameters under the condition that the judgment result of the database operation side inquiry unit is negative, and storing complete data corresponding to the new side of the database operation in the database operation distributed database.
In one embodiment, as shown in fig. 9, the database operating apparatus further includes:
the circulation module 91: and returning to the step of reading the source file according to the set configuration parameters when the time interval from the last execution of the step of reading the source file according to the set configuration parameters reaches a set time threshold.
In one embodiment, the parameter data source configuration information and the target source configuration information are configured;
the database operation data source configuration information comprises data source paths and the meaning of fields of a database operation source file; the database operation data source path is used for indicating a reading path of a database operation source file;
the database operation target source configuration information comprises a target path, identification information and attribute information of a vertex corresponding to point data in a database operation source file, and attribute information of an edge corresponding to edge data in the database operation source file; the database operation target path is used for indicating a path for updating the brief map data in the database operation cache according to the database operation source file and/or a path for updating the picture data in the database operation distributed map database according to the database operation source file.
In an example of the present application, the database operating apparatus further includes a task scheduling module, and as shown in fig. 10, the task scheduling module 101 includes a scheduler uploading unit: for uploading a scheduler; the timing command writing unit is also included: the trigger loop module 91 is used to write timing commands. And is also used for sending execution commands to each execution module of the database operating device.
According to an embodiment of the present application, an electronic device and a readable storage medium are also provided.
Fig. 11 is a block diagram of an electronic device according to an embodiment of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 11, the electronic apparatus includes: one or more processors 1101, a memory 1102, and interfaces for connecting the various components, including a high speed interface and a low speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display Graphical information for a Graphical User Interface (GUI) on an external input/output device, such as a display device coupled to the Interface. In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). In fig. 11, a processor 1101 is taken as an example.
The memory 1102 is a non-transitory computer readable storage medium as provided herein. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the database operation method provided herein. The non-transitory computer-readable storage medium of the present application stores computer instructions for causing a computer to perform the database operation method provided herein.
The memory 1102, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules (e.g., the configuration file reading module 61, the updating module 62 shown in fig. 6) corresponding to the database operation method in the embodiment of the present application. The processor 1101 executes various functional applications of the server and data processing, i.e., implements the database operation method in the above-described method embodiment, by executing non-transitory software programs, instructions, and modules stored in the memory 1102.
The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created from use of the database operating electronic device, and the like. Further, the memory 1102 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 1102 may optionally include memory located remotely from the processor 1101, which may be connected to a database operating electronic devices via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device of the database operation method may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected by a bus or other means, and are exemplified by being connected by a bus in fig. 11.
The input device 1103 may receive input numeric or character information and generate key signal inputs relating to user settings and function controls of the database operating electronic device, such as a touch screen, keypad, mouse, track pad, touch pad, pointer stick, one or more mouse buttons, track ball, joystick or other input device. The output devices 1104 may include a display device, auxiliary lighting devices (e.g., LEDs), tactile feedback devices (e.g., vibrating motors), and the like. The Display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) Display, and a plasma Display. In some implementations, the display device can be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, Integrated circuitry, Application Specific Integrated Circuits (ASICs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (Cathode Ray Tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the application, the configured user-defined map is supported, a user only needs to put original data into a certain path in a text file mode and write a configuration file, the system can automatically create a schema (a collection of database objects) of the map, and then the relation between points and edges is extracted from the original data according to the configuration file, and the map database is filled in batches. While supporting task scheduling.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A method of database operation, comprising:
reading a source file according to the set configuration parameters; the source file comprises updated content of map data;
updating brief map data corresponding to the source file in a cache and corresponding map data in a distributed map database according to the source file and the configuration parameters; the brief map data and the map data have a corresponding relation;
the profile data comprises vertices; according to the source file and the configuration parameters, updating the brief map data corresponding to the source file in the cache and the map data corresponding to the distributed map database, wherein the brief map data and the map data corresponding to the source file in the cache comprise:
performing deduplication on the point data in the source file;
judging whether the data of the residual points after the duplication removal is consistent with the existing vertexes in the cache;
if yes, updating the corresponding complete data of the existing vertex in the distributed graph database according to the residual point data and the configuration parameters;
if not, establishing a new vertex in the cache according to the residual point data and the configuration parameters, and storing complete data corresponding to the new vertex in the distributed database.
2. The method of claim 1, wherein the profile data includes vertices and edge data corresponding to the vertices; according to the source file and the configuration parameters, updating the brief map data corresponding to the source file in the cache and the map data corresponding to the distributed map database, wherein the brief map data and the map data corresponding to the source file in the cache comprise:
generating edge processing and loading parameters according to the source file;
screening the edge data in the source file by using the edge processing and loading parameters;
inquiring whether vertexes corresponding to the screened residual edge data are consistent with the existing vertexes in the cache or not;
if so, updating the corresponding complete data of the existing vertex in the distributed graph database according to the side data and the configuration parameters;
if not, establishing a new edge in the cache according to the edge data and the configuration parameters, and storing complete data corresponding to the new edge in the distributed graph database.
3. The method according to claim 1, wherein after updating corresponding profile map data in a cache and corresponding map data in a distributed map database for the source file based on the source file and the configuration parameters, further comprising:
and when the time interval from the last execution of the step of reading the source file according to the set configuration parameters reaches a set time threshold, returning to the step of reading the source file according to the set configuration parameters.
4. The method of claim 1, wherein the configuration parameters comprise data source configuration information and target source configuration information;
the data source configuration information comprises a data source path and the meaning of the field of the source file; the data source path is used for indicating a read path of the source file;
the target source configuration information comprises a target path, identification information and attribute information of a vertex corresponding to the point data in the source file, and attribute information of an edge corresponding to the edge data in the source file; the target path is used for indicating a path for updating the brief map data in the cache according to the source file and/or a path for updating the picture data in the distributed map database according to the source file.
5. A database processing apparatus, comprising:
a configuration file reading module: the device is used for reading a source file according to the set configuration parameters; the source file comprises updated content of map data;
an update module: the device is used for updating the brief map data corresponding to the source file in the cache and the map data corresponding to the distributed map database according to the source file and the configuration parameters; the brief map data and the map data have a corresponding relation;
the profile data comprises vertices; the update module includes:
a point data deduplication unit: the source file is used for carrying out deduplication on the point data in the source file;
a point data judgment unit: the vertex data processing module is used for judging whether the residual point data after the duplication removal is consistent with the existing vertexes in the cache or not;
a first point data operation unit: the system comprises a point data judging unit, a distributed graph database and a vertex data updating unit, wherein the point data judging unit is used for judging whether the existing vertex corresponds to the distributed graph database or not according to the configuration parameters;
a second dot data operation unit: and the vertex data judging unit is used for establishing a new vertex in the cache according to the residual point data and the configuration parameters and storing complete data corresponding to the new vertex in the distributed database under the condition that the judgment result of the point data judging unit is negative.
6. The apparatus of claim 5, wherein the profile data includes vertices and edge data corresponding to the vertices; the update module includes:
a side loading unit: the system comprises a source file, a side processing module and a loading module, wherein the source file is used for generating side processing and loading parameters according to the source file;
side data screening unit: the device is used for screening the edge data in the source file by using the edge processing and loading parameters;
an edge query unit: the vertex corresponding to the screened residual edge data is inquired whether to be consistent with the existing vertex in the cache or not;
a first side data operation unit: the side query unit is used for updating the corresponding complete data of the existing vertex in the distributed graph database according to the side data and the configuration parameters under the condition that the judgment result of the side query unit is yes;
a second side data operation unit: and the edge query unit is used for establishing a new edge in the cache according to the edge data and the configuration parameters under the condition that the judgment result of the edge query unit is negative, and storing complete data corresponding to the new edge in the distributed graph database.
7. The apparatus of claim 5, further comprising:
a circulation module: and returning to the step of reading the source file according to the set configuration parameters when the time interval from the last execution of the step of reading the source file according to the set configuration parameters reaches a set time threshold.
8. The apparatus of claim 5, wherein the configuration parameters comprise data source configuration information and target source configuration information;
the data source configuration information comprises a data source path and the meaning of the field of the source file; the data source path is used for indicating a read path of the source file;
the target source configuration information comprises a target path, identification information and attribute information of a vertex corresponding to the point data in the source file, and attribute information of an edge corresponding to the edge data in the source file; the target path is used for indicating a path for updating the brief map data in the cache according to the source file and/or a path for updating the picture data in the distributed map database according to the source file.
9. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.
10. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910874903.5A CN110598059B (en) | 2019-09-16 | 2019-09-16 | Database operation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910874903.5A CN110598059B (en) | 2019-09-16 | 2019-09-16 | Database operation method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110598059A CN110598059A (en) | 2019-12-20 |
| CN110598059B true CN110598059B (en) | 2022-07-05 |
Family
ID=68860114
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910874903.5A Active CN110598059B (en) | 2019-09-16 | 2019-09-16 | Database operation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110598059B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110851178B (en) * | 2020-01-14 | 2020-05-19 | 江苏鸿程大数据技术与应用研究院有限公司 | Inter-process program static analysis method based on distributed graph reachable computation |
| CN112015819A (en) * | 2020-08-31 | 2020-12-01 | 杭州欧若数网科技有限公司 | Data updating method, device, equipment and medium for distributed graph database |
| CN112328669A (en) * | 2020-10-12 | 2021-02-05 | 联通智网科技有限公司 | Data exchange method and device for heterogeneous data source and computer equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105045536A (en) * | 2015-07-29 | 2015-11-11 | 百度在线网络技术（北京）有限公司 | Data storage method, apparatus and system |
| CN105867837A (en) * | 2015-12-02 | 2016-08-17 | 乐视体育文化产业发展（北京）有限公司 | Method, equipment and system for updating configurations of clients in distributed high-speed cache systems |
| CN108062308A (en) * | 2016-11-07 | 2018-05-22 | 天脉聚源（北京）科技有限公司 | A kind of method and system of Distributed Storage |
| CN109255055A (en) * | 2018-08-06 | 2019-01-22 | 四川蜀天梦图数据科技有限公司 | A kind of diagram data access method and device based on packet associated table |
| CN109670089A (en) * | 2018-12-29 | 2019-04-23 | 颖投信息科技(上海)有限公司 | Knowledge mapping system and its figure server |
-
2019
- 2019-09-16 CN CN201910874903.5A patent/CN110598059B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105045536A (en) * | 2015-07-29 | 2015-11-11 | 百度在线网络技术（北京）有限公司 | Data storage method, apparatus and system |
| CN105867837A (en) * | 2015-12-02 | 2016-08-17 | 乐视体育文化产业发展（北京）有限公司 | Method, equipment and system for updating configurations of clients in distributed high-speed cache systems |
| CN108062308A (en) * | 2016-11-07 | 2018-05-22 | 天脉聚源（北京）科技有限公司 | A kind of method and system of Distributed Storage |
| CN109255055A (en) * | 2018-08-06 | 2019-01-22 | 四川蜀天梦图数据科技有限公司 | A kind of diagram data access method and device based on packet associated table |
| CN109670089A (en) * | 2018-12-29 | 2019-04-23 | 颖投信息科技(上海)有限公司 | Knowledge mapping system and its figure server |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110598059A (en) | 2019-12-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111258609B (en) | Upgrading method and device of Kubernetes cluster, electronic equipment and medium | |
| CN110598059B (en) | Database operation method and device | |
| CN111079938B (en) | Question-answer reading understanding model obtaining method and device, electronic equipment and storage medium | |
| CN111045985B (en) | File storage processing method, server, electronic device and storage medium | |
| CN111080243A (en) | Service processing method, device, system, electronic equipment and storage medium | |
| CN104423960A (en) | Continuous project integration method and continuous project integration system | |
| EP3825865A2 (en) | Method and apparatus for processing data | |
| CN111506803A (en) | Content recommendation method and device, electronic equipment and storage medium | |
| CN111783952A (en) | Configuration method, device, system, electronic equipment and storage medium | |
| CN111737399A (en) | Method and device for expanding question and answer set, electronic equipment and readable storage medium | |
| CN112818048A (en) | Hierarchical construction method and device of data warehouse, electronic equipment and storage medium | |
| CN110633281A (en) | Method and device for processing multi-type data sources | |
| CN111858796A (en) | Geographic information system engine system, implementation method, device and storage medium | |
| CN111259090A (en) | Graph generation method and device of relational data, electronic equipment and storage medium | |
| CN112115113B (en) | Data storage system, method, device, equipment and storage medium | |
| CN113254043A (en) | Web front-end project processing method and device, electronic equipment and storage medium | |
| CN112541513B (en) | Model training method, device, equipment and storage medium | |
| CN111796851A (en) | Client base library updating method and device, electronic equipment and storage medium | |
| CN111782147A (en) | Method and apparatus for cluster scale-up | |
| CN111026916A (en) | Text description conversion method and device, electronic equipment and storage medium | |
| CN112560928B (en) | Negative sample mining method and device, electronic equipment and storage medium | |
| CN111767444B (en) | Page feature construction method, device, equipment and storage medium | |
| CN112527527A (en) | Consumption speed control method and device of message queue, electronic equipment and medium | |
| CN112346512A (en) | Time synchronization method and device | |
| CN111292223A (en) | Graph calculation processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |