CN111046050A - Data Schema processing method, device, equipment and storage medium - Google Patents
Data Schema processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111046050A CN111046050A CN201811194008.0A CN201811194008A CN111046050A CN 111046050 A CN111046050 A CN 111046050A CN 201811194008 A CN201811194008 A CN 201811194008A CN 111046050 A CN111046050 A CN 111046050A
- Authority
- CN
- China
- Prior art keywords
- schema
- data
- database
- data schema
- meeting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 10
- 238000000034 method Methods 0.000 claims abstract description 41
- 238000012545 processing Methods 0.000 claims abstract description 29
- 230000008859 change Effects 0.000 claims abstract description 13
- 238000004590 computer program Methods 0.000 claims description 13
- 238000012216 screening Methods 0.000 claims description 6
- 238000012217 deletion Methods 0.000 claims description 5
- 230000037430 deletion Effects 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data Schema processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring a database log and a message queue real-time data stream; analyzing the database log and the message queue real-time data stream to obtain a data Schema corresponding to the database log and the message queue real-time data stream; and updating the Schema database according to the data Schema meeting the preset conditions. The method reduces the manual change of program codes, reduces the labor cost and realizes the automatic processing and sharing of the data Schema.
Description
Technical Field
The invention relates to the technical field of internet application, in particular to a data Schema processing method, a data Schema processing device, data Schema processing equipment and a storage medium.
Background
With the advent of the cloud era, big data attracts more and more attention, and it needs a special technology to effectively process a large amount of data sets, and has the characteristics of massive data scale, rapid data circulation, various data types, low value density and the like. In the process of storage, calculation and shared use of data, a data Schema refers to the problem concerned by a data platform and a data user, wherein the data Schema refers to meta information composed of fields, structures, types, formats and the like of service data. When the upstream service system database changes the field, in order to make the big data platform and the downstream data user know the change of the data Schema in real time, the application program or the data table field needs to be changed.
In the prior art, a business system registers or updates a data Schema through a Schema registry, and a downstream user acquires the updated Schema through the registry, thereby implementing sharing of the data Schema.
However, in the conventional technology, the data Schema needs to be manually sent to the registry, the different modules of different components need to be matched with each other, the processing process is complex, the evolution processing and sharing of the data Schema cannot be automatically realized, and the flexibility is low.
Disclosure of Invention
In view of the foregoing defects or shortcomings in the prior art, it is desirable to provide a method, an apparatus, a device and a storage medium for processing a data Schema, which can implement automatic update and distribution of the data Schema.
In a first aspect, the present invention provides a method for processing a data Schema, where the method includes:
acquiring a database log and a message queue real-time data stream;
analyzing the database log and the message queue real-time data stream to obtain a data Schema corresponding to the database log and the message queue real-time data stream;
and updating the Schema database according to the data Schema meeting the preset conditions.
In one embodiment, the analyzing the database log and the message queue real-time data stream to obtain the data Schema corresponding to the database log and the message queue real-time data stream includes:
analyzing the database log to obtain an operation record in the database log; the operation record comprises creation, deletion and change fields;
analyzing the message queue real-time data stream to obtain a specified data field in the message queue real-time data stream;
and determining the operation record and the data field as the data Schema.
In one embodiment, the updating the Schema database according to the data Schema meeting the preset condition includes:
sending a query instruction to the Schema database, and judging whether a data Schema with the same file name as the data Schema exists in the Schema database;
if the data Schema is consistent with the preset condition, updating the data Schema with the same file name as the data Schema consistent with the preset condition according to the data Schema consistent with the preset condition;
if the data does not exist, the data Schema meeting the preset conditions is registered in the Schema database.
In one embodiment, after updating the data Schema database according to the data Schema meeting the preset condition, the method further includes:
and issuing the data Schema meeting the preset conditions.
In one embodiment, before the updating the Schema database according to the data Schema meeting the preset condition, the method further includes:
screening out data schemas meeting preset conditions from the data schemas; the preset conditions include: and the file format of the data Schema conforms to a preset standard format.
In a second aspect, an embodiment of the present application provides a device for processing a data Schema, where the device includes:
the first acquisition module is used for acquiring a database log and a message queue real-time data stream;
the second obtaining module is configured to analyze the database log and the message queue real-time data stream, and obtain a data Schema corresponding to the database log and the message queue real-time data stream;
and the updating module is used for updating the Schema database according to the data Schema meeting the preset conditions.
In a third aspect, an embodiment of the present application provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the processing method of the data Schema according to any one of the foregoing descriptions when executing the computer program.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the processing method of the data Schema described in any one of the above.
The method, the device, the equipment and the storage medium for processing the data Schema provided by the embodiment of the invention are used for acquiring the database log and the real-time data stream of the message queue, acquiring the data Schema corresponding to the database log and the real-time data stream of the message queue by analyzing the database log and the real-time data stream of the message queue, and updating the Schema database according to the data Schema meeting the preset conditions. According to the technical scheme, the database logs and the real-time data stream of the message queue are monitored in real time, so that the data Schema can be obtained in real time, and the Schema database is automatically updated according to the obtained data Schema meeting the preset conditions, so that the operations of changing program codes manually and the like are reduced to a great extent, the labor cost is reduced, and the automatic update of the data Schema is realized.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
fig. 1 is a schematic flow chart of a data Schema processing method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of obtaining a data Schema according to an embodiment of the present invention;
fig. 3 is a schematic flow chart of updating a Schema database according to a data Schema meeting preset conditions according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a processing apparatus for providing a data Schema according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the present invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
As mentioned in the background art, when the upstream service system database changes the field, the big data platform and the downstream data user cannot know the change of the data Schema in the first time, so that the application service needs to manually change the application program or the bottom data table field, which brings inconvenience to the construction, management and data use of the data warehouse. At present, a registry service is provided, a service system registers or updates a data Schema, and a downstream user acquires the latest data Schema through the registry so as to realize sharing of the data Schema.
Based on the above defects, embodiments of the present invention provide a method for processing a data Schema, which can monitor a database log and a message queue real-time data stream in real time, and automatically update a Schema database by analyzing the database log and the message queue real-time data stream.
The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Fig. 1 is a schematic flow chart of a processing method of a data Schema provided in an embodiment of the present invention, which relates to a method for updating a Schema database according to a data Schema when a database log and a real-time data stream of a message queue are obtained. As shown in fig. 1, the method includes:
s101, acquiring a database log and a message queue real-time data stream.
Specifically, each database at least comprises two operating system files, namely a data file and a log file, wherein the data file comprises data and objects such as a table, an index, a storage process and a view, and the log file, namely a database log, stores various operations on the database, such as data addition, modification, deletion and the like, and can be used for recovering the database when the content of the database is lost; the message queue is an important component for exchanging information among distributed applications, and can solve the problems of application decoupling, asynchronous messages, traffic cut and the like, for example: kafka is a kind of message queue, which is a distributed, open source stream processing platform, and can process all the action stream data in a customer-scale website for a real-time service data stream source.
Optionally, in the embodiment of the present invention, the database log may be monitored by calling a related command in the listener, and the log file is read in real time to obtain the database log; the message queue real-time data stream can be obtained by opening a client to consume messages of the message queue.
S102, analyzing the database log and the message queue real-time data stream, and acquiring a data Schema corresponding to the database log and the message queue real-time data stream.
Specifically, after acquiring a database log and a message queue real-time data stream, searching and analyzing the database log, analyzing change information such as creating, deleting and changing fields in the database log, and determining the change information as a data Schema corresponding to the database log; and meanwhile, opening the message of the client consumption queue, analyzing the real-time data stream of the message queue, acquiring a data field corresponding to the real-time data stream of the message queue, and determining the data field as a data Schema corresponding to the real-time data stream of the message queue.
And S103, updating the Schema database according to the data Schema meeting the preset conditions.
Specifically, after the data Schema is acquired, the data Schema meeting the preset condition may be screened from the data Schema, and a query instruction is sent to the Schema database to find whether the acquired data Schema meeting the preset condition exists in the Schema database, and when the acquired data Schema meeting the preset condition does not exist in the Schema database, an update operation may be performed on the Schema database according to the acquired data Schema meeting the preset condition, where the update operation may include: and deleting, adding, modifying and the like the fields in the data Schema. Optionally, the preset condition may be that a file format including the data Schema conforms to a preset standard format.
According to the method for processing the data Schema provided by the embodiment of the invention, when the real-time data stream of the database log and the message queue is obtained, the real-time data stream of the database log and the message queue is analyzed, the data Schema corresponding to the real-time data stream of the database log and the message queue is obtained, and the Schema database is updated according to the data Schema meeting the preset conditions. The database log and the message queue real-time data stream are monitored in real time, so that the data Schema can be obtained in real time, the data Schema meeting the preset condition is screened out, the Schema database is automatically updated according to the data Schema meeting the preset condition, the operations of changing program codes manually and the like are reduced to a great extent, the labor cost is reduced, the automatic updating of the data Schema is realized, and the data sharing and use are further better realized.
Fig. 2 is a schematic flow chart of obtaining the data Schema according to the embodiment of the present invention. Optionally, as shown in fig. 2, a possible implementation manner of S102 includes:
s201, analyzing a database log, and acquiring an operation record in the database log; the operation record includes create, delete, and change fields.
Illustratively, the database log records all changes on the database, and the changes are stored in a disk in a binary form and used for checking change history of the database, incremental backup and reply of the database, and copying of the database; taking the database mysql as an example, the database log is an important component of the mysql database, in which the client connection status of the mysql database and the execution status of the sql statement are recorded, and the operation record corresponding to the database log can be determined by calling the relevant statement instruction, setting the query log to be open, and defining the file address name of the query log.
S202, analyzing the message queue real-time data stream, and acquiring the specified data field in the message queue real-time data stream.
The message queue is a middleware of data storage, for example, a message queue kafka is taken as an example, which is a high-throughput distributed publish-subscribe message system, and all action flow data in a customer-scale website can be processed by unifying online and offline message processing through a parallel loading mechanism of Hadoop.
Optionally, the designated data field in the real-time data stream corresponding to the message queue may be obtained by performing asynchronous processing on the message queue, where the designated data field may be a field representing a data format, for example, a field of a json data format.
S203, determining the operation records in the database log and the specified data fields in the message queue real-time data stream as data Schema.
Specifically, the data Schema is a combination of database objects, one user generally corresponds to one Schema, the data Schema can be determined by analyzing operation records corresponding to database logs and data fields corresponding to message queue real-time data streams, and is represented in an Avro file format, and by using the Avro file, interaction between stored data and RPC (Remote Procedure Call) can be realized without generating an additional API (Application Programming Interface) code.
Optionally, before performing step S103, the method may further include:
and screening the data Schema meeting the preset conditions from the data Schema.
Specifically, the data Schema meeting the preset condition is a legal data Schema, the data Schema meeting the preset condition is screened from the acquired data Schema, and whether the file format of the data Schema is legal or not can be judged, so that the legal data Schema is screened and updated.
Specifically, the Avro is a binary file format supporting data intensive type, after the data Schema is determined, whether the file format of the data Schema conforms to the standard format of the Avro file or not can be checked, if the file format conforms to the standard format of the Avro file, the data Schema is represented as an effective file, and whether the data Schema is the effective file or not is judged, so that all effective files in the Schema database are ensured.
In the method for processing the data Schema provided in this embodiment, the operation record corresponding to the database log is determined by analyzing the database log, the real-time data stream of the message queue is analyzed, the data field corresponding to the real-time data stream of the message queue is determined, the data Schema is determined according to the operation record corresponding to the database and the data field corresponding to the real-time data stream of the message queue, and the data Schema is stored in an Avro file. The operation records corresponding to the database logs and the data fields corresponding to the message queue real-time data streams can be determined more completely by analyzing the database logs and the message queue real-time data streams, and therefore the data Schema can be determined more accurately.
Fig. 3 is a schematic flow chart of updating a Schema database according to a data Schema meeting preset conditions according to an embodiment of the present invention. Optionally, as shown in fig. 3, as an implementation manner of step S103, the method includes:
s301, sending a query instruction to the Schema database, and judging whether a data Schema which is the same as the data Schema file name meeting the preset condition exists in the Schema database.
S302, if the data Schema exists, updating the data Schema with the same file name as the data Schema meeting the preset condition according to the data Schema meeting the preset condition;
and S303, if the data does not exist, registering the data Schema meeting the preset conditions in the Schema database.
It should be noted that, after the data Schema is determined, and when it is determined that the data Schema is legal, an inquiry instruction is sent to the Schema database, where the inquiry instruction may include inquiring a file name of a data Schema meeting a preset condition, so as to determine whether the Schema database contains the data Schema meeting the preset condition.
Further, if the data Schema meeting the preset condition exists in the Schema database, the data Schema meeting the preset condition with the same file name in the database is added, deleted, modified and the like, and the database Schema is updated; and if the data Schema meeting the preset condition does not exist in the Schema database, adding the data Schema meeting the preset condition into the Schema database in a registration mode.
The embodiment of the invention realizes the automatic updating and maintenance of the business data Schema, ensures that the Schema database can be automatically maintained in the continuous processing and changing operation of the data, and is convenient for the effective management and use of the data.
Optionally, after performing step S103, the method may further include:
and issuing the data Schema meeting the preset conditions.
It should be noted that, after the data Schema is determined, the data Schema is stored in the database Schema, and the data Schema meeting the preset condition is published, optionally, the data Schema may be packaged in a Jar format of an entity class and published in a Maven warehouse, and when a user needs to use the data, the data Schema may be downloaded in the Maven warehouse and used in an application program thereof; the data can also be released to the kafka of the message queue through the Json file format to be used for storing real-time data, so that the user can conveniently use the data in a real-time data processing scene; the data can also be pushed to a table storage path of a Hadoop Distributed file system (Hadoop Distributed file system for short) through an Avro file format so as to be suitable for data application in an offline big data scene; and the user can be informed of the updated data Schema through a third-party mail by triggering the mail system, so that the sharing of the data Schema is better realized, the method and the system can be applied to different application scenes, and the sharing and the use of the data are further better realized.
It should be noted that while the operations of the method of the present invention are depicted in the drawings in a particular order, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Rather, the steps depicted in the flowcharts may change the order of execution. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
Fig. 4 is a schematic structural diagram of a data Schema processing apparatus according to an embodiment of the present invention. As shown in fig. 4, the apparatus may implement the method shown in fig. 1, and the apparatus may include:
a first obtaining module 10, configured to obtain a database log and a message queue real-time data stream;
a second obtaining module 20, configured to analyze the database log and the message queue real-time data stream, and obtain a data Schema corresponding to the database log and the message queue real-time data stream;
and the updating module 30 is configured to update the Schema database according to the data Schema meeting the preset condition.
Preferably, the second obtaining module 20 may include:
an operation record obtaining unit 201, configured to parse the database log and determine an operation record corresponding to the database log; the operation record comprises creation, deletion and change fields;
a data field obtaining unit 202, configured to parse the message queue real-time data stream, and obtain a specified data field in the message queue real-time data stream;
a determining unit 203, configured to determine the operation record and the data field as the data Schema.
In the processing apparatus for the data Schema provided in this embodiment, because the database log and the real-time data stream of the message queue are analyzed, the operation record corresponding to the database log and the data field corresponding to the real-time data stream of the message queue can be determined more completely, so that the data Schema is determined more accurately.
Optionally, the apparatus further comprises:
the screening module 40 is configured to screen a data Schema meeting a preset condition from the data Schema; the preset conditions include: and the file format of the data Schema conforms to a preset standard format.
Optionally, the update module 30 includes:
a determining unit 301, configured to send a query instruction to a Schema database, and determine whether a data Schema identical to the data Schema file name meeting the preset condition exists in the Schema database;
an updating unit 302, configured to, when a data Schema identical to the data Schema file name meeting the preset condition exists in the Schema database, perform an updating operation on the data Schema identical to the data Schema file name meeting the preset condition according to the data Schema;
a registering unit 303, configured to register the data Schema meeting the preset condition in the Schema database when the data Schema identical to the data Schema file name does not exist in the Schema database.
Optionally, after the updating module 103 is executed, the method further includes:
and the publishing module 50 is configured to publish the data Schema meeting the preset condition.
Optionally, the publishing module 50 is specifically configured to send the data Schema to the user in any one or a combination of a Jar file format, a Json file format, an Avro file format, and a third-party email.
The processing apparatus of the data Schema provided in this embodiment may execute the embodiment of the method shown in fig. 3, and the implementation principle and the technical effect are similar, and are not described herein again.
Fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present invention. As shown in fig. 5, a schematic structural diagram of a computer system 500 suitable for implementing the terminal device or the server of the embodiment of the present application is shown.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data necessary for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM503 are connected to each other via a bus 504. An input/output (I/O) interface 506 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. A driver 510 is also connected to the I/O interface 506 as needed. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, the processes described above with reference to fig. 1-3 may be implemented as computer software programs, according to the above-described embodiments corresponding to the processing method of the data Schema of the present disclosure. For example, an embodiment of the disclosure includes a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the method of processing the data Schema of fig. 1-3. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units or modules described in the embodiments of the present application may be implemented by software or hardware. The described units or modules may also be provided in a processor, and may be described as: a processor includes a first sub-region generating unit, a second sub-region generating unit, and a display region generating unit. Where the names of these units or modules do not in some cases constitute a definition of the unit or module itself, for example, the display area generating unit may also be described as a "unit for generating a display area of text from the first sub-area and the second sub-area".
As another aspect, the present application also provides a computer-readable storage medium, which may be the computer-readable storage medium included in the foregoing device in the foregoing embodiment; or it may be a separate computer readable storage medium not incorporated into the device. The computer readable storage medium stores one or more programs for use by one or more processors in performing the processing methods of the data Schema described in the present application.
In summary, according to the method, the device, the apparatus, and the storage medium for processing the data Schema provided in the embodiments of the present invention, when the database log and the real-time data stream of the message queue are obtained, the database log and the real-time data stream of the message queue are analyzed, the data Schema corresponding to the database log and the real-time data stream of the message queue is obtained, and the Schema database is updated according to the data Schema meeting the preset condition. The database log and the message queue real-time data stream are monitored in real time, so that the data Schema can be obtained in real time, the data Schema meeting the preset condition is screened out, the Schema database is automatically updated according to the data Schema meeting the preset condition, the operations of changing program codes manually and the like are reduced to a great extent, the labor cost is reduced, the automatic updating of the data Schema is realized, and the data sharing and use are further better realized.
The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by a person skilled in the art that the scope of the invention as referred to in the present application is not limited to the embodiments with a specific combination of the above-mentioned features, but also covers other embodiments with any combination of the above-mentioned features or their equivalents without departing from the inventive concept. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.
Claims (12)
1. A method for processing data Schema is characterized by comprising the following steps:
acquiring a database log and a message queue real-time data stream;
analyzing the database log and the message queue real-time data stream to obtain a data Schema corresponding to the database log and the message queue real-time data stream;
and updating the Schema database according to the data Schema meeting the preset conditions.
2. The method according to claim 1, wherein the analyzing the database log and the message queue real-time data stream to obtain the data Schema corresponding to the database log and the message queue real-time data stream includes:
analyzing the database log to obtain an operation record in the database log; the operation record comprises creation, deletion and change fields;
analyzing the message queue real-time data stream to obtain a specified data field in the message queue real-time data stream;
and determining the operation record and the data field as the data Schema.
3. The method according to claim 1, wherein the updating the Schema database according to the data Schema meeting the preset condition comprises:
sending a query instruction to the Schema database, and judging whether a data Schema which has the same file name as the data Schema meeting the preset condition exists in the Schema database;
if the data Schema is consistent with the preset condition, updating the data Schema with the same file name as the data Schema consistent with the preset condition according to the data Schema consistent with the preset condition;
if the data does not exist, the data Schema meeting the preset conditions is registered in the Schema database.
4. The method for processing the data Schema according to claim 1, wherein after updating the data Schema database according to the data Schema meeting the preset condition, the method further comprises:
and issuing the data Schema meeting the preset conditions.
5. The method for processing the data Schema according to any one of the claims 1 to 4, wherein before updating the Schema database according to the data Schema meeting the preset condition, the method further comprises:
screening out data schemas meeting preset conditions from the data schemas; the preset conditions include: and the file format of the data Schema conforms to a preset standard format.
6. A device for processing data Schema, the device comprising:
the first acquisition module is used for acquiring a database log and a message queue real-time data stream;
the second obtaining module is configured to analyze the database log and the message queue real-time data stream, and obtain a data Schema corresponding to the database log and the message queue real-time data stream;
and the updating module is used for updating the Schema database according to the data Schema meeting the preset conditions.
7. The apparatus of claim 6, wherein the second obtaining module comprises:
the operation record acquisition unit is used for analyzing the database log and acquiring the operation record in the database log; the operation record comprises creation, deletion and change fields;
a data field obtaining unit, configured to analyze the message queue real-time data stream, and obtain a specified data field in the message queue real-time data stream;
and the determining unit is used for determining the operation record and the data field as the data Schema.
8. The apparatus of claim 6, the update module, comprising:
a judging unit, configured to send a query instruction to the Schema database, and judge whether a data Schema identical to the data Schema file name meeting the preset condition exists in the Schema database;
the updating unit is used for updating the data Schema which is the same as the file name of the data Schema meeting the preset condition when the data Schema meeting the preset condition exists in the Schema database;
and the registration unit is used for registering the data Schema meeting the preset conditions in the Schema database when the data Schema does not exist in the Schema database.
9. The apparatus of claim 6, the apparatus further comprising:
and the release module is used for releasing the data Schema meeting the preset conditions.
10. The apparatus of any of claims 6-9, further comprising:
the screening module is used for screening the data Schema meeting the preset conditions from the data Schema; the preset conditions include: and the file format of the data Schema conforms to a preset standard format.
11. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor realizes the steps of the method according to any of claims 1-5 when executing the computer program.
12. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811194008.0A CN111046050A (en) | 2018-10-12 | 2018-10-12 | Data Schema processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811194008.0A CN111046050A (en) | 2018-10-12 | 2018-10-12 | Data Schema processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111046050A true CN111046050A (en) | 2020-04-21 |
Family
ID=70230353
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811194008.0A Pending CN111046050A (en) | 2018-10-12 | 2018-10-12 | Data Schema processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111046050A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113779018A (en) * | 2020-08-03 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Data processing method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107368517A (en) * | 2017-06-02 | 2017-11-21 | 上海恺英网络科技有限公司 | A kind of method and apparatus of high amount of traffic inquiry |
| CN107885881A (en) * | 2017-11-29 | 2018-04-06 | 顺丰科技有限公司 | Business datum real-time report, acquisition methods, device, equipment and its storage medium |
| CN107995242A (en) * | 2016-10-27 | 2018-05-04 | 北京京东尚科信息技术有限公司 | A kind of method for processing business and system |
| CN108073625A (en) * | 2016-11-14 | 2018-05-25 | 北京京东尚科信息技术有限公司 | For the system and method for metadata information management |
-
2018
- 2018-10-12 CN CN201811194008.0A patent/CN111046050A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107995242A (en) * | 2016-10-27 | 2018-05-04 | 北京京东尚科信息技术有限公司 | A kind of method for processing business and system |
| CN108073625A (en) * | 2016-11-14 | 2018-05-25 | 北京京东尚科信息技术有限公司 | For the system and method for metadata information management |
| CN107368517A (en) * | 2017-06-02 | 2017-11-21 | 上海恺英网络科技有限公司 | A kind of method and apparatus of high amount of traffic inquiry |
| CN107885881A (en) * | 2017-11-29 | 2018-04-06 | 顺丰科技有限公司 | Business datum real-time report, acquisition methods, device, equipment and its storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113779018A (en) * | 2020-08-03 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11249981B2 (en) | Data quality analysis | |
| US20200050588A1 (en) | Automatic file version verification within electronic mail | |
| US9135093B2 (en) | Event-driven approach for collecting monitoring data of messaging systems | |
| CN108764808B (en) | Data analysis processing system and online model deployment method thereof | |
| US20200159742A1 (en) | Maintaining data lineage to detect data events | |
| US8452733B2 (en) | Data decay management | |
| CA3092722A1 (en) | Data retention handling for data object stores | |
| US20160182314A1 (en) | Streamlined provisioning system and method | |
| CN110308917B (en) | Applet issuing method, device, equipment and computer storage medium | |
| US8549539B2 (en) | Programmatic modification of a message flow during runtime | |
| CN112765152B (en) | Method and apparatus for merging data tables | |
| US20170017677A1 (en) | Application event bridge | |
| CN111046050A (en) | Data Schema processing method, device, equipment and storage medium | |
| CN113190517A (en) | Data integration method and device, electronic equipment and computer readable medium | |
| CN114416703A (en) | Method, device, equipment and medium for automatically monitoring data integrity | |
| CN111159142A (en) | Data processing method and device | |
| CN113779017A (en) | Method and apparatus for data asset management | |
| CN114138758A (en) | Data cleaning and merging method and device based on multi-source system and readable medium | |
| US10061773B1 (en) | System and method for processing semi-structured data | |
| CN111723088B (en) | Method and device for pushing summary layer table | |
| CN112130924B (en) | Application system data analysis method and device | |
| CN115309612B (en) | Method and device for monitoring data | |
| CN112181937B (en) | Method and device for transferring data | |
| CN113590425A (en) | Data processing method, apparatus, device, medium, and program product | |
| CN113743842A (en) | Multi-source data difference warning method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |