CN113656812A - Data management system and method based on big data technology - Google Patents
Data management system and method based on big data technology Download PDFInfo
- Publication number
- CN113656812A CN113656812A CN202110846525.7A CN202110846525A CN113656812A CN 113656812 A CN113656812 A CN 113656812A CN 202110846525 A CN202110846525 A CN 202110846525A CN 113656812 A CN113656812 A CN 113656812A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- real
- time
- management system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013523 data management Methods 0.000 title claims abstract description 40
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000005516 engineering process Methods 0.000 title claims abstract description 21
- 238000012544 monitoring process Methods 0.000 claims abstract description 12
- 230000002159 abnormal effect Effects 0.000 claims abstract description 11
- 230000008859 change Effects 0.000 claims abstract description 8
- 230000017105 transposition Effects 0.000 claims abstract description 8
- 238000000586 desensitisation Methods 0.000 claims description 22
- 238000007726 management method Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 4
- 238000013480 data collection Methods 0.000 claims description 2
- 230000008676 import Effects 0.000 claims description 2
- 238000009938 salting Methods 0.000 claims description 2
- 238000004519 manufacturing process Methods 0.000 abstract description 6
- 230000008447 perception Effects 0.000 abstract 1
- 238000003860 storage Methods 0.000 description 15
- 238000012545 processing Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 9
- 238000011161 development Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 239000008280 blood Substances 0.000 description 2
- 210000004369 blood Anatomy 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- ZLIBICFPKPWGIZ-UHFFFAOYSA-N pyrimethanil Chemical compound CC1=CC(C)=NC(NC=2C=CC=CC=2)=N1 ZLIBICFPKPWGIZ-UHFFFAOYSA-N 0.000 description 1
- 150000003839 salts Chemical class 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请公开了一种基于大数据技术的数据管理系统和方法，包括：数据采集模块，用于读取数据库后台日志、数据库环境变量信息及读取系统视图并根据业务数据的实时变化，通过数据解析程式分析后台日志和数据库系统视图，通过数据采集需求业务表分析出实时增量数据；数据转置模块，用于按照所述数据采集模块配置的模板将所述实时增量数据接入到大数据环境；源端元数据感知模块，用于当所述实时增量数据的表结构发生变化时，自动感知到表结构的变化，调整数据消息的版本号，通知下游的系统；监控预警模块，用于查看业务数据流的实时流量数量及数据延迟的情况，当数据流量发生异常或数据延迟较大时，设定预警策略；本申请能解决数据生产过程中质量问题。
The present application discloses a data management system and method based on big data technology, including: a data acquisition module for reading database background logs, database environment variable information and reading system views, and according to real-time changes of business data, through data The parsing program analyzes the background log and the database system view, and analyzes the real-time incremental data through the data acquisition demand business table; the data transposition module is used to access the real-time incremental data to the large data according to the template configured by the data acquisition module. data environment; the source-end metadata perception module is used to automatically perceive the change of the table structure when the table structure of the real-time incremental data changes, adjust the version number of the data message, and notify the downstream system; the monitoring and early warning module, It is used to check the real-time traffic volume and data delay of the business data flow. When the data traffic is abnormal or the data delay is relatively large, an early warning strategy is set; this application can solve the quality problem in the data production process.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a data management system and a data management method based on big data technology.
Background
At present, commercial big data acquisition software is few, and the shortage exists, the excellence in log collection and the excellence in RDBMS collection tool do not have effective unified integration and have poor timeliness, a unified management platform does not exist, along with the increase of stock data, the blood relationship of tracking data, the analysis of metadata becomes abnormal and difficult, the problem of data quality can be avoided in the middle of the data production process, the data management difficulty is increased, and the big data development efficiency is not high. Direct blood relationship management of data and data is increasingly difficult, and the main targets of the data management platform are to unify and standardize acquisition, transposition, monitoring, metadata management and data analysis management of internal data of an enterprise, and solve the problem of data quality easily occurring in the data production process. In addition, the large data of the enterprise is difficult to land, which can hinder the digital innovation of the enterprise and is not beneficial to the digital upgrading of the industry.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a data management system and method based on big data technology, which can help enterprises to build applications of a data acquisition platform, improve development efficiency, and solve quality problems in the data production process, aiming at the above defects in the prior art.
In a first aspect, the present application provides a data management system based on big data technology, the system comprising:
the data acquisition module is used for reading the background log of the database, the environment variable information of the database and the system view, analyzing the background log and the system view of the database through a data analysis program according to the real-time change of the service data, and analyzing the real-time incremental data through a data acquisition demand service table;
the data transposition module is used for accessing the real-time incremental data to a big data environment according to the template configured by the data acquisition module;
the source end metadata sensing module is used for automatically sensing the change of the table structure when the table structure of the real-time incremental data changes, adjusting the version number of the data message and informing a downstream system;
and the monitoring and early warning module is used for checking the real-time flow quantity of the service data flow and the data delay condition, and setting an early warning strategy when the data flow is abnormal or the data delay is large.
In an optional implementation manner, the system further includes a login module, configured to provide multi-tenant management, where an administrator has a right to add an account and a password, the added account and password may be used by service personnel, and the service personnel logs in the data management system through the added account and password.
In an optional implementation manner, the system further includes a Web management module, and the Web management module is configured to uniformly manage the login module, the data acquisition module, the data transposition module, the source metadata sensing module, the data desensitization module, and the monitoring and early warning module in a form of a Web system.
In an optional implementation manner, the data acquisition module is further configured to encrypt data to form an encrypted data file; according to the service volume requirement of the data acquisition requirement service table, if the service needs full data acquisition, starting program full acquisition configuration in the encrypted data file, performing full data acquisition, and automatically performing incremental data acquisition after the acquisition is completed.
In an alternative implementation, the data transpose module imports real-time incremental data into a big data environment based on a real-time streaming high availability architecture.
In an optional implementation manner, the data desensitization module is configured to perform real-time desensitization on privacy-sensitive data in the real-time incremental data, where the privacy-sensitive data includes, but is not limited to, an identity card, a system amount, a name, and an account number.
In an alternative implementation, the desensitizing comprises: direct replacement, desensitization salting, regular expression and supporting developers to develop jar package personalized desensitization in a customized manner.
In an optional implementation manner, the monitoring and early warning module is further configured to send an email or a short message to alarm and notify the person responsible for the order in time through the early warning policy when data traffic is abnormal or data delay is large.
In a second aspect, the present application provides a data management method based on big data technology, including the steps of:
reading a database background log, database environment variable information and a reading system view, analyzing the background log and the database system view through a data analysis program according to the real-time change of service data, and analyzing real-time incremental data through a data acquisition demand service table;
accessing the real-time incremental data to a big data environment according to a template configured by the data acquisition module; when the data message is changed, the version number of the data message is adjusted, and a downstream system is informed;
and checking the real-time flow quantity of the service data flow and the data delay condition, and setting an early warning strategy when the data flow is abnormal or the data delay is large.
In a third aspect, an embodiment of the present application provides a computer device including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the data management method according to the second aspect when executing the computer program.
The data management system based on the big data technology helps enterprises to construct a data acquisition platform application, improves development efficiency, helps the enterprises to solve the problem of big data landing in a one-stop manner, helps the enterprises to digitally innovate, promotes industrial digital upgrading, and solves the quality problem in the data production process.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a block diagram of a data management system based on big data technology in the embodiment;
FIG. 2 is a schematic diagram of a login interface of a data management system based on big data technology in the embodiment;
FIG. 3 is a flow chart of a data management method based on big data technology in the embodiment;
FIG. 4 is a block diagram of an example of an electronic device of a data management system based on big data technology in the embodiment;
fig. 5 is a computer readable storage medium of a data management system based on big data technology in the embodiment.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a schematic diagram of a data management system 100 based on a big data technology according to an embodiment of the present invention, where the data management system includes a login module 101, a data collection module 102, a data transpose module 103, a source metadata sensing module 104, a data desensitization module 105, a monitoring and early warning module 106, and a web management module 107.
Specifically, the login module 101 is used for a system administrator to log in to the data acquisition platform through a user name and a password.
By utilizing the same login process used by the service, the user does not need to perform a separate login process in order to login to the application, which is different from the service login process. Once a user logs into an application, the application may perform one or more operations on behalf of the user with respect to the service. The operation may be performed using the identity with which the user has been associated within the application through a "shared" login process.
The data acquisition module 102 is configured to read a database background log, database environment variable information, and a system view, analyze the background log and the database system view through a data analysis program according to real-time changes of service data, analyze real-time incremental data through a data acquisition requirement service table, and support a relational database.
The data transpose module 103 is used for a stream-oriented computing method of a large data processing framework, and can synchronize incremental data and full data. Meanwhile, real-time incremental data can be accessed to a big data environment.
The source metadata sensing module 104 is used for automatically sensing the change of the table structure when the table structure of the data source is changed, adjusting the version number of the data message, and notifying a downstream system through a kafka message and an email.
Kafka is an open source stream processing platform developed by the Apache software foundation, written in Scala and Java. Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all the action flow data of a consumer in a web site. This action (web browsing, searching and other user actions) is a key factor in many social functions on modern networks. These data are typically addressed by handling logs and log aggregations due to throughput requirements. This is a viable solution to the limitations of Hadoop-like log data and offline analysis systems, but which require real-time processing. The purpose of Kafka is to unify online and offline message processing through the parallel loading mechanism of Hadoop, and also to provide real-time messages through clustering.
The data desensitization module 105 is used for desensitizing the identity card, the system amount, the name and the account number to data which is relatively privacy sensitive in the data message in real time. Data desensitization refers to data deformation of some sensitive information through desensitization rules, and reliable protection of sensitive private data is achieved. Sensitive information is shielded by means of a data desensitization technology, and the original data format and attributes of the shielded information are reserved so as to ensure that a system or an application program can normally run in the development and test, data analysis and data trial calculation processes of desensitization data. Data desensitization can be directly replaced, desensitization and salt addition are performed, and a regular expression is adopted, and developers are supported to custom develop jar package personalized desensitization.
The monitoring and early warning module 106 is configured to check the real-time traffic amount of the data stream and the data delay condition, and when the data traffic is abnormal or the data delay is large, configure a rule to set an early warning policy, send a mail or a short message to alarm and notify a person responsible for the order in time.
The Web management module 107 is used for managing a login module, a data acquisition module, a data transposition module, a source metadata learning module, a data desensitization module and a monitoring and early warning module in the form of a Web system.
Referring to fig. 2, a reference schematic diagram is shown for a login method of a data management system based on a big data technology according to an embodiment of the present invention, as shown in fig. 2, a user or an administrator may log in a big data management system platform for operation in a manner of a user name 201 and a password 202, and at the same time, some new users may also log in 204 to the big data management system platform for operation by registering 203 the user name and the password.
Optionally, the provided multi-tenant management can not only manage resources of each tenant, but also provide various services to the tenant, and the tenant first needs to apply for resources, for example: storing resources, computing resources and the like, and normally using various services provided by tenant management after successfully applying for the resources. Based on the above, the resource information may be information such as the resource amount of the storage resource, the resource amount of the computing resource, etc., requested by the target tenant, for example, the resource amount of the database resource, etc.
Please refer to fig. 3, which is a flowchart of a data management method based on big data technology according to an embodiment of the present invention, and as shown in fig. 3, the data management method based on big data technology includes the following steps:
step 301: the method comprises the steps of reading a background log of a database, environment variable information of the database and a system view, analyzing the background log and the system view of the database through a data analysis program according to real-time changes of service data, analyzing real-time incremental data through a data acquisition demand service table, opening program full acquisition configuration in an encrypted data file if the service needs full data acquisition, performing full data acquisition, and automatically performing incremental data acquisition after the acquisition is completed.
Step 302: accessing the real-time incremental data to a big data environment according to a template configured by the data acquisition module; when the data message is changed, the version number of the data message is adjusted, and a downstream system is informed;
step 303: and checking the real-time flow quantity of the service data flow and the data delay condition, and setting an early warning strategy when the data flow is abnormal or the data delay is large. When the data flow is abnormal or the data delay is large, sending mails or short messages to alarm and informing persons responsible for the orders in time through the early warning strategy.
An electronic device 400 according to this embodiment of the invention is described below with reference to fig. 4. The electronic device 400 shown in fig. 4 is only an example and should not bring any limitation to the function and the scope of use of the embodiments of the present invention. As shown in fig. 4, electronic device 400 is embodied in the form of a general purpose computing device. The components of electronic device 400 may include, but are not limited to: the at least one processing unit 410, the at least one memory unit 420, and a bus 430 that couples various system components including the memory unit 420 and the processing unit 410. Wherein the storage unit stores program code that can be executed by the processing unit 410 such that the processing unit 410 performs the steps according to various exemplary embodiments of the present invention as described in the above section "example methods" of the present specification. The storage unit 420 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)421 and/or a cache memory unit 422, and may further include a read only memory unit (ROM) 423. The storage unit 420 may also include a program/utility 424 having a set (at least one) of program modules 425, such program modules 425 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
According to another aspect of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above section "exemplary methods" of the present description, when said program product is run on the terminal device.
As shown in fig. 5, a program product 500 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
Furthermore, the above-described figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
The big data management platform focuses on the aspects of big data management, application development, intelligent visual analysis and the like, helps enterprises to solve the problem of falling of big data in a one-stop manner, and helps the enterprises to digitally innovate and promote industrial digital upgrading. It has the following functions:
the workstation environment of the big data management platform is divided into a hardware environment and a software environment. For a hardware environment, a common PC computer can be selected for work, and the basic configuration is 1) CPU (Central processing Unit) more than Intel core i3, 2) memory more than 1G, 3) hard disk more than 100G, 4) network card (Modem or 10M/100M network card). For a software environment, the following software must be installed in the workstation computer 1) Windows 2003/XP/7/8/8.1; 2) IE 8.0 or higher (google browser is suggested, only needed to open server operations).
In summary, in the data management system based on the big data technology in this embodiment, a login module, a data acquisition module, a data transposition module, a source metadata sensing module, a data desensitization module, a monitoring and early warning module, and a Web management module are provided. By adopting the embodiment of the invention, the enterprise can be helped to construct a data acquisition platform application, the development efficiency is improved, the enterprise is helped to solve the problem of large data landing in a one-stop manner, the enterprise digital innovation is boosted, and the industrial digital upgrade is promoted to solve the quality problem in the data production process.
The foregoing is merely an example of the present invention, and common general knowledge in the field of known specific structures and characteristics is not described herein in any greater extent than that known in the art at the filing date or prior to the priority date of the application, so that those skilled in the art can now appreciate that all of the above-described techniques in this field and have the ability to apply routine experimentation before this date can be combined with one or more of the present teachings to complete and implement the present invention, and that certain typical known structures or known methods do not pose any impediments to the implementation of the present invention by those skilled in the art. It should be noted that, for those skilled in the art, without departing from the structure of the present invention, several changes and modifications can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicability of the patent. The scope of the claims of the present application shall be determined by the contents of the claims, and the description of the embodiments and the like in the specification shall be used to explain the contents of the claims.
Claims (10)
1. A data management system based on big data technology, comprising:
the data acquisition module is used for reading the background log of the database, the environment variable information of the database and the system view, analyzing the background log and the system view of the database through a data analysis program according to the real-time change of the service data, and analyzing the real-time incremental data through a data acquisition demand service table;
the data transposition module is used for accessing the real-time incremental data to a big data environment according to the template configured by the data acquisition module;
the source end metadata sensing module is used for automatically sensing the change of the table structure when the table structure of the real-time incremental data changes, adjusting the version number of the data message and informing a downstream system;
and the monitoring and early warning module is used for checking the real-time flow quantity of the service data flow and the data delay condition, and setting an early warning strategy when the data flow is abnormal or the data delay is large.
2. The data management system of claim 1, further comprising:
and the login module is used for providing multi-tenant management, and an administrator has the authority of adding account and passwords, the added account and passwords can be used by business personnel, and the business personnel log in the data management system through the added account and passwords.
3. The data management system of claim 1, further comprising:
and the Web management module is used for uniformly managing the login module, the data acquisition module, the data transposition module, the source end metadata sensing module, the data desensitization module and the monitoring and early warning module in a form of a Web system.
4. The data management system of claim 1, wherein the data collection module is further configured to encrypt data to form an encrypted data file; according to the service volume requirement of the data acquisition requirement service table, if the service needs full data acquisition, starting program full acquisition configuration in the encrypted data file, performing full data acquisition, and automatically performing incremental data acquisition after the acquisition is completed.
5. The data management system of claim 1, wherein the data transpose module imports real-time incremental data into a big data environment based on a real-time streaming high availability architecture.
6. The data management system of claim 1, further comprising:
and the data desensitization module is used for performing real-time desensitization on privacy-sensitive data in the real-time incremental data, wherein the privacy-sensitive data comprises but is not limited to an identity card, a system amount, a name and an account number.
7. The data management system of claim 6, wherein the desensitization comprises: direct replacement, desensitization salting, regular expression and supporting developers to develop jar package personalized desensitization in a customized manner.
8. The data management system of any one of claims 1 to 7, wherein the monitoring and early warning module is further configured to send an email or a short message to alarm and notify a person in charge of a document in time through the early warning policy when data traffic is abnormal or data delay is large.
9. A data management method based on big data technology is characterized by comprising the following steps:
reading a database background log, database environment variable information and a reading system view, analyzing the background log and the database system view through a data analysis program according to the real-time change of service data, and analyzing real-time incremental data through a data acquisition demand service table;
accessing the real-time incremental data to a big data environment according to a template configured by the data acquisition module; when the data message is changed, the version number of the data message is adjusted, and a downstream system is informed;
and checking the real-time flow quantity of the service data flow and the data delay condition, and setting an early warning strategy when the data flow is abnormal or the data delay is large.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data management method according to claim 9 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110846525.7A CN113656812A (en) | 2021-07-26 | 2021-07-26 | Data management system and method based on big data technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110846525.7A CN113656812A (en) | 2021-07-26 | 2021-07-26 | Data management system and method based on big data technology |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113656812A true CN113656812A (en) | 2021-11-16 |
Family
ID=78478705
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110846525.7A Pending CN113656812A (en) | 2021-07-26 | 2021-07-26 | Data management system and method based on big data technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113656812A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114500615A (en) * | 2022-04-18 | 2022-05-13 | 深圳日晨物联科技有限公司 | Intelligent terminal based on thing allies oneself with sensing technology |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20150136369A (en) * | 2014-05-27 | 2015-12-07 | 주식회사 커머스톤컨설팅 | Integration control system using log security and big-data |
| CN107958158A (en) * | 2017-10-27 | 2018-04-24 | 国网辽宁省电力有限公司 | The dynamic data desensitization method and system of a kind of big data platform |
| CN109885448A (en) * | 2018-12-29 | 2019-06-14 | 中兴智能交通股份有限公司 | A kind of database data increment method for real-time monitoring |
| CN110083647A (en) * | 2019-03-31 | 2019-08-02 | 广州建皓信息技术有限公司 | Big data management platform |
| CN111092852A (en) * | 2019-10-16 | 2020-05-01 | 平安科技（深圳）有限公司 | Network security monitoring method, device, equipment and storage medium based on big data |
-
2021
- 2021-07-26 CN CN202110846525.7A patent/CN113656812A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20150136369A (en) * | 2014-05-27 | 2015-12-07 | 주식회사 커머스톤컨설팅 | Integration control system using log security and big-data |
| CN107958158A (en) * | 2017-10-27 | 2018-04-24 | 国网辽宁省电力有限公司 | The dynamic data desensitization method and system of a kind of big data platform |
| CN109885448A (en) * | 2018-12-29 | 2019-06-14 | 中兴智能交通股份有限公司 | A kind of database data increment method for real-time monitoring |
| CN110083647A (en) * | 2019-03-31 | 2019-08-02 | 广州建皓信息技术有限公司 | Big data management platform |
| CN111092852A (en) * | 2019-10-16 | 2020-05-01 | 平安科技（深圳）有限公司 | Network security monitoring method, device, equipment and storage medium based on big data |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114500615A (en) * | 2022-04-18 | 2022-05-13 | 深圳日晨物联科技有限公司 | Intelligent terminal based on thing allies oneself with sensing technology |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10713664B1 (en) | Automated evaluation and reporting of microservice regulatory compliance | |
| US11449379B2 (en) | Root cause and predictive analyses for technical issues of a computing environment | |
| US20240419433A1 (en) | Methods and systems for application integration and macrosystem aware integration | |
| US11546380B2 (en) | System and method for creation and implementation of data processing workflows using a distributed computational graph | |
| US9418085B1 (en) | Automatic table schema generation | |
| US9363195B2 (en) | Configuring cloud resources | |
| US20240370881A1 (en) | Management of programmatic and compliance workflows using robotic process automation | |
| US10223329B2 (en) | Policy based data collection, processing, and negotiation for analytics | |
| US20210385251A1 (en) | System and methods for integrating datasets and automating transformation workflows using a distributed computational graph | |
| CN113157545A (en) | Method, device and equipment for processing service log and storage medium | |
| US11531539B2 (en) | Automated compliance and testing framework for software development | |
| US11676158B2 (en) | Automatic remediation of non-compliance events | |
| US20220405535A1 (en) | Data log content assessment using machine learning | |
| US20200193058A1 (en) | Enterprise risk, security and compliance automation systems and methods | |
| US11256659B1 (en) | Systems and methods for aggregating and displaying data from multiple data sources | |
| CN111200595B (en) | Authority management method and device for accessing container and electronic equipment | |
| US9246920B2 (en) | Cloud resource cloning based on collaborative content | |
| US20170078138A1 (en) | Real-time tagger | |
| CN113656812A (en) | Data management system and method based on big data technology | |
| US20160379134A1 (en) | Cluster based desktop management services | |
| CN113778818A (en) | Method, apparatus, apparatus, and computer-readable medium for optimizing a system | |
| US20150268986A1 (en) | Distributed batch job promotion within enterprise computing environments | |
| US11880473B2 (en) | Removing data having a data type from a data set | |
| CN115033574A (en) | Information generation method, information generation device, electronic device, and storage medium | |
| US11657030B2 (en) | Multi-dimensional data tagging and reuse |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Country or region after: China Address after: 334000, 5th floor, Zhongke Shuchuang Park, Chasheng East Road, High speed Railway Economic Experimental Zone, Shangrao City, Jiangxi Province Applicant after: Zhongke Digital Economy Research Institute Address before: 334000 3rd floor, building a, Guangxin building, Xinzhou District, Shangrao City, Jiangxi Province Applicant before: INSTITUTE OF BIG DATA, CLOUD COMPUTING CENTER OF THE CHINESE ACADEMY, SHANGRAO Country or region before: China |