CN112131291A - JSON data-based structured analysis method, device, equipment and storage medium - Google Patents
JSON data-based structured analysis method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112131291A CN112131291A CN202010956101.1A CN202010956101A CN112131291A CN 112131291 A CN112131291 A CN 112131291A CN 202010956101 A CN202010956101 A CN 202010956101A CN 112131291 A CN112131291 A CN 112131291A
- Authority
- CN
- China
- Prior art keywords
- data
- target
- module
- field
- source field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 17
- 238000013075 data extraction Methods 0.000 claims abstract description 15
- 238000012545 processing Methods 0.000 claims description 36
- 238000000034 method Methods 0.000 claims description 27
- 238000004140 cleaning Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 11
- 238000010586 diagram Methods 0.000 description 7
- 238000000605 extraction Methods 0.000 description 6
- 238000013507 mapping Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 206010019233 Headaches Diseases 0.000 description 1
- 208000025174 PANDAS Diseases 0.000 description 1
- 208000021155 Paediatric autoimmune neuropsychiatric disorders associated with streptococcal infection Diseases 0.000 description 1
- 240000000220 Panda oleosa Species 0.000 description 1
- 235000016496 Panda oleosa Nutrition 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013506 data mapping Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 231100000869 headache Toxicity 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a JSON data-based structured analytic method, a JSON data-based structured analytic device, JSON data-based structured analytic equipment and a JSON data-based structured analytic storage medium. The data standardization of JSON data is realized, the problem of data diversity is solved, a more flexible, concise and clear configuration mode is realized, and the comprehensiveness and stability of data extraction are ensured.
Description
Technical Field
The invention relates to the field of system flow design, in particular to a JSON data-based structured analysis method, a JSON data-based structured analysis device, JSON data-based structured analysis equipment and a storage medium.
Background
JSON (javascript Object notification) is a lightweight data exchange format, JSON data belongs to semi-structured data, and has a loose data structure, and can store complex data types, so JSON is widely applied to databases such as MongoDB (database based on distributed file storage) and the like to store high-concurrency read-write data instead of a relational database. When targeted mining analysis is performed on mass data, sometimes structured analysis needs to be performed on JSON data, and the JSON data is converted into structured data so as to facilitate data mining analysis.
Currently, when performing structured parsing on JSON data, each piece of JSON data is integrated into a data table mainly by parsing key, value key value pair, to form structured data. Aiming at the existing JSON data structured analysis method, the JSON data is analyzed in a mode of directly extracting key and value key value pairs, when the data volume of the JSON data is large, and the key value is not fixed or the path hierarchy is deep, errors are easy to make in the JSON data structured analysis, and the JSON data structured analysis efficiency is low.
The JSON structured data mapping system is widely applied to the field of data processing, and has the value of enabling people to be more concise and clear when processing data, so that the data analysis is facilitated. However, there is a complicated problem in the step of data standardization, and the data sources are different, which means the complexity of the data. The problem of data diversity is a headache when data standardization is performed. Therefore, a more flexible, concise and clear configuration is urgently needed to ensure the comprehensiveness and stability of data extraction.
Disclosure of Invention
In view of the above, it is necessary to provide a method, an apparatus, a device and a storage medium for structured parsing based on JSON data.
A structured parsing method based on JSON data, the method comprising: acquiring target JSON data, identifying the data attribute of the target JSON data, and dividing the target JSON data into a plurality of data modules according to the data attribute, wherein each data module corresponds to one data attribute; acquiring a source field structure of the data module, performing path configuration on the source field to obtain a target field, and establishing a relationship between the source field and the target field to obtain a configuration file; extracting data of the data module to obtain original data, and processing the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed; and merging and cleaning the data to be processed to obtain target data.
In one embodiment, the obtaining a source field of the data module, performing path configuration on the source field to obtain a target field, establishing a relationship between the source field and the target field, and obtaining a configuration file further includes: and dividing the data module into a simple type, an advanced type and a complex type according to the relation between the source field and the target field, wherein the simple type is that the source field and the target field are in one-to-one correspondence, the advanced type is that one source field exists and corresponds to a plurality of target fields, and the complex type is that a plurality of source fields exist and correspond to a plurality of target fields.
In one embodiment, the extracting data of the data module to obtain original data, and processing the original data according to a relationship between the source field and the target field in the configuration file to obtain to-be-processed data specifically includes: extracting data of the data module to obtain original data; and when the data module belongs to a simple class, the source field and the target field are in one-to-one correspondence, and the data to be processed is directly obtained according to the original data.
In one embodiment, the extracting data of the data module to obtain original data, and processing the original data according to a relationship between the source field and the target field in the configuration file to obtain to-be-processed data specifically includes: extracting data of the data module to obtain original data; and when the data module belongs to an advanced class, the source field exists in multiple levels, a one-to-many relationship exists between the source field and the target field, and the original data is subjected to flattening processing according to the relationship between the source field and the target field in the configuration file to obtain the data to be processed.
In one embodiment, the extracting data of the data module to obtain original data, and processing the original data according to a relationship between the source field and the target field in the configuration file to obtain to-be-processed data specifically includes: extracting data of the data module to obtain original data; and when the data module belongs to a complex class, the source field exists in multiple levels, the target field also exists in multiple levels, and the original data is processed according to the relation between the source field and the target field in the configuration file to obtain the data to be processed.
In one embodiment, after the merging and cleaning the data to be processed to obtain the target data, the method further includes: and establishing a database table, and filling the target data into the database table to obtain a standardized database table.
A JSON data-based structured analysis device comprises a data dividing module, a path configuration module, a data extraction module and a data processing module, wherein: the data dividing module is used for acquiring target JSON data, identifying the data attribute of the target JSON data, and dividing the target JSON data into a plurality of data modules according to the data attribute, wherein each data module corresponds to one data attribute; the path configuration module is used for acquiring the source field structure of the data module, performing path configuration on the source field to obtain a target field, and establishing the relationship between the source field and the target field to obtain a configuration file; the data extraction module is used for extracting the data of the data module to obtain original data, and processing the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed; and the data processing module is used for merging and cleaning the data to be processed to obtain target data.
In one embodiment, the apparatus further comprises a table building module, wherein: the table establishing module is used for establishing a database table, and filling the target data into the database table to obtain a standardized database table.
An apparatus comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the JSON data-based structured parsing method described in the above embodiments.
A storage medium on which a computer program is stored, the program, when executed by a processor, implementing the steps of the JSON data-based structured parsing method described in the various embodiments above.
According to the method, the device, the equipment and the storage medium for structured analysis based on JSON data, the JSON data are subjected to module division according to data attributes to obtain a plurality of data modules, a configuration file of a source field and target field mapping relation is generated for each data module, then data extraction is carried out on the data modules to obtain original data, flattening processing is carried out on the original data according to the configuration file to obtain data to be processed, finally merging and cleaning processing are carried out on the data to be processed to obtain target data, and then the target data are filled into a preset database table to obtain a standardized database table. The data standardization of JSON data is realized, the problem of data diversity is solved, a more flexible, concise and clear configuration mode is realized, and the comprehensiveness and stability of data extraction are ensured.
Drawings
FIG. 1 is a schematic flow chart of a JSON data-based structured parsing method in one embodiment;
FIG. 2 is a schematic flow chart of a JSON data-based structured parsing method in another embodiment;
FIG. 3 is a diagram of a simple class's mapping of source and target fields in one embodiment;
FIG. 4 is a diagram illustrating a mapping between source and destination fields of an advanced class in one embodiment;
FIG. 5 is a diagram of a mapping relationship of source and target fields of a complex class in one embodiment;
FIG. 6 is a block diagram of a JSON data based structured parsing device in one embodiment;
fig. 7 is an internal configuration diagram of the device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings by way of specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In one embodiment, as shown in fig. 2, there is provided a structured parsing method based on JSON data, including the following steps:
s110, target JSON data is obtained, data attributes of the target JSON data are identified, the target JSON data are divided into a plurality of data modules according to the data attributes, and each data module corresponds to one data attribute.
Specifically, target JSON data is obtained firstly, then data attributes of the target JSON data are identified, multiple data attributes exist in the target JSON data, the target JSON data are divided into multiple data modules according to different data attributes, and each data module corresponds to one data attribute.
S120, obtaining the source field structure of the data module, performing path configuration on the source field to obtain a target field, and establishing the relationship between the source field and the target field to obtain a configuration file.
Specifically, the data sources of the modules have regional differences, the modules also have hierarchical differences, and the same dest _ key (target) in the same module may also have multiple source _ keys (components). This phenomenon is particularly evident in the declared information. The declaration information can be roughly divided into four types of declarations according to the difference of taxpayers and the difference of collection items. Slight difference exists in the same declaration type, and declaration types presented by the unified declaration type in different years are inconsistent, so that dest _ key correspondence of each module in different regions may be a plurality of source _ keys. For the sake of clarity, a proprietary path profile needs to be generated based on the different types of declaration.
In one embodiment, step S120 is followed by: and dividing the data module into a simple type, an advanced type and a complex type according to the relation between the source field and the target field, wherein the simple type is that the source field and the target field are in one-to-one correspondence, the advanced type is that one source field corresponds to a plurality of target fields, and the complex type is that a plurality of source fields correspond to a plurality of target fields. Specifically, according to the complexity of the path configuration, several modules are divided into 3 classes, one class is a simple class, one class is an advanced class, and the other class is a complex class, wherein, as shown in fig. 3, paths existing in source fields of configuration files of the simple class are in one-to-one correspondence with target fields, and only the paths are directly written into the configuration files; as shown in fig. 4, the advanced class is that there is one source field corresponding to multiple target fields, i.e. there is one required source field at the parent peer level of the required detail data field; as shown in fig. 5, a complex class is a complex class in which a plurality of source fields correspond to a plurality of target fields, that is, a data type of a type has a one-to-many relationship while a plurality of one-to-one relationships exist.
S130, extracting data of the data module to obtain original data, and processing the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed.
Specifically, the structured data has a multi-level problem, which is not beneficial to data cleaning operation, so that the original data needs to be flattened, and each original data is extracted to the same level by reading the configuration file generated in the first step. When processing advanced type or complex type data, the configuration file needs to be read twice, the classification information is read for the first time, and then the configuration file of the detail data is loaded on the basis of the classification data.
In one embodiment, step S130 specifically includes: extracting data of the data module to obtain original data; and when the data module belongs to the simple class, the source field and the target field are in one-to-one correspondence, and the data to be processed is directly obtained according to the original data. Specifically, as shown in fig. 3, when the data module belongs to a simple class, the original data directly corresponds to the data to be processed because the source field and the target field are in a one-to-one correspondence relationship.
In one embodiment, step S130 specifically includes: extracting data of the data module to obtain original data; and when the data module belongs to the advanced class, the source field exists in multiple levels and has a one-to-many relation with the target field, and the original data is subjected to flattening processing according to the relation between the source field and the target field in the configuration file to obtain the data to be processed. Specifically, as shown in fig. 4, when the data module belongs to the advanced class, the configuration file is made into the same hierarchical structure as the source field, and the first layer generates a configuration capable of screening out all records containing an Ori _ a field or specifying Ori _ a as a specific value; the second layer generates configuration files of other detail fields, the method is the same as a simple type, and the operation is essentially to flatten the original data according to the configuration files and then obtain the data to be processed.
In one embodiment, step S130 specifically includes: extracting data of the data module to obtain original data; and when the data module belongs to a complex class, the source field exists in multiple levels, the target field also exists in multiple levels, and the original data is processed according to the relation between the source field and the target field in the configuration file to obtain the data to be processed. Specifically, as shown in fig. 5, when the data module belongs to a complex class, that is, there is a one-to-many relationship and a many-to-one relationship, firstly, the one-to-many relationship is processed according to an advanced type processing method, and firstly, a classification configuration is generated according to the Ori _ a field; aiming at the many-to-one relation of the detail part, the detail part is flattened by a target field coding mode, such as: the Dest _ B field may be composed of two fields, Ori _ D and Ori _ E, and records in the configuration file that the value of the Ori _ D path is Dest _ B _0, the value of the Ori _ E path is Dest _ B _1, and if there are more, the label is Dest _ B _ n once, where n is the number corresponding to the source field. The rest fields are the same as the simple processing method. The essence of such operation is to flatten the original data according to the configuration file, and then obtain the data to be processed.
S140, merging and cleaning the data to be processed to obtain target data.
Specifically, classification data exists in the advanced type data module, and classification data Dest _ a needs to be allocated to each corresponding detail record, which can be implemented by the expode method of pandas. For a complex type data module, after completing the advanced type data processing, field logic processing is also required, and as shown in fig. 5, a new field Dest _ B is generated from Dest _ B _0 and Dest _ B _1 according to the field logic. And finally, performing deduplication operation, namely cleaning, on all the data according to the service logic, and finally obtaining target data, wherein the target data is standardized data, so that the structured analysis of the target JSON data is completed.
In one embodiment, as shown in fig. 2, after step S140, step S150 is further included: and establishing a database table, and filling the target data into the database table to obtain a standardized database table. Specifically, the target data is stored in the corresponding database tables according to each data module, so that the standardized database tables are obtained. The database table may be preset, or may be established after the target data is obtained.
In the embodiment, the JSON data is divided into modules according to data attributes to obtain a plurality of data modules, a configuration file of a source field and target field mapping relationship is generated for each data module, data extraction is performed on the data modules to obtain original data, flattening processing is performed on the original data according to the configuration file to obtain data to be processed, merging and cleaning processing are performed on the data to be processed to obtain target data, and the target data is filled into a preset database table to obtain a standardized database table. The data standardization of JSON data is realized, the problem of data diversity is solved, a more flexible, concise and clear configuration mode is realized, and the comprehensiveness and stability of data extraction are ensured.
In one embodiment, as shown in fig. 6, there is provided a JSON data-based structured parsing apparatus 200, which includes a data dividing module 210, a path configuration module 220, a data extraction module 230, and a data processing module 240, wherein:
the data dividing module 210 is configured to obtain target JSON data, identify a data attribute of the target JSON data, and divide the target JSON data into a plurality of data modules according to the data attribute, where each data module corresponds to one data attribute;
the path configuration module 220 is configured to obtain a source field configuration of the data module, perform path configuration on the source field to obtain a target field, and establish a relationship between the source field and the target field to obtain a configuration file;
the data extraction module 230 is configured to extract data of the data module to obtain original data, and process the original data according to a relationship between a source field and a target field in a configuration file to obtain data to be processed;
the data processing module 240 is configured to merge and clean the data to be processed to obtain target data.
In one embodiment, the apparatus further comprises a data classification module, wherein: the data classification module is used for dividing the data module into a simple type, an advanced type and a complex type according to the relation between the source field and the target field, wherein the simple type is that the source field and the target field are in one-to-one correspondence, the advanced type is that one source field corresponds to a plurality of target fields, and the complex type is that a plurality of source fields correspond to a plurality of target fields.
In one embodiment, the data extraction module 230 further comprises an extraction unit and a processing unit, wherein: the extraction unit is used for extracting the data of the data module to obtain original data; and the processing unit is used for obtaining the data to be processed directly according to the original data in a one-to-one correspondence relationship between the source field and the target field when the data module belongs to the simple class.
In one embodiment, the data extraction module 230 further comprises an extraction unit and a processing unit, wherein: the extraction unit is used for extracting the data of the data module to obtain original data; and the processing unit is used for flattening the original data according to the relationship between the source field and the target field in the configuration file to obtain the data to be processed.
In one embodiment, the data extraction module 230 further comprises an extraction unit and a processing unit, wherein: the extraction unit is used for extracting the data of the data module to obtain original data; and the processing unit is used for processing the original data according to the relation between the source field and the target field in the configuration file to obtain the data to be processed when the data module belongs to the complex class and the source field and the target field exist in the multiple hierarchies.
In one embodiment, the apparatus further comprises a table building module, wherein: the table establishing module is used for establishing a database table, and filling the target data into the database table to obtain a standardized database table.
In one embodiment, a device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 7. The device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the device is configured to provide computing and control capabilities. The memory of the device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the device is used for storing configuration templates and also can be used for storing target webpage data. The network interface of the device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a structured parsing method based on JSON data.
Those skilled in the art will appreciate that the configuration shown in fig. 6 is a block diagram of only a portion of the configuration associated with the present application and does not constitute a limitation on the devices to which the present application may be applied, and that a particular device may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a storage medium is further provided, which stores a computer program comprising program instructions, which when executed by a computer, which may be part of the above-mentioned JSON data-based structured parsing apparatus, cause the computer to perform the method according to the preceding embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
It will be apparent to those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be centralized on a single computing device or distributed across a network of computing devices, and optionally they may be implemented in program code executable by a computing device, such that they may be stored on a computer storage medium (ROM/RAM, magnetic disks, optical disks) and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The foregoing is a more detailed description of the present invention that is presented in conjunction with specific embodiments, and the practice of the invention is not to be considered limited to those descriptions. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (10)
1. A structured analytic method based on JSON data is characterized by comprising the following steps:
acquiring target JSON data, identifying the data attribute of the target JSON data, and dividing the target JSON data into a plurality of data modules according to the data attribute, wherein each data module corresponds to one data attribute;
acquiring a source field structure of the data module, performing path configuration on the source field to obtain a target field, and establishing a relationship between the source field and the target field to obtain a configuration file;
extracting data of the data module to obtain original data, and processing the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed;
and merging and cleaning the data to be processed to obtain target data.
2. The method of claim 1, wherein the obtaining of the source field structure of the data module, performing path configuration on the source field to obtain a target field, establishing a relationship between the source field and the target field, and obtaining a configuration file further comprises:
and dividing the data module into a simple type, an advanced type and a complex type according to the relation between the source field and the target field, wherein the simple type is that the source field and the target field are in one-to-one correspondence, the advanced type is that one source field exists and corresponds to a plurality of target fields, and the complex type is that a plurality of source fields exist and correspond to a plurality of target fields.
3. The method according to claim 2, wherein the extracting data of the data module obtains original data, and the processing is performed on the original data according to a relationship between the source field and the target field in the configuration file to obtain data to be processed, specifically:
extracting data of the data module to obtain original data;
and when the data module belongs to a simple class, the source field and the target field are in one-to-one correspondence, and the data to be processed is directly obtained according to the original data.
4. The method according to claim 2, wherein the extracting data of the data module obtains original data, and the processing is performed on the original data according to a relationship between the source field and the target field in the configuration file to obtain data to be processed, specifically:
extracting data of the data module to obtain original data;
and when the data module belongs to an advanced class, the source field exists in multiple levels, a one-to-many relationship exists between the source field and the target field, and the original data is subjected to flattening processing according to the relationship between the source field and the target field in the configuration file to obtain the data to be processed.
5. The method according to claim 2, wherein the extracting data of the data module obtains original data, and the processing is performed on the original data according to a relationship between the source field and the target field in the configuration file to obtain data to be processed, specifically:
extracting data of the data module to obtain original data;
and when the data module belongs to a complex class, the source field exists in multiple levels, the target field also exists in multiple levels, and the original data is processed according to the relation between the source field and the target field in the configuration file to obtain the data to be processed.
6. The method of claim 1, wherein after the merging and cleaning the data to be processed to obtain the target data, the method further comprises:
and establishing a database table, and filling the target data into the database table to obtain a standardized database table.
7. The structured analysis device based on JSON data is characterized by comprising a data dividing module, a path configuration module, a data extraction module and a data processing module, wherein:
the data dividing module is used for acquiring target JSON data, identifying the data attribute of the target JSON data, and dividing the target JSON data into a plurality of data modules according to the data attribute, wherein each data module corresponds to one data attribute;
the path configuration module is used for acquiring the source field structure of the data module, performing path configuration on the source field to obtain a target field, and establishing the relationship between the source field and the target field to obtain a configuration file;
the data extraction module is used for extracting the data of the data module to obtain original data, and processing the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed;
and the data processing module is used for merging and cleaning the data to be processed to obtain target data.
8. The apparatus of claim 7, further comprising a table building module, wherein:
the table establishing module is used for establishing a database table, and filling the target data into the database table to obtain a standardized database table.
9. An apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 6 are implemented when the computer program is executed by the processor.
10. A storage medium having a computer program stored thereon, the computer program, when being executed by a processor, realizing the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010956101.1A CN112131291B (en) | 2020-09-11 | 2020-09-11 | Structured analysis method, device and equipment based on JSON data and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010956101.1A CN112131291B (en) | 2020-09-11 | 2020-09-11 | Structured analysis method, device and equipment based on JSON data and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112131291A true CN112131291A (en) | 2020-12-25 |
| CN112131291B CN112131291B (en) | 2023-12-15 |
Family
ID=73845525
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010956101.1A Active CN112131291B (en) | 2020-09-11 | 2020-09-11 | Structured analysis method, device and equipment based on JSON data and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112131291B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112966299A (en) * | 2021-03-03 | 2021-06-15 | 北京中安星云软件技术有限公司 | Data desensitization system and method based on JSON analysis |
Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130326491A1 (en) * | 2012-05-29 | 2013-12-05 | International Business Machines Corporation | De-Serialization of Objects Based on Difference in Information Content |
| US20170060973A1 (en) * | 2015-08-26 | 2017-03-02 | Oracle International Corporation | Efficient in-memory db query processing over any semi-structured data formats |
| CN107918865A (en) * | 2017-12-08 | 2018-04-17 | 中国平安财产保险股份有限公司 | Declaration form data correct processing method, device, server and storage medium |
| CN107992620A (en) * | 2017-12-22 | 2018-05-04 | 武汉楚鼎信息技术有限公司 | A kind of method and system device of json data Fast synchronization and record |
| CN108009282A (en) * | 2017-12-22 | 2018-05-08 | 武汉楚鼎信息技术有限公司 | A kind of json data are synchronized to the method and system device of relevant database |
| CN109299183A (en) * | 2018-11-20 | 2019-02-01 | 北京锐安科技有限公司 | A kind of data processing method, device, terminal device and storage medium |
| CN109325009A (en) * | 2018-09-19 | 2019-02-12 | 亚信科技（成都）有限公司 | The method and device of log parsing |
| CN109753536A (en) * | 2019-01-15 | 2019-05-14 | 顺丰科技有限公司 | A kind of data docking facilities and method |
| CN109766100A (en) * | 2018-12-11 | 2019-05-17 | 新华三技术有限公司合肥分公司 | Data processing method and device |
| CN109819020A (en) * | 2019-01-03 | 2019-05-28 | 福建天泉教育科技有限公司 | Third-party platform based on configurationization logs in interconnection method, storage medium |
| CN109885569A (en) * | 2018-12-29 | 2019-06-14 | 天津南大通用数据技术股份有限公司 | Field extraction and structural method are carried out to XML data based on configuration file |
| CN110618983A (en) * | 2019-08-15 | 2019-12-27 | 复旦大学 | JSON document structure-based industrial big data multidimensional analysis and visualization method |
| CN110955714A (en) * | 2019-12-03 | 2020-04-03 | 中国银行股份有限公司 | Method and device for converting unstructured text into structured text |
| CN111062189A (en) * | 2018-10-16 | 2020-04-24 | 鸿合科技股份有限公司 | Data analysis method and device and electronic equipment |
| CN111090640A (en) * | 2019-11-13 | 2020-05-01 | 山东中磁视讯股份有限公司 | ETL data cleaning method and system |
| CN111125215A (en) * | 2019-12-06 | 2020-05-08 | 厦门天锐科技股份有限公司 | Method capable of configuring JSON to convert database |
| CN111241060A (en) * | 2020-01-08 | 2020-06-05 | 苏州科达科技股份有限公司 | Data migration method, system, device and storage medium |
| CN111241182A (en) * | 2020-01-19 | 2020-06-05 | 北京奇艺世纪科技有限公司 | Data processing method and apparatus, storage medium, and electronic apparatus |
| CN111552840A (en) * | 2020-05-06 | 2020-08-18 | 山东汇贸电子口岸有限公司 | Method for converting JSON data into tree-shaped hierarchical data |
| CN111563368A (en) * | 2020-04-03 | 2020-08-21 | 江苏苏宁物流有限公司 | Report generation method and device, computer equipment and storage medium |
-
2020
- 2020-09-11 CN CN202010956101.1A patent/CN112131291B/en active Active
Patent Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130326491A1 (en) * | 2012-05-29 | 2013-12-05 | International Business Machines Corporation | De-Serialization of Objects Based on Difference in Information Content |
| US20170060973A1 (en) * | 2015-08-26 | 2017-03-02 | Oracle International Corporation | Efficient in-memory db query processing over any semi-structured data formats |
| CN107918865A (en) * | 2017-12-08 | 2018-04-17 | 中国平安财产保险股份有限公司 | Declaration form data correct processing method, device, server and storage medium |
| CN107992620A (en) * | 2017-12-22 | 2018-05-04 | 武汉楚鼎信息技术有限公司 | A kind of method and system device of json data Fast synchronization and record |
| CN108009282A (en) * | 2017-12-22 | 2018-05-08 | 武汉楚鼎信息技术有限公司 | A kind of json data are synchronized to the method and system device of relevant database |
| CN109325009A (en) * | 2018-09-19 | 2019-02-12 | 亚信科技（成都）有限公司 | The method and device of log parsing |
| CN111062189A (en) * | 2018-10-16 | 2020-04-24 | 鸿合科技股份有限公司 | Data analysis method and device and electronic equipment |
| CN109299183A (en) * | 2018-11-20 | 2019-02-01 | 北京锐安科技有限公司 | A kind of data processing method, device, terminal device and storage medium |
| CN109766100A (en) * | 2018-12-11 | 2019-05-17 | 新华三技术有限公司合肥分公司 | Data processing method and device |
| CN109885569A (en) * | 2018-12-29 | 2019-06-14 | 天津南大通用数据技术股份有限公司 | Field extraction and structural method are carried out to XML data based on configuration file |
| CN109819020A (en) * | 2019-01-03 | 2019-05-28 | 福建天泉教育科技有限公司 | Third-party platform based on configurationization logs in interconnection method, storage medium |
| CN109753536A (en) * | 2019-01-15 | 2019-05-14 | 顺丰科技有限公司 | A kind of data docking facilities and method |
| CN110618983A (en) * | 2019-08-15 | 2019-12-27 | 复旦大学 | JSON document structure-based industrial big data multidimensional analysis and visualization method |
| CN111090640A (en) * | 2019-11-13 | 2020-05-01 | 山东中磁视讯股份有限公司 | ETL data cleaning method and system |
| CN110955714A (en) * | 2019-12-03 | 2020-04-03 | 中国银行股份有限公司 | Method and device for converting unstructured text into structured text |
| CN111125215A (en) * | 2019-12-06 | 2020-05-08 | 厦门天锐科技股份有限公司 | Method capable of configuring JSON to convert database |
| CN111241060A (en) * | 2020-01-08 | 2020-06-05 | 苏州科达科技股份有限公司 | Data migration method, system, device and storage medium |
| CN111241182A (en) * | 2020-01-19 | 2020-06-05 | 北京奇艺世纪科技有限公司 | Data processing method and apparatus, storage medium, and electronic apparatus |
| CN111563368A (en) * | 2020-04-03 | 2020-08-21 | 江苏苏宁物流有限公司 | Report generation method and device, computer equipment and storage medium |
| CN111552840A (en) * | 2020-05-06 | 2020-08-18 | 山东汇贸电子口岸有限公司 | Method for converting JSON data into tree-shaped hierarchical data |
Non-Patent Citations (2)
| Title |
|---|
| T KAWAMURA ET AL.: "Dividing Huge XML Trees Using the m-bridge Technique over One-to-one Corresponding Binary Trees", 《INFORMATION & MEDIA TECHNOLOGIES》, pages 111 - 121 * |
| 胡章兵: "基于JSON的时态数据建模与查询处理研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》, pages 138 - 975 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112966299A (en) * | 2021-03-03 | 2021-06-15 | 北京中安星云软件技术有限公司 | Data desensitization system and method based on JSON analysis |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112131291B (en) | 2023-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107958057B (en) | Code generation method and device for data migration in heterogeneous database | |
| US10509804B2 (en) | Method and apparatus for storing sparse graph data as multi-dimensional cluster | |
| US9607063B1 (en) | NoSQL relational database (RDB) data movement | |
| AU2015315203B2 (en) | Conditional validation rules | |
| US11347740B2 (en) | Managed query execution platform, and methods thereof | |
| CN106033437A (en) | Method and system for processing distributed transaction | |
| US9619492B2 (en) | Data migration | |
| CN110019116B (en) | Data tracing method, device, data processing equipment and computer storage medium | |
| CN112199935B (en) | Data comparison method and device, electronic equipment and computer readable storage medium | |
| US10031936B2 (en) | Database table data fabrication | |
| US10235401B2 (en) | Method and system for handling binary large objects | |
| CN107168866B (en) | Parameter analysis method and device for configuration file | |
| US11797487B2 (en) | Maintaining stable record identifiers in the presence of updated data records | |
| US20180336171A1 (en) | System and method for constructing extensible event log with javascript object notation (json) encoded payload data | |
| CN112131291A (en) | JSON data-based structured analysis method, device, equipment and storage medium | |
| CN113886419A (en) | SQL statement processing method and device, computer equipment and storage medium | |
| US9805091B2 (en) | Processing a database table | |
| CN110928941B (en) | Data fragment extraction method and device | |
| CN109522915B (en) | Virus file clustering method and device and readable medium | |
| CN108920256A (en) | Check task executing method and device | |
| CN114356454A (en) | Account checking data processing method, account checking data processing device, account checking data storage medium and program product | |
| CN106557564A (en) | A kind of object data analysis method and device | |
| US20200273351A1 (en) | Methods and Systems for Data Parameter Reasonability Analysis | |
| CN116610664B (en) | Data monitoring method, device, computer equipment, storage medium and product | |
| CN110888929B (en) | Data processing method, data processing device, data node and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 401121 Chongqing Yubei District Huangshan Avenue No. 53 with No. 2 Kirin C Block 9 Floor Patentee after: Chongqing Yucun Technology Co.,Ltd. Country or region after: China Address before: 401121 Chongqing Yubei District Huangshan Avenue No. 53 with No. 2 Kirin C Block 9 Floor Patentee before: CHONGQING SOCIALCREDITS BIG DATA TECHNOLOGY CO.,LTD. Country or region before: China |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Structured parsing method, device, equipment, and storage medium based on JSON data Granted publication date: 20231215 Pledgee: Chongqing Branch of Guangdong Nanyue Bank Co.,Ltd. Pledgor: Chongqing Yucun Technology Co.,Ltd. Registration number: Y2024500000066 |