CN112131291B - Structured analysis method, device and equipment based on JSON data and storage medium - Google Patents
Structured analysis method, device and equipment based on JSON data and storage medium Download PDFInfo
- Publication number
- CN112131291B CN112131291B CN202010956101.1A CN202010956101A CN112131291B CN 112131291 B CN112131291 B CN 112131291B CN 202010956101 A CN202010956101 A CN 202010956101A CN 112131291 B CN112131291 B CN 112131291B
- Authority
- CN
- China
- Prior art keywords
- data
- target
- field
- module
- relation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004458 analytical method Methods 0.000 title description 6
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000013075 data extraction Methods 0.000 claims abstract description 15
- 238000012916 structural analysis Methods 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims description 33
- 238000004590 computer program Methods 0.000 claims description 10
- 238000004140 cleaning Methods 0.000 claims description 9
- 238000003672 processing method Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 8
- 238000000605 extraction Methods 0.000 description 6
- 238000013507 mapping Methods 0.000 description 5
- 238000010606 normalization Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 206010019233 Headaches Diseases 0.000 description 1
- 208000025174 PANDAS Diseases 0.000 description 1
- 208000021155 Paediatric autoimmune neuropsychiatric disorders associated with streptococcal infection Diseases 0.000 description 1
- 240000000220 Panda oleosa Species 0.000 description 1
- 235000016496 Panda oleosa Nutrition 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013506 data mapping Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 231100000869 headache Toxicity 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a structural analysis method, a device, equipment and a storage medium based on JSON data. The data standardization of the JSON data is realized, the problem of diversity of the data is solved, and a more flexible, concise and clear configuration mode is realized, so that the comprehensiveness and stability of data extraction are ensured.
Description
Technical Field
The present application relates to the field of system flow design, and in particular, to a JSON data-based structured analysis method, apparatus, device, and storage medium.
Background
JSON (JavaScript Object Notation) is a lightweight data exchange format, and JSON data belongs to semi-structured data, has a loose data structure and can store more complex data types, so JSON is widely applied to databases such as MongoDB (database based on distributed file storage) and the like to replace relational databases to store data with high concurrency. When targeted mining analysis is performed on mass data, sometimes structural analysis is required to be performed on JSON data, and the JSON data is converted into structural data so as to facilitate data mining analysis.
At present, when the JSON data is structured and analyzed, each JSON data is integrated into a data table mainly by analyzing key and value key value pairs to form structured data. Aiming at the existing structural analysis method of the JSON data, the JSON data is analyzed in a mode of directly extracting key and value key value pairs, when the data size of the JSON data is large, and when the key value is unfixed or path layers are deep, the structural analysis of the JSON data is easy to make errors, so that the efficiency of structural analysis of the JSON data is low.
The JSON structured data mapping system is widely applied in the field of data processing, and has value for enabling us to be more concise and clear when processing data, thereby being beneficial to data analysis. However, there is a very complex problem in the step of data normalization, and the difference in data sources means the complexity of data. The problem of diversity of data is a headache problem when data normalization is performed. Therefore, a more flexible and concise configuration manner is urgently needed to ensure the comprehensiveness and stability of data extraction.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a method, apparatus, device and storage medium for structured parsing based on JSON data.
A structured parsing method based on JSON data, the method comprising: acquiring target JSON data, identifying the data attribute of the target JSON data, and dividing the target JSON data into a plurality of data modules according to the data attribute, wherein each data module corresponds to one data attribute; acquiring a source field composition of the data module, carrying out path configuration on the source field to obtain a target field, and establishing a relation between the source field and the target field to obtain a configuration file; extracting data of the data module to obtain original data, and processing the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed; and merging and cleaning the data to be processed to obtain target data.
In one embodiment, the obtaining the source field of the data module, performing path configuration on the source field to obtain a target field, and after establishing a relationship between the source field and the target field to obtain a configuration file, further includes: according to the relation between the source field and the target field, the data module is divided into three types of simple type, advanced type and complex type, wherein the simple type is that the source field and the target field are in one-to-one correspondence, the advanced type is that one source field corresponds to a plurality of target fields, and the complex type is that a plurality of source fields correspond to a plurality of target fields.
In one embodiment, the extracting the data of the data module obtains original data, and processes the original data according to the relationship between the source field and the target field in the configuration file to obtain data to be processed, specifically: extracting data of the data module to obtain original data; when the data module belongs to a simple class, the source field and the target field are in one-to-one correspondence, and the data to be processed is directly obtained according to the original data.
In one embodiment, the extracting the data of the data module obtains original data, and processes the original data according to the relationship between the source field and the target field in the configuration file to obtain data to be processed, specifically: extracting data of the data module to obtain original data; when the data module belongs to the advanced class, the source field exists in a multi-level and has a one-to-many relation with the target field, and flattening processing is performed on the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed.
In one embodiment, the extracting the data of the data module obtains original data, and processes the original data according to the relationship between the source field and the target field in the configuration file to obtain data to be processed, specifically: extracting data of the data module to obtain original data; when the data module belongs to complex class, the source field exists in multiple levels, the target field also exists in multiple levels, and the original data is processed according to the relation between the source field and the target field in the configuration file, so as to obtain data to be processed.
In one embodiment, after the merging and cleaning of the data to be processed to obtain the target data, the method further includes: and establishing a database table, and filling the target data into the database table to obtain a standardized database table.
The structured analysis device based on JSON data comprises a data dividing module, a path configuration module, a data extraction module and a data processing module, wherein: the data dividing module is used for acquiring target JSON data, identifying the data attribute of the target JSON data, dividing the target JSON data into a plurality of data modules according to the data attribute, wherein each data module corresponds to one data attribute; the path configuration module is used for acquiring the source field composition of the data module, carrying out path configuration on the source field to obtain a target field, and establishing the relation between the source field and the target field to obtain a configuration file; the data extraction module is used for extracting the data of the data module to obtain original data, and processing the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed; the data processing module is used for merging and cleaning the data to be processed to obtain target data.
In one embodiment, the apparatus further comprises a table establishment module, wherein: the table establishing module is used for establishing a database table, and filling the target data into the database table to obtain a standardized database table.
An apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the JSON data-based structured parsing method described in the various embodiments above when the program is executed.
A storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the JSON data-based structured parsing method described in the various embodiments above.
According to the structural analysis method, the device, the equipment and the storage medium based on the JSON data, the JSON data is subjected to module division according to the data attribute to obtain a plurality of data modules, a configuration file of the mapping relation between the source field and the target field is generated for each data module, then the data modules are subjected to data extraction to obtain original data, the original data is flattened according to the configuration file to obtain data to be processed, finally the data to be processed is combined and cleaned to obtain target data, and then the target data is filled into a preset database table to obtain a standardized database table. The data standardization of the JSON data is realized, the problem of diversity of the data is solved, and a more flexible, concise and clear configuration mode is realized, so that the comprehensiveness and stability of data extraction are ensured.
Drawings
FIG. 1 is a flow diagram of a method of structured parsing based on JSON data in one embodiment;
FIG. 2 is a flow diagram of a method of structured parsing based on JSON data in another embodiment;
FIG. 3 is a mapping relationship diagram of source fields and target fields of a simple class in one embodiment;
FIG. 4 is a mapping relationship diagram of source field and target field of an advanced class in one embodiment;
FIG. 5 is a mapping relationship diagram of source fields and target fields of complex classes in one embodiment;
FIG. 6 is a block diagram of a structured parsing apparatus based on JSON data in one embodiment;
fig. 7 is an internal structural diagram of the device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail by the following detailed description with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
In one embodiment, as shown in fig. 2, a structural analysis method based on JSON data is provided, including the following steps:
s110, acquiring target JSON data, identifying the data attribute of the target JSON data, and dividing the target JSON data into a plurality of data modules according to the data attribute, wherein each data module corresponds to one data attribute.
Specifically, first, target JSON data is acquired, then, the data attributes of the target JSON data are identified, multiple data attributes exist in the target JSON data, the target JSON data are divided into multiple data modules according to different data attributes, and each data module corresponds to one data attribute.
S120, acquiring a source field composition of the data module, carrying out path configuration on the source field to obtain a target field, and establishing a relation between the source field and the target field to obtain a configuration file.
Specifically, there are regional differences between the data sources of the modules, and there are also hierarchical differences between the modules, and there may be multiple source_keys (structures) in the same dest_key (target) in the same module. This phenomenon is particularly evident in the declaration information. The declaration information can be largely classified into four types of declarations according to the types of tax payers and the collection items. In the same reporting type, subtle differences exist, reporting patterns presented by the unified reporting type in different years are inconsistent, so that the dest_key correspondence of each module in different areas can be a plurality of source_keys. For clarity, proprietary path profiles need to be generated according to different declaration types.
In one embodiment, step S120 further includes: according to the relation between the source field and the target field, the data module is divided into three types of simple type, advanced type and complex type, wherein the simple type is that the source field corresponds to the target field one by one, the advanced type is that one source field corresponds to a plurality of target fields, and the complex type is that a plurality of source fields correspond to a plurality of target fields. Specifically, according to the complexity of path configuration, the modules are divided into 3 major classes, one is a simple class, one is a progressive class, and one is a complex class, wherein as shown in fig. 3, the paths of the source fields and the target fields of the configuration file of the simple class are in one-to-one correspondence, and the configuration file is directly written; as shown in fig. 4, the advanced class is that there is one source field corresponding to a plurality of target fields, that is, there is one required source field at the parent node peer level of the required detail data field; as shown in fig. 5, the complex class is that there are multiple source fields corresponding to multiple target fields, i.e., there is a one-to-many relationship with the data type of the type and there is a one-to-one relationship.
S130, extracting data of the data module to obtain original data, and processing the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed.
Specifically, there is a multi-level problem for structured data, which is not conducive to data cleaning operation, so that the original data needs to be flattened, and each original data is extracted to the same level by reading the configuration file generated in the first step. When processing advanced type or complex type data, the configuration file needs to be read twice, classification information is read for the first time, and then the configuration file of detail data is loaded on the basis of classification data.
In one embodiment, step S130 is specifically: extracting data of the data module to obtain original data; when the data module belongs to a simple class, the source field and the target field are in one-to-one correspondence, and the data to be processed is directly obtained according to the original data correspondence. Specifically, as shown in fig. 3, when the data module belongs to a simple class, the original data directly corresponds to the data to be processed because the source field and the target field are in a one-to-one correspondence.
In one embodiment, step S130 is specifically: extracting data of the data module to obtain original data; when the data module belongs to the advanced class, the source field exists in a multi-level and has a one-to-many relation with the target field, and flattening processing is carried out on the original data according to the relation between the source field and the target field in the configuration file to obtain the data to be processed. Specifically, as shown in fig. 4, when the data module belongs to the advanced class, the configuration file is made into the same hierarchical structure as the source field, and the first layer generates a configuration capable of screening all records containing the ori_a field or designating ori_a as a specific value; the second layer generates configuration files of other detail fields, the method is the same as simple type, and the essence of the operation is that the original data is flattened according to the configuration files, and then the data to be processed is obtained.
In one embodiment, step S130 is specifically: extracting data of the data module to obtain original data; when the data module belongs to complex class, the source field exists in the multi-level and the target field also exists in the multi-level, and the original data is processed according to the relation between the source field and the target field in the configuration file to obtain the data to be processed. Specifically, as shown in fig. 5, when the data module belongs to a complex class, that is, there is a one-to-many relationship and there is a many-to-one relationship, firstly, the one-to-many relationship is processed according to a processing method of an advanced type, and firstly, a classification configuration is generated according to an ori_a field; aiming at the relation of the detail part in many-to-one mode, flattening the relation by a target field coding mode, such as: the dest_b field may be formed by two fields, namely ori_d and ori_e, where the value of the ori_d path is dest_b_0, the value of the ori_e path is dest_b_1, and if there are more, the number of the ori_d paths is dest_b_n, where n is the corresponding number of the source fields. The other fields are the same as the simple processing method. The essence of the operation is to flatten the original data according to the configuration file and then obtain the data to be processed.
And S140, merging and cleaning the data to be processed to obtain target data.
Specifically, the advanced data module has classification data, and the classification data dest_a needs to be allocated to each piece of corresponding detail record, which can be implemented by using the expode method of pandas. For complex type data modules, after the advanced type data processing is completed, field logic processing is further required, as shown in fig. 5, and new fields dest_b are generated according to the field logic by dest_b_0 and dest_b_1. And finally, performing de-duplication operation on all the data according to the business logic, namely cleaning, and finally obtaining target data, wherein the target data is standardized data, so that the structural analysis of the target JSON data is completed.
In one embodiment, as shown in fig. 2, after step S140, step S150 is further included: and establishing a database table, filling the target data into the database table, and obtaining a standardized database table. Specifically, the target data are stored in the corresponding database tables according to the data modules respectively, so that standardized database tables are obtained. The database table may be preset or built after the target data is obtained.
In the above embodiment, the JSON data is divided into the modules according to the data attribute to obtain a plurality of data modules, a configuration file of the mapping relationship between the source field and the target field is generated for each data module, then the data modules are subjected to data extraction to obtain the original data, the original data is flattened according to the configuration file to obtain the data to be processed, finally the data to be processed is combined and cleaned to obtain the target data, and then the target data is filled into a preset database table to obtain the standardized database table. The data standardization of the JSON data is realized, the problem of diversity of the data is solved, and a more flexible, concise and clear configuration mode is realized, so that the comprehensiveness and stability of data extraction are ensured.
In one embodiment, as shown in fig. 6, there is provided a JSON data-based structured parsing apparatus 200, which includes a data partitioning module 210, a path configuration module 220, a data extraction module 230, and a data processing module 240, wherein:
the data dividing module 210 is configured to obtain target JSON data, identify a data attribute of the target JSON data, and divide the target JSON data into a plurality of data modules according to the data attribute, where each data module corresponds to one data attribute;
the path configuration module 220 is configured to obtain a source field configuration of the data module, perform path configuration on the source field to obtain a target field, and establish a relationship between the source field and the target field to obtain a configuration file;
the data extraction module 230 is configured to extract data of the data module to obtain original data, and process the original data according to a relationship between a source field and a target field in the configuration file to obtain data to be processed;
the data processing module 240 is configured to combine and clean the data to be processed to obtain target data.
In one embodiment, the apparatus further comprises a data classification module, wherein: the data classification module is used for classifying the data module into three types of simple, advanced and complex according to the relation between the source field and the target field, wherein the simple type is that the source field and the target field are in one-to-one correspondence, the advanced type is that one source field corresponds to a plurality of target fields, and the complex type is that a plurality of source fields correspond to a plurality of target fields.
In one embodiment, the data extraction module 230 further comprises an extraction unit and a processing unit, wherein: the extraction unit is used for extracting the data of the data module to obtain original data; and the processing unit is used for directly obtaining the data to be processed according to the one-to-one correspondence of the source field and the target field when the data module belongs to the simple class.
In one embodiment, the data extraction module 230 further comprises an extraction unit and a processing unit, wherein: the extraction unit is used for extracting the data of the data module to obtain original data; and the processing unit is used for flattening the original data according to the relation between the source field and the target field in the configuration file to obtain the data to be processed.
In one embodiment, the data extraction module 230 further comprises an extraction unit and a processing unit, wherein: the extraction unit is used for extracting the data of the data module to obtain original data; the processing unit is used for processing the original data according to the relation between the source field and the target field in the configuration file to obtain the data to be processed when the data module belongs to the complex class and the source field exists in the multiple layers and the target field also exists in the multiple layers.
In one embodiment, the apparatus further comprises a table establishment module, wherein: the table establishing module is used for establishing a database table, filling target data into the database table, and obtaining a standardized database table.
In one embodiment, an apparatus is provided, which may be a server, the internal structure of which may be as shown in fig. 7. The device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the device is configured to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the device is used for storing configuration templates and can also be used for storing target webpage data. The network interface of the device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a structured parsing method based on JSON data.
It will be appreciated by persons skilled in the art that the structure shown in fig. 6 is merely a block diagram of a portion of the structure associated with the present inventive arrangements and does not constitute a limitation of the apparatus to which the present inventive arrangements are applied, and that a particular apparatus may include more or less components than those shown in the drawings, or may combine certain components, or have a different arrangement of components.
In one embodiment, there is also provided a storage medium storing a computer program comprising program instructions which, when executed by a computer, cause the computer to perform a method as described in the previous embodiments, the computer being part of a JSON data-based structured parsing apparatus as referred to above.
Those skilled in the art will appreciate that implementing all or part of the above-described methods in accordance with the embodiments may be accomplished by way of a computer program stored on a computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), or the like.
It will be appreciated by those skilled in the art that the modules or steps of the application described above may be implemented in a general purpose computing device, they may be centralized on a single computing device, or distributed across a network of computing devices, or they may alternatively be implemented in program code executable by computing devices, such that they may be stored on a computer storage medium (ROM/RAM, magnetic or optical disk) for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than what is shown or described herein, or they may be individually manufactured as individual integrated circuit modules, or a plurality of modules or steps in them may be manufactured as a single integrated circuit module. Therefore, the present application is not limited to any specific combination of hardware and software.
The foregoing is a further detailed description of the application in connection with specific embodiments, and is not intended to limit the practice of the application to such descriptions. It will be apparent to those skilled in the art that several simple deductions or substitutions may be made without departing from the spirit of the application, and these should be considered to be within the scope of the application.
Claims (6)
1. The structural analysis method based on JSON data is characterized by comprising the following steps of:
acquiring target JSON data, identifying the data attribute of the target JSON data, and dividing the target JSON data into a plurality of data modules according to the data attribute, wherein each data module corresponds to one data attribute;
acquiring a source field composition of the data module, carrying out path configuration on the source field to obtain a target field, and establishing a relation between the source field and the target field to obtain a configuration file; according to the relation between the source field and the target field, the data module is divided into three types of simple types, advanced types and complex types, wherein the simple types are that the source field and the target field are in one-to-one correspondence, the advanced types are that one source field corresponds to a plurality of target fields, and the complex types are that a plurality of source fields correspond to a plurality of target fields;
extracting data of the data module to obtain original data, and processing the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed; the method comprises the following steps:
when the data module belongs to a simple class, the source field and the target field are in one-to-one correspondence, and the data to be processed is directly obtained according to the original data correspondence;
when the data module belongs to the advanced class, the source field exists in a multi-level and has a one-to-many relation with the target field, and flattening processing is carried out on the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed;
when the data module belongs to complex class, the source field exists in multiple levels, the target field also exists in multiple levels, and the original data is processed according to the relation between the source field and the target field in the configuration file to obtain the data to be processed; specifically: firstly, processing a one-to-many relationship according to a processing method of a step type; aiming at the many-to-one relation of detail parts, flattening the relation by using a target field coding mode, wherein the other fields are the same as a simple processing method;
merging and cleaning the data to be processed to obtain target data; the data module of the advanced type has classification data, and the classification data is required to be distributed to each piece of corresponding detail record; the complex type data module also needs field logic processing after finishing advanced type data processing.
2. The method of claim 1, wherein after the merging and cleaning of the data to be processed to obtain the target data, further comprises:
and establishing a database table, and filling the target data into the database table to obtain a standardized database table.
3. The structured parsing device based on JSON data is characterized by comprising a data dividing module, a path configuration module, a data extraction module and a data processing module, wherein:
the data dividing module is used for acquiring target JSON data, identifying the data attribute of the target JSON data, dividing the target JSON data into a plurality of data modules according to the data attribute, wherein each data module corresponds to one data attribute;
the path configuration module is used for acquiring the source field composition of the data module, carrying out path configuration on the source field to obtain a target field, and establishing the relation between the source field and the target field to obtain a configuration file; according to the relation between the source field and the target field, the data module is divided into three types of simple types, advanced types and complex types, wherein the simple types are that the source field and the target field are in one-to-one correspondence, the advanced types are that one source field corresponds to a plurality of target fields, and the complex types are that a plurality of source fields correspond to a plurality of target fields;
the data extraction module is used for extracting the data of the data module to obtain original data, and processing the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed; the method comprises the following steps:
when the data module belongs to a simple class, the source field and the target field are in one-to-one correspondence, and the data to be processed is directly obtained according to the original data correspondence;
when the data module belongs to the advanced class, the source field exists in a multi-level and has a one-to-many relation with the target field, and flattening processing is carried out on the original data according to the relation between the source field and the target field in the configuration file to obtain data to be processed;
when the data module belongs to complex class, the source field exists in multiple levels, the target field also exists in multiple levels, and the original data is processed according to the relation between the source field and the target field in the configuration file to obtain the data to be processed; specifically: firstly, processing a one-to-many relationship according to a processing method of a step type; aiming at the many-to-one relation of detail parts, flattening the relation by using a target field coding mode, wherein the other fields are the same as a simple processing method;
the data processing module is used for merging and cleaning the data to be processed to obtain target data; the data module of the advanced type has classification data, and the classification data is required to be distributed to each piece of corresponding detail record; the complex type data module also needs field logic processing after finishing advanced type data processing.
4. The apparatus of claim 3, further comprising a table creation module, wherein:
the table establishing module is used for establishing a database table, and filling the target data into the database table to obtain a standardized database table.
5. An apparatus comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method of any one of claims 1 to 2 when the computer program is executed.
6. A storage medium having stored thereon a computer program, which when executed by a processor, implements the steps of the method of any of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010956101.1A CN112131291B (en) | 2020-09-11 | 2020-09-11 | Structured analysis method, device and equipment based on JSON data and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010956101.1A CN112131291B (en) | 2020-09-11 | 2020-09-11 | Structured analysis method, device and equipment based on JSON data and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112131291A CN112131291A (en) | 2020-12-25 |
| CN112131291B true CN112131291B (en) | 2023-12-15 |
Family
ID=73845525
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010956101.1A Active CN112131291B (en) | 2020-09-11 | 2020-09-11 | Structured analysis method, device and equipment based on JSON data and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112131291B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112966299A (en) * | 2021-03-03 | 2021-06-15 | 北京中安星云软件技术有限公司 | Data desensitization system and method based on JSON analysis |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107918865A (en) * | 2017-12-08 | 2018-04-17 | 中国平安财产保险股份有限公司 | Declaration form data correct processing method, device, server and storage medium |
| CN108009282A (en) * | 2017-12-22 | 2018-05-08 | 武汉楚鼎信息技术有限公司 | A kind of json data are synchronized to the method and system device of relevant database |
| CN109299183A (en) * | 2018-11-20 | 2019-02-01 | 北京锐安科技有限公司 | A kind of data processing method, device, terminal device and storage medium |
| CN109325009A (en) * | 2018-09-19 | 2019-02-12 | 亚信科技（成都）有限公司 | The method and device of log parsing |
| CN109753536A (en) * | 2019-01-15 | 2019-05-14 | 顺丰科技有限公司 | A kind of data docking facilities and method |
| CN109766100A (en) * | 2018-12-11 | 2019-05-17 | 新华三技术有限公司合肥分公司 | Data processing method and device |
| CN109819020A (en) * | 2019-01-03 | 2019-05-28 | 福建天泉教育科技有限公司 | Third-party platform based on configurationization logs in interconnection method, storage medium |
| CN110618983A (en) * | 2019-08-15 | 2019-12-27 | 复旦大学 | JSON document structure-based industrial big data multidimensional analysis and visualization method |
| CN110955714A (en) * | 2019-12-03 | 2020-04-03 | 中国银行股份有限公司 | Method and device for converting unstructured text into structured text |
| CN111062189A (en) * | 2018-10-16 | 2020-04-24 | 鸿合科技股份有限公司 | Data analysis method and device and electronic equipment |
| CN111090640A (en) * | 2019-11-13 | 2020-05-01 | 山东中磁视讯股份有限公司 | ETL data cleaning method and system |
| CN111241182A (en) * | 2020-01-19 | 2020-06-05 | 北京奇艺世纪科技有限公司 | Data processing method and apparatus, storage medium, and electronic apparatus |
| CN111552840A (en) * | 2020-05-06 | 2020-08-18 | 山东汇贸电子口岸有限公司 | Method for converting JSON data into tree-shaped hierarchical data |
| CN111563368A (en) * | 2020-04-03 | 2020-08-21 | 江苏苏宁物流有限公司 | Report generation method and device, computer equipment and storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2502531A (en) * | 2012-05-29 | 2013-12-04 | Ibm | De-serializing a source object to a target object |
| US10467243B2 (en) * | 2015-08-26 | 2019-11-05 | Oracle International Corporation | Efficient in-memory DB query processing over any semi-structured data formats |
| CN107992620A (en) * | 2017-12-22 | 2018-05-04 | 武汉楚鼎信息技术有限公司 | A kind of method and system device of json data Fast synchronization and record |
| CN109885569A (en) * | 2018-12-29 | 2019-06-14 | 天津南大通用数据技术股份有限公司 | Field extraction and structural method are carried out to XML data based on configuration file |
| CN111125215A (en) * | 2019-12-06 | 2020-05-08 | 厦门天锐科技股份有限公司 | Method capable of configuring JSON to convert database |
| CN111241060A (en) * | 2020-01-08 | 2020-06-05 | 苏州科达科技股份有限公司 | Data migration method, system, device and storage medium |
-
2020
- 2020-09-11 CN CN202010956101.1A patent/CN112131291B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107918865A (en) * | 2017-12-08 | 2018-04-17 | 中国平安财产保险股份有限公司 | Declaration form data correct processing method, device, server and storage medium |
| CN108009282A (en) * | 2017-12-22 | 2018-05-08 | 武汉楚鼎信息技术有限公司 | A kind of json data are synchronized to the method and system device of relevant database |
| CN109325009A (en) * | 2018-09-19 | 2019-02-12 | 亚信科技（成都）有限公司 | The method and device of log parsing |
| CN111062189A (en) * | 2018-10-16 | 2020-04-24 | 鸿合科技股份有限公司 | Data analysis method and device and electronic equipment |
| CN109299183A (en) * | 2018-11-20 | 2019-02-01 | 北京锐安科技有限公司 | A kind of data processing method, device, terminal device and storage medium |
| CN109766100A (en) * | 2018-12-11 | 2019-05-17 | 新华三技术有限公司合肥分公司 | Data processing method and device |
| CN109819020A (en) * | 2019-01-03 | 2019-05-28 | 福建天泉教育科技有限公司 | Third-party platform based on configurationization logs in interconnection method, storage medium |
| CN109753536A (en) * | 2019-01-15 | 2019-05-14 | 顺丰科技有限公司 | A kind of data docking facilities and method |
| CN110618983A (en) * | 2019-08-15 | 2019-12-27 | 复旦大学 | JSON document structure-based industrial big data multidimensional analysis and visualization method |
| CN111090640A (en) * | 2019-11-13 | 2020-05-01 | 山东中磁视讯股份有限公司 | ETL data cleaning method and system |
| CN110955714A (en) * | 2019-12-03 | 2020-04-03 | 中国银行股份有限公司 | Method and device for converting unstructured text into structured text |
| CN111241182A (en) * | 2020-01-19 | 2020-06-05 | 北京奇艺世纪科技有限公司 | Data processing method and apparatus, storage medium, and electronic apparatus |
| CN111563368A (en) * | 2020-04-03 | 2020-08-21 | 江苏苏宁物流有限公司 | Report generation method and device, computer equipment and storage medium |
| CN111552840A (en) * | 2020-05-06 | 2020-08-18 | 山东汇贸电子口岸有限公司 | Method for converting JSON data into tree-shaped hierarchical data |
Non-Patent Citations (2)
| Title |
|---|
| Dividing Huge XML Trees Using the m-bridge Technique over One-to-one Corresponding Binary Trees;T Kawamura et al.;《Information & Media Technologies》;111-121 * |
| 基于JSON的时态数据建模与查询处理研究;胡章兵;《中国优秀硕士学位论文全文数据库 信息科技辑》;I138-975 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112131291A (en) | 2020-12-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10509804B2 (en) | Method and apparatus for storing sparse graph data as multi-dimensional cluster | |
| EP3259668B1 (en) | System and method for generating an effective test data set for testing big data applications | |
| CN107430611B (en) | Filtering data lineage graph | |
| US11347740B2 (en) | Managed query execution platform, and methods thereof | |
| Karnitis et al. | Migration of relational database to document-oriented database: structure denormalization and data transformation | |
| US8972460B2 (en) | Data model optimization using multi-level entity dependencies | |
| CN110781231B (en) | Database-based batch import method, device, equipment and storage medium | |
| US10303702B2 (en) | System and method for analysis and management of data distribution in a distributed database environment | |
| CN107251021B (en) | Filtering data lineage graph | |
| US10133767B1 (en) | Materialization strategies in journal-based databases | |
| CN115964374B (en) | Query processing method and device based on pre-calculation scene | |
| CN112860777B (en) | Data processing method, device and equipment | |
| CN112199935B (en) | Data comparison method and device, electronic equipment and computer readable storage medium | |
| CN108062314B (en) | Dynamic sub-table data processing method and device | |
| US20190340272A1 (en) | Systems and related methods for updating attributes of nodes and links in a hierarchical data structure | |
| CN112131291B (en) | Structured analysis method, device and equipment based on JSON data and storage medium | |
| CN107168866B (en) | Parameter analysis method and device for configuration file | |
| CN112817984B (en) | Data processing method and device, and data source acquisition method and device | |
| Bante et al. | Big data analytics using hadoop map reduce framework and data migration process | |
| CN113434542A (en) | Data relation identification method and device, electronic equipment and storage medium | |
| CN109241163A (en) | The generation method and terminal device of electronic certificate | |
| CN117609181A (en) | Method and system for migrating TCHouse database | |
| CN116955469A (en) | Service alarm tracing method based on blood margin analysis | |
| CN110888888A (en) | Personnel relationship analysis method and device, electronic equipment and storage medium | |
| CN115658680A (en) | Data storage method, data query method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: 401121 Chongqing Yubei District Huangshan Avenue No. 53 with No. 2 Kirin C Block 9 Floor Patentee after: Chongqing Yucun Technology Co.,Ltd. Country or region after: China Address before: 401121 Chongqing Yubei District Huangshan Avenue No. 53 with No. 2 Kirin C Block 9 Floor Patentee before: CHONGQING SOCIALCREDITS BIG DATA TECHNOLOGY CO.,LTD. Country or region before: China |
|
| CP03 | Change of name, title or address | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Structured parsing method, device, equipment, and storage medium based on JSON data Granted publication date: 20231215 Pledgee: Chongqing Branch of Guangdong Nanyue Bank Co.,Ltd. Pledgor: Chongqing Yucun Technology Co.,Ltd. Registration number: Y2024500000066 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right |