CN109344193B

CN109344193B - Method and system for converting structured data into unstructured data

Info

Publication number: CN109344193B
Application number: CN201811244279.2A
Authority: CN
Inventors: 翁星晨; 朱晓峰
Original assignee: Bank of China Ltd
Current assignee: Bank of China Ltd
Priority date: 2018-10-24
Filing date: 2018-10-24
Publication date: 2020-09-15
Anticipated expiration: 2038-10-24
Also published as: CN109344193A

Abstract

The application discloses a method and a system for converting structured data into unstructured data, wherein the method comprises the following steps: and configuring a JSON template, analyzing the JSON template to a structure array, and reading an array source according to the structure array to obtain a JSON file. The method and the device can efficiently and accurately complete the conversion from the structured data to the unstructured data format.

Description

Method and system for converting structured data into unstructured data

Technical Field

The present application relates to the field of data transformation technologies, and in particular, to a method and a system for transforming structured data into unstructured data.

Background

At present, in a development project using a non-relational database such as a database based on distributed file storage, it is often related to importing of bottom data or code tables, and structured data, such as tables, needs to be converted into a format of non-relational database expenditure, such as JSON (JavaScript Object Notation, JS Object Notation, a lightweight data exchange format). The method commonly used at present is manual conversion, the repeated work efficiency of copying and pasting is very low, and errors are easy to occur. And if the traditional method for converting unstructured data is used, most of the unstructured data is strongly coupled with the JSON message, and codes are required to be modified for many times when the JSON format is required to be modified.

Therefore, how to efficiently and accurately complete the conversion from the structured data to the unstructured data format is an urgent problem to be solved.

Disclosure of Invention

In view of this, the present application provides a method for converting structured data into unstructured data, which can efficiently and accurately complete the conversion from structured data to unstructured data format.

The present application provides a method of converting structured data into unstructured data, the method comprising:

configuring a JSON template;

analyzing the JSON template to a structure array;

and reading an array source according to the structure array to obtain the JSON file.

Preferably, the parsing the JSON template into a structure array includes:

and analyzing the JSON template to a structure array through a recursive function.

Preferably, the reading the array source according to the structure array to obtain the JSON file includes:

and reading data from the table according to the structure array, and generating a JSON file.

Preferably, a first node in the JSON template is used as a main index, the main index corresponds to a main field of a main table, and each line in the main table corresponds to each line of the JSON file.

and reading data from a database according to the structure array to generate a JSON file.

A system for converting structured data into unstructured data, comprising:

the configuration module is used for configuring the JSON template;

the analysis module is used for analyzing the JSON template to a structure array;

and the conversion module is used for reading the array source according to the structure array to obtain the JSON file.

Preferably, the parsing module is specifically configured to:

Preferably, the conversion module is specifically configured to:

In summary, the present application discloses a method for converting structured data into unstructured data, where when the structured data needs to be converted into unstructured data, a JSON template is configured first, then the JSON template is parsed to a structure array, and finally an array source is read according to the structure array to obtain a JSON file. The method and the device can efficiently and accurately complete the conversion from the structured data to the unstructured data format.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a flow chart of a method of embodiment 1 of a method of converting structured data to unstructured data as disclosed herein;

FIG. 2 is a flow chart of a method of embodiment 2 of a method of converting structured data to unstructured data as disclosed herein;

FIG. 3 is a flowchart of a method of embodiment 3 of a method of converting structured data to unstructured data as disclosed herein;

FIG. 4 is a flowchart of an algorithm disclosed herein for generating JSON files;

FIG. 5 is a schematic diagram of an embodiment 1 of a system for converting structured data into unstructured data according to the present disclosure;

FIG. 6 is a schematic diagram of the structure of example 2 of a system for converting structured data into unstructured data according to the present disclosure;

FIG. 7 is a schematic structural diagram of example 3 of a system for converting structured data into unstructured data as disclosed herein.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

As shown in fig. 1, a method flow diagram of embodiment 1 of a method for converting structured data into unstructured data is provided, and the method may include the following steps:

s101, configuring a JSON template;

the structured data, namely the row data, is stored in a database, and the data can be logically expressed by a two-dimensional table structure; the structure definition is not easy to change, and the data is fixed in length. Unstructured data, i.e., data that is not conveniently represented by a database two-dimensional logical table. The fields thereof are of variable length and the record of each field may in turn be composed of sub-fields that may or may not be repeatable. When structured data needs to be converted into unstructured data, a required JSON template is defined first.

S102, analyzing the JSON template to a structure array;

and after defining the needed JSON template, further analyzing the JSON template to a structure array.

S103, reading an array source according to the structure array to obtain the JSON file.

And then reading an array source according to the obtained structure array, namely reading data needing to be converted, and converting to obtain the JSON file.

In summary, in the above embodiments, when structured data needs to be converted into unstructured data, a JSON template is configured first, then the JSON template is parsed to a structure array, and finally an array source is read according to the structure array to obtain a JSON file. The method and the device can efficiently and accurately complete the conversion from the structured data to the unstructured data format.

As shown in fig. 2, a method flow diagram of embodiment 2 of a method for converting structured data into unstructured data provided for application may include the following steps:

s201, configuring a JSON template;

S202, resolving the JSON template to a structure array;

And S203, reading data from the database according to the structure array, and generating a JSON file.

And then reading an array source from the database according to the obtained structure array, namely reading data needing to be converted, and converting to obtain the JSON file.

In summary, in the above embodiments, when structured data needs to be converted into unstructured data, a JSON template is configured first, then the JSON template is parsed into a structure array, and finally data is read from a database according to the structure array, so as to obtain a JSON file. The method and the device can efficiently and accurately complete the conversion from the structured data to the unstructured data format.

As shown in fig. 3, a method for converting structured data into unstructured data provided for the application embodiment 3 is a flow chart of the method, and the method may include the following steps:

s301, configuring a JSON template;

The form is as follows (taking excel as an example of a data source):

{ "classNm": "expenditure (main table) [3]", "classifyId": "expenditure (main table) [4]", "isShow": "expenditure (main table) [5]", "keyword": keyword [2] "," title ":" expenditure (main table) [2] "," tranCat ": expenditure (main table) [1]", "MCC": [ { "mccName": MCC [1] "," code ": MCC [2] }," tranTypes "[ {" trancode ": transaction type [1]", "traDesc": transaction type [2], "editFlag": transaction type [3] }, ")

Wherein, the corresponding value of the label is: the table name [ column number ] indicates the data source location of the tag.

S302, analyzing the JSON template to a structure array through a recursive function;

after defining the needed JSON template, further analyzing the JSON template to a structure array through a recursive function:

the structure array is defined as:

Public Type jsonspot

spotno As String' node numbering, 1,2,3

spotype As String' node type: m: intermediate node middle L: leaf node leaf

spotdepth As String' node hierarchy: 1-first layer, 2-second layer.

The spotlable As String' node tag value corresponds to the json element tag value

parentspot As String' parent node number: 0: root node, 1: node 1.

The sheetname As String' leaf node uses: the name of the sheet where the node is located, or the name of the table

columeno As String' leaf node use: the column name of the node or the name of colony

The extrakeycolumeno As String' leaf node uses: in the sheet where the leaf node is located, the column where the main label is located

spotvaluetype As String' node value type: s: character N: numerical value SA: character array

O: object NA numerical array OA object array

End Type

With the above template example: the simplified structure of the structure data obtained by recursive analysis of the template is shown in the following table:

"classNm"
		"classifyId"
"merch"
		"keyword"
"title"
		"tranCat"
″mcc″	"mccName"
			"code"
"tranTypes"	"trancode"
			"editFlag"

the first column is the first layer, the second column is the second layer, and so on. The front column in the same row is the parent node of the back column.

The complete structure is as follows:

and S303, reading data from the table according to the structure array, and generating a JSON file.

And taking the first node of the JSON template as a main index, wherein the main index corresponds to a main field of a main table, and each line in the main table corresponds to each line of the JSON file. The other tables have a column corresponding to this primary index.

All the active columns in the table have tag name labels.

Specifically, the algorithm flow for generating the JSON file is shown in fig. 4.

In conclusion, the method is simple to use, and the required JSON file can be generated by one key only by simply configuring the JSON template; the processing efficiency is high, the JSON format and the data source which need to be generated are well designated, the table data which need to be processed is pasted, the processing can be well carried out only in a few seconds, and much time is saved compared with manual conversion; the problem of high error rate of manual processing is avoided, and no operation error occurs after the format is correctly configured; the JSON template can be flexibly configured to adapt to different use scenes.

As shown in fig. 5, a schematic structural diagram of an embodiment 1 of a system for converting structured data into unstructured data provided for application may include:

a configuration module 401, configured to configure a JSON template;

The analyzing module 402 is used for analyzing the JSON template to a structure array;

And the conversion module 403 is configured to read an array source according to the structure array to obtain a JSON file.

As shown in fig. 6, a schematic structural diagram of embodiment 2 of a system for converting structured data into unstructured data provided for application may include:

a configuration module 501, configured to configure a JSON template;

The analyzing module 502 is used for analyzing the JSON template to a structure array;

And the conversion module 503 is configured to read data from the database according to the structure array, and generate a JSON file.

As shown in fig. 7, a schematic structural diagram of embodiment 3 of a system for converting structured data into unstructured data provided for application may include:

a configuration module 601, configured to configure a JSON template;

The form is as follows (taking excel as an example of a data source):

The analysis module 602 is used for analyzing the JSON template to a structure array through a recursive function;

the structure array is defined as:

Public Type jsonspot

spotno As String' node numbering, 1,2,3

spotype As String' node type: m: intermediate node middle L: leaf node leaf

spotdepth As String' node hierarchy: 1-first layer, 2-second layer.

parentspot As String' parent node number: 0: root node, 1: node 1.

O: object NA numerical array OA object array

End Type

The complete structure is as follows:

and the conversion module 603 is configured to read data from the table according to the structure array, and generate a JSON file.

All the active columns in the table have tag name labels.

The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.

Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of converting structured data into unstructured data, the method comprising:

configuring a JSON template;

analyzing the JSON template to a structure array;

reading an array source according to the structure array, and obtaining a JSON file based on an algorithm process for generating the JSON file;

the algorithm process for generating the JSON file comprises the following steps:

a1: outputting a json function and then executing A2;

a2: finding the table name and the label name of the first node structure array, traversing all main table rows, and then executing A3;

a3: judging whether all the row data of the main list are processed completely, if so, executing A15, and otherwise, executing A4;

a4: initializing json line data starting from the jth node, wherein j =2, and then performing a 5;

a5: judging whether j is smaller than the total number of labels, wherein the total number of labels is the number of elements of the structure array, if so, executing A6, and otherwise, executing A9;

a6: judging the node type with the depth of 1, if the node type is a leaf node, executing A7, and if the node type is an intermediate node, executing A11;

a7: calling a function to obtain a value corresponding to the label and assigning the value to temp, and then executing A8;

a8: concatenate json row data and let j = j +1, then perform a 5;

a9: after the processing of one line of jsonnine is finished, removing the tail "," and adding; "and line break, then perform A10;

a10: writing a line jsonnine into the file, and then executing A3;

a11: judging the type of the intermediate node value, if the type of the intermediate node value is an OA type, executing A12, and if the type of the intermediate node value is an O type, executing A14;

a12: judging whether the OA type has a nested object, if so, reporting an error, and if not, executing A13;

a13: calling a function to obtain the value of the OA type, assigning the value to temp, and then executing A8;

a14: calling a function to obtain a label value and assigning the label value to temp, and then executing A8;

a15: and the JSON file generation is finished.

2. The method of claim 1, wherein parsing the JSON template into a struct array comprises:

3. The method according to claim 2, wherein reading an array source according to the structure array, and obtaining a JSON file based on an algorithm process for generating the JSON file comprises:

and reading data from the table according to the structure array, and generating the JSON file based on the algorithm process for generating the JSON file.

4. The method of claim 3 wherein the first node in the JSON template is used as a primary index, the primary index corresponding to a primary field of a primary table, and wherein each row in the primary table corresponds to each row of the JSON file.

5. The method according to claim 3, wherein reading an array source according to the structure array, and obtaining a JSON file based on an algorithm process for generating the JSON file comprises:

and reading data from a two-dimensional table of a database according to the structure array, and generating the JSON file based on the algorithm process for generating the JSON file.

6. A system for converting structured data into unstructured data, comprising:

the configuration module is used for configuring the JSON template;

the conversion module is used for reading an array source according to the structure array and obtaining a JSON file based on an algorithm process for generating the JSON file, wherein the algorithm process for generating the JSON file comprises the following steps: a1: outputting a json function and then executing A2; a2: finding the table name and the label name of the first node structure array, traversing all main table rows, and then executing A3; a3: judging whether all the row data of the main list are processed completely, if so, executing A15, and otherwise, executing A4; a4: initializing json line data starting from the jth node, wherein j =2, and then performing a 5; a5: judging whether j is smaller than the total number of labels, wherein the total number of labels is the number of elements of the structure array, if so, executing A6, and otherwise, executing A9; a6: judging the node type with the depth of 1, if the node type is a leaf node, executing A7, and if the node type is an intermediate node, executing A11; a7: calling a function to obtain a value corresponding to the label and assigning the value to temp, and then executing A8; a8: concatenate json row data and let j = j +1, then perform a 5; a9: after the processing of one line of jsonnine is finished, removing the tail "," and adding; "and line break, then perform A10; a10: writing a line jsonnine into the file, and then executing A3; a11: judging the type of the intermediate node value, if the type of the intermediate node value is an OA type, executing A12, and if the type of the intermediate node value is an O type, executing A14; a12: judging whether the OA type has a nested object, if so, reporting an error, and if not, executing A13; a13: calling a function to obtain the value of the OA type, assigning the value to temp, and then executing A8; a14: calling a function to obtain a label value and assigning the label value to temp, and then executing A8; a15: and the JSON file generation is finished.

7. The system of claim 6, wherein the parsing module is specifically configured to:

8. The system of claim 7, wherein the conversion module is specifically configured to:

9. The system of claim 8 wherein a first node in the JSON template serves as a primary index, the primary index corresponding to a primary field of a primary table, each row in the primary table corresponding to each row of a JSON file.

10. The system of claim 7, wherein the conversion module is specifically configured to: