CN114003603A - Data export method, system and storage medium based on big data platform - Google Patents
Data export method, system and storage medium based on big data platform Download PDFInfo
- Publication number
- CN114003603A CN114003603A CN202111270483.3A CN202111270483A CN114003603A CN 114003603 A CN114003603 A CN 114003603A CN 202111270483 A CN202111270483 A CN 202111270483A CN 114003603 A CN114003603 A CN 114003603A
- Authority
- CN
- China
- Prior art keywords
- data
- data processing
- exported
- configuration
- table configuration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
- G06F9/4451—User profiles; Roaming
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application is suitable for the technical field of artificial intelligence and provides a data export method, a data export system and a storage medium based on a big data platform. The method comprises the steps of creating table configuration information through a configuration generation server, wherein the table configuration information is used for determining a table structure, automatically acquiring and distributing the table configuration information through a configuration distribution server, enabling a data processing server to receive and load the table configuration information according to a load condition, and generating a corresponding table configuration interface, so that data to be exported can be exported according to the table configuration interface and the corresponding table structure, the modeling and the flow of table export are realized, the response speed of table export requirements is improved, and the development time cost of a table export script is greatly reduced.
Description
Technical Field
The application belongs to the technical field of data processing, and particularly relates to a data export method, a data export system and a storage medium based on a big data platform.
Background
At present, enterprises and public institutions which have data processing services or generate a large amount of data in the process of processing the services usually choose to import the data into a database for unified storage and management, and the operation database needs to have certain professional knowledge, so that users lacking the professional knowledge of the database cannot export specified data in the database according to actual requirements, and developers can design corresponding table export scripts according to the actual requirements of the users to realize automatic export of the specified data.
When the actual requirements of users are continuously increased and the content types of designated data are also continuously increased, developers need to design corresponding table export scripts according to the content types of each type of designated data, and a certain number of same codes exist among multiple table export scripts, so that the time consumed for developing the table export scripts is too long and the same codes are repeatedly compiled, and therefore, how to solve the problem that the development rate of the table export scripts is poor becomes the problem which needs to be solved urgently at present.
Disclosure of Invention
In view of this, embodiments of the present application provide a data export method, system and storage medium based on a big data platform, so as to solve the problems that time consumed for developing a table export script is too long, the same code is repeatedly written, and the development efficiency of the table export script is poor.
A first aspect of an embodiment of the present application provides a data export method based on a big data platform, which is applied to a data processing server built based on the big data platform, and the method includes:
receiving at least one piece of form configuration information sent by the configuration distribution server, wherein the form configuration information is used for determining a form structure;
packaging each piece of table configuration information to generate a corresponding table configuration interface;
receiving data to be exported and an export instruction sent by the demand unit;
acquiring the content type of the data to be exported;
calling a corresponding table configuration interface according to the content type of the data to be exported and an export instruction, wherein the content type of the data to be exported is matched with the table structure of the corresponding table configuration interface;
and determining the table structure through the corresponding table configuration interface, so that the data to be exported are exported according to the table structure.
In an embodiment, the encapsulating each piece of the table configuration information to generate a corresponding table configuration interface includes:
packaging each piece of table configuration information to generate a corresponding annotation class;
packaging each annotation class to generate a corresponding annotation method;
and packaging each annotation method to generate a corresponding table configuration interface.
In one embodiment, the method further comprises:
and recording a table configuration interface corresponding to the content type of each data to be exported, and establishing a table corresponding to the content type and the interface.
In an embodiment, before the invoking a corresponding table configuration interface according to the content type of the data to be exported and the export instruction, the method further includes:
acquiring the authority level of the export instruction;
determining a corresponding table configuration interface list according to the permission level of the export instruction;
judging whether the content type of the data to be exported and a table configuration interface corresponding to the export instruction are located in a table configuration interface list or not;
when the content type of the data to be exported and a table configuration interface corresponding to the export instruction are not located in a table configuration interface list, the data to be exported is stopped being exported;
and when the content type of the data to be exported and the table configuration interface corresponding to the export instruction are located in a table configuration interface list, executing the calling of the corresponding table configuration interface according to the content type of the data to be exported and the export instruction.
In the data export method applied to the data processing server, the data processing server generates the corresponding table configuration interface by receiving the table configuration information sent by the configuration distribution server, and calls the corresponding table configuration interface in response to the export instruction when receiving the data to be exported and the export instruction, so that the data to be exported can be exported according to the table structure, thereby realizing the modeling and the flow of table export, improving the response speed of table export requirements, and greatly reducing the time cost for developing a table export script.
A second aspect of the embodiments of the present application provides a data export method based on a big data platform, which is applied to a configuration distribution server, and the method includes:
reading the table configuration information when the table configuration information is newly added to the configuration generation server;
and sending the table configuration information to the data processing server according to the load condition of the data processing server.
In one embodiment, the reading and sending the table configuration information to the data processing server according to the load condition of the data processing server includes:
and reading and sending the table configuration information to the unloaded data processing units according to the load condition of each data processing unit.
In the data export method applied to the configuration distribution server, when the configuration distribution server adds the table configuration information to the configuration generation server, the configuration distribution server automatically reads the added table configuration information and distributes the table configuration information according to the load condition of the data processing server, so that the automatic acquisition and distribution of the table configuration information are realized, the performance of the data processing server is fully exerted, and the table export efficiency is improved.
A third aspect of the embodiments of the present application provides a data export method based on a big data platform, which is applied to a configuration generation server, and the method includes:
adding table configuration information according to an input instruction of a user, wherein the table configuration information is used for determining a table structure.
In the data export method applied to the configuration generation server, the configuration generation server receives the input instruction of the user to add new form configuration information, and the user can be supported to set the form configuration information according to actual needs, so that different form structures are determined, and flexibility of configuring the form structures is improved.
A fourth aspect of the embodiments of the present application provides a data export system based on a big data platform, including a configuration generation server, a configuration distribution server, and a data processing server, where the configuration distribution server is connected to the configuration generation server and the data processing server respectively;
the configuration generation server is configured to execute the steps of the data export method provided by the third aspect of the embodiment of the present application;
the configuration distribution server is used for executing the steps of the data export method provided by the second aspect of the embodiment of the application;
the data processing server is configured to perform the steps of the data export method provided in the first aspect of the embodiment of the present application.
In one embodiment, the data processing server comprises at least one data processing unit, each of said data processing units being connected to a corresponding one of the demand units.
In the data export system provided in the fourth aspect of the embodiment of the present application, the configuration generation server receives newly added form configuration information of the user input instruction sent by the demand unit, so as to determine different form structures; the newly added form configuration information is automatically read through a configuration distribution server, and the form configuration information is distributed according to the load condition of the data processing server, so that the automatic acquisition and distribution of the form configuration information are realized; and receiving the form configuration information sent by the configuration distribution server through the data processing server, generating a corresponding form configuration interface, responding to the export instruction to call the corresponding form configuration interface when receiving the data to be exported and the export instruction, so that the data to be exported can be exported according to a form structure, the mode and the process of form export are realized, the automation degree of form export is improved, the response speed of form export requirements is improved, and the time cost for developing a form export script is greatly reduced.
A fifth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps of the data export method provided in the first aspect to the third aspect of the embodiments of the present application.
It is to be understood that, the beneficial effects of the fifth aspect may be referred to in the description of the first aspect to the third aspect, and are not described herein again.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic view of a scenario of a data export system provided in an embodiment of the present application;
FIG. 2 is a schematic structural diagram of a data processing server according to an embodiment of the present application;
fig. 3 is a schematic flowchart of a data export method applied to a data processing server according to an embodiment of the present application;
fig. 4 is a schematic flowchart of a data export method applied to a configuration distribution server according to an embodiment of the present application;
fig. 5 is a flowchart illustrating a data export method applied to a configuration generation server according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to" determining "or" in response to detecting ". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".
Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing or implying relative importance.
Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.
In application, when data to be exported has multiple content types, corresponding table export scripts need to be developed according to the multiple content types, and a certain number of same codes exist in the development process of the multiple table export scripts, so that the time consumed for developing the table export scripts is too long, the same codes are repeatedly written, and the development efficiency of the table export scripts is poor.
In view of the above technical problems, an embodiment of the present application provides a data export method based on a big data platform, where a configuration generation server creates table configuration information, where the table configuration information is used to determine a table structure, and a configuration distribution server automatically obtains and distributes the table configuration information, so that a data processing server can receive and load the table configuration information according to a load condition, and generate a corresponding table configuration interface, and thus data to be exported can be exported according to the table configuration interface and the corresponding table structure thereof, thereby implementing schematization and flow of table export, further improving response speed to a table export demand, and greatly reducing development time cost of a table export script.
The data export method based on the big data platform can be applied to any terminal equipment capable of recording or processing data. The terminal device may be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an Augmented Reality (AR)/Virtual Reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), and the like, and the embodiment of the present application does not limit the specific type of the electronic device.
The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.
The server in the embodiment of the application may be an independent server, or may be a cloud server that provides basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, Network service, cloud communication, middleware service, domain name service, security service, Content Delivery Network (CDN), and an artificial intelligence platform, and specifically may be a server built based on a large data platform.
As shown in fig. 1, the data export system based on the big data platform provided in the embodiment of the present application includes a configuration generation server 100, a configuration distribution server 200, and a data processing server 300, where the configuration distribution server 200 is connected to the configuration generation server 100 and the data processing server 300 respectively;
the data processing server 300 is configured to execute the data export method applied to the data processing server corresponding to fig. 3;
the configuration distribution server 200 is configured to execute a data export method applied to the configuration distribution server corresponding to fig. 4;
the configuration generation server 100 is configured to execute the data export method applied to the configuration generation server corresponding to fig. 5.
In one embodiment, the data processing server 300 comprises at least one data processing unit 301, each data processing unit 301 being connected to a corresponding one of the demand units 401.
Fig. 2 schematically shows a structural diagram of the data processing server 300, specifically, the data processing server 300 may include at least one processor 302 and at least one memory 303, the memory 303 stores a computer program 304, and the at least one processor 302 may implement the data deriving method applied to the data processing server corresponding to fig. 2 when executing the computer program 304.
In application, one data processing unit may comprise one processor and one memory, or a plurality of data processing units may comprise one processor and one memory.
In an Application, the Processor may be a Central Processing Unit (CPU), and the Processor may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
In some embodiments, the storage may be an internal storage unit of the terminal device, such as a hard disk or a memory of the terminal device. The memory may also be an external storage device of the terminal device in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the terminal device. Further, the memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used for storing an operating system, application programs, a BootLoader (BootLoader), data, and other programs, such as program codes of computer programs. The memory may also be used to temporarily store data that has been output or is to be output.
In application, the structures of the configuration generation server 100 and the configuration distribution server 200 are the same as those of the data processing server 300, and are not described herein again.
It is to be understood that the illustrated structure of the embodiments of the present application does not constitute a specific limitation to the data export system. In other embodiments of the present application, the data export system may include more or fewer components than shown, or some components may be combined, or different components may include, for example, input output devices, network access devices, etc. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
As shown in fig. 3, the data export method based on the big data platform provided in the embodiment of the present application is applied to a data processing server, and includes the following steps S301 to S306:
step S301, at least one piece of form configuration information sent by the configuration distribution server is received, and the form configuration information is used for determining a form structure.
In application, the Data processing server is configured to receive at least one piece of table configuration information sent by the configuration distribution server, where the table configuration information may be written in a Data exchange Format (Data exchange Format), and the Data exchange Format may be based on a text record table structure and has an advantage suitable for any operating system (e.g., Windows, Linux, macOS, or the like), so that the table configuration information may be used to determine the table structure and may be read by servers of multiple different operating systems. The data exchange format of the table configuration information may specifically be JSON (JavaScript Object Notation) Language, and may also be XML (Extensible Markup Language), and the specific Language type of the data exchange format is not limited in this embodiment of the application.
In application, each piece of table configuration information can be stored in a URL (Uniform Resource Locator), and the URL is used as a carrier for storing the table configuration information, so that the table configuration information can be transmitted between different servers, and the method has the advantages of high transmission speed, high transmission stability and difficulty in information leakage.
In the application, when the configuration distribution server and the data processing server are located in one network environment, the communication range of the URL may be set in the one network environment; when the configuration distribution server and the data processing server are located in two different network environments, the communication range of the URL may not be limited, where the network environment may specifically be a local area network or a wide area network, and the communication range of the URL is adjusted according to the network environment where the configuration distribution server and the data processing server are located, so that the flexibility of obtaining the form configuration information may be improved.
In one embodiment, step S301 comprises:
the table configuration information includes at least one of:
table name, title information, title attribute, document name, and document extension.
In an application, the form configuration information may include a plurality of configuration contents such as a form name, title information, a title attribute, a document name, a document extension name, and the like, where the title information and the title attribute may be used to determine a title of the form; the document extension name may be used to determine a form type, where the form type may specifically be an EXCEL (workbook) document in a format of xls, xlsx, and the like, and may also be an XML document in a format of XML, and the like, so as to determine a form structure according to form configuration information.
Step S302, packaging each piece of table configuration information to generate a corresponding table configuration interface.
In application, the data processing server may be configured to analyze the table configuration information, implement encapsulation of each piece of table configuration information, and generate one or more corresponding table configuration interfaces, so that when the data processing server receives a data export demand, the data processing server may invoke the table configuration interface adapted to the data to be exported according to a demand of a demand unit, and increase a data export speed, where a number of table structures included in each piece of table configuration information determines a number of corresponding table configuration interfaces to be generated.
In one embodiment, step S302 includes:
packaging the configuration information of each table to generate a corresponding annotation class;
packaging each annotation class to generate a corresponding annotation method;
and packaging each annotation method to generate a corresponding table configuration interface.
In the application, each piece of table configuration information is packaged to generate one or more corresponding table configuration interfaces, three steps of packaging the table configuration information into an annotation class, packaging the annotation class into an annotation method, and packaging the annotation method into the table configuration interfaces are required, and the following takes the generation of a corresponding table configuration interface by one piece of table configuration information as an example to explain the three steps:
the data processing server may process a plurality of table configuration information in parallel according to the actual processing capability, wherein, when processing a piece of table configuration information, each item of content in the table configuration information may be read first, and a corresponding annotation is generated for each item of configuration content, for example, if the configuration content "table name" of the table configuration information includes "order information," the configuration content "title information" includes "order ID" and "predetermined date," and the configuration content "title attribute" includes "order ID" and "bookmark date," the data processing server may add annotation "order information" to the configuration content "table name," add annotation "order ID" and "predetermined date" to the configuration content "title information," and add annotation "order ID" and "bookmark date" to the configuration content "title attribute," thereby generating three annotations corresponding to the configuration content of the table configuration information, forming an annotation class corresponding to the table configuration information;
the data processing server may package the annotation class to generate a corresponding annotation method, so as to give processing logic to the annotation class when extracting data to be derived, and the processing logic may be specifically implemented based on a Reflection mechanism (Reflection) of JAVA. Specifically, through a JAVA reflection mechanism, corresponding data to be exported may be extracted according to an annotation class, for example, the configuration content "table name" of the table configuration information of the data to be exported includes "order information", the configuration content "title information" includes "order ID" and "predetermined date", the configuration content "title attribute" includes "order ID" and "bookmark date", and it is easy to obtain that the configuration content included in the data to be exported is the same as the configuration content of the annotation classes of the above three annotations, so that data included in three identical configuration contents may be extracted, that is, two columns of data included in the table name "order information" whose title information is "order ID" and "predetermined date" and whose title attribute is "order ID" and "bookmark date". It should be noted that, the above-mentioned extraction of the data to be exported according to the table name, the title information and the three configuration contents of the title attribute is only exemplary, and the extraction of the data to be exported can be performed according to the configuration contents of the actually set table configuration information;
the data processing server can package the annotation method to generate a corresponding form configuration interface, so that when the data processing server calls any one form configuration interface, the data to be exported is quickly extracted by using the packaged annotation method, and the form configuration interface can be provided with a preset form type for determining the form type of the exported data, specifically can be EXCEL documents with document extensions of xls, xlsx and the like, so that after the data to be exported is input through the form configuration interface, the data can be automatically extracted and the exported data of the preset form type can be output, and the data export efficiency is greatly improved.
In the application, the form configuration interface can also be provided with preset format settings, the preset format settings can be used for determining font settings, cell settings and the like, specifically, fonts, font sizes, font colors, underlines and bold font effects of different cells can be adjusted through the font settings, and cell colors, cell lengths and widths, cell data types, cell borders and the like of the different cells can be adjusted through the cell settings. The embodiment of the present application does not set any limit to the specific content of the preset format setting.
And step S303, receiving the data to be exported and the export instruction sent by the demand unit.
In application, the data processing server can receive data to be exported and export instructions of the demand units in real time, each piece of data to be exported can have a corresponding export instruction, and the data processing server can respond to the export instructions to export the data to be exported.
In application, the demand unit can send data to be exported, which is stored in any kind of database, and the data processing Server can support data sent by different types of databases, such as Oracle, MySQL, MongoDB, SQL Server, IBM Db2 and Dameng database.
Step S304, acquiring the content type of the data to be exported;
step S305, calling a corresponding table configuration interface according to the content type of the data to be exported and the export instruction, wherein the content type of the data to be exported is matched with the table structure of the corresponding table configuration interface.
In application, the content type of the data to be exported can be obtained from the export instruction; the corresponding content type may also be obtained by analyzing the data to be exported, and the corresponding table configuration interface may be matched, specifically, the data processing server may perform matching retrieval on the multiple types of table configuration interfaces by reading a first preset number of configuration contents of the data to be exported, for example, a table name, title information, a title attribute, a document name, and the like, according to the read first preset number of configuration contents, and if the first preset number of configuration contents of any one type of table configuration interface is the same as the first preset number of configuration contents of the data to be exported, it is described that the content type of the data to be exported matches the table structure of the table configuration interface, and the table configuration interface may be used as the corresponding table configuration interface.
In application, the data processing server may further determine the corresponding table configuration interface to be invoked according to the export instruction or/and the content type of the data to be exported. Specifically, the export instruction may be used to determine a called table configuration interface; alternatively, the export instruction may be used to determine multiple invokable form configuration interfaces, and according to the content type of the data to be exported, matching search may be performed based on the multiple invokable form configuration interfaces to determine a corresponding form configuration interface. The data processing server may preset a table of correspondence between content types and interfaces, where the table of correspondence includes a correspondence between a content type of data to be exported and a table configuration interface, so that the table configuration interface may be determined by looking up a table.
In one embodiment, step S305 further comprises:
and recording a table configuration interface corresponding to the content type of each data to be exported, and establishing a table corresponding to the content type and the interface.
In application, after a table configuration interface corresponding to the content type of any data to be exported is acquired, the content type of the data to be exported and the corresponding table configuration interface are recorded, so that a content type and interface corresponding table is generated.
In one embodiment, step S305 further comprises:
acquiring the authority level of the export instruction;
determining a corresponding table configuration interface list according to the permission level of the export instruction;
judging whether the content type of the data to be exported and a table configuration interface corresponding to the export instruction are positioned in a table configuration interface list or not;
when the content type of the data to be exported and the table configuration interface corresponding to the export instruction are not located in the table configuration interface list, stopping exporting the data to be exported;
and when the content type of the data to be exported and the table configuration interface corresponding to the export instruction are positioned in the table configuration interface list, calling the corresponding table configuration interface according to the content type of the data to be exported and the export instruction.
In an application, the export instruction may have permission levels, and the permission levels may include at least two levels, each permission level having a corresponding table configuration interface list, each table configuration interface list including at least one table configuration interface. For example, it is assumed that the permission level of the export instruction includes three levels, i.e., a first permission level, a second permission level, and a third permission level, the third permission level corresponds to a third table configuration interface list, the second permission level corresponds to a second table configuration interface list, and the first permission level corresponds to a first table configuration interface list. The number and the type of the table configuration interfaces included in each table configuration interface list can be set according to actual needs.
In application, the authority level of the export instruction can be determined by acquiring Identity characteristics such as department, employee ID (Identity identification number) or device MAC (Media Access Control Address) of a demand unit sending the export instruction. The corresponding relation between the identity characteristics and the authority levels can be set according to actual needs.
In application, acquiring a corresponding table configuration interface list according to the authority level of the export instruction, and judging whether the table configuration interface corresponding to the export instruction is positioned in the table configuration interface list or not according to the table configuration interface corresponding to the export instruction; if yes, the process goes to step S305 to export the data to be exported through the corresponding table configuration interface, and if not, the export of the data to be exported is stopped.
In application, by obtaining the permission level of the export instruction, the corresponding table configuration interface list can be determined, and whether the table configuration interface corresponding to the export instruction belongs to the table configuration interface list corresponding to the export instruction is judged, so that whether a demand unit sending the export instruction has permission to call the corresponding table configuration interface is monitored, the demand unit without permission is prevented from exporting data to be exported, and the safety of data export is improved.
And S306, determining a table structure through the corresponding table configuration interface, and leading the data to be led out according to the table structure.
In application, after a table configuration interface corresponding to the content type of the data to be exported is obtained, the data to be exported can be exported according to the table structure of the table configuration interface through a JAVA reflection mechanism, so that the data to be exported can be rapidly exported.
In application, the data processing server generates a corresponding table configuration interface by receiving the table configuration information sent by the configuration distribution server, and responds to the export instruction to call the corresponding table configuration interface when receiving the data to be exported and the export instruction, so that the data to be exported can be exported according to a table structure, the modeling and the flow of table export are realized, the response speed of table export requirements is improved, and the time cost for developing a table export script is greatly reduced.
As shown in fig. 4, the data export method based on the big data platform provided in the embodiment of the present application is applied to a configuration distribution server, and includes the following steps S401 and S402:
step S401, when the configuration generation server adds the table configuration information, reading the table configuration information.
In application, the configuration distribution server can monitor whether the configuration generation server adds the table configuration information in real time, and when the configuration generation server adds the table configuration information, the configuration distribution server reads the table configuration information from the configuration generation server and stores the table configuration information.
Step S402, according to the load condition of the data processing server, sending the table configuration information to the data processing server.
In application, the configuration distribution server can also read the load condition of the data processing server, and the load is positively correlated with the quantity of the table configuration information received by the data processing server. The table configuration information of the second preset number may be sent to the data output server when the load condition of the data processing server is smaller than the preset threshold, so as to update the table structure supported by the data processing server. The preset threshold may be specifically 60%, 70%, or 80%, and the second preset number may be 3, 4, or 5, and the specific numerical values of the preset threshold and the second preset number are not limited in the embodiment of the present application.
In one embodiment, step S402 includes:
and reading and sending the table configuration information to the unloaded data processing units according to the load condition of each data processing unit.
In application, the data processing server comprises at least one data processing unit, the configuration distribution server is respectively connected with each data processing unit, and each data processing unit is used for receiving a piece of form configuration information and generating a corresponding form configuration interface. When the data processing server includes a plurality of data processing units, each data processing unit may receive a piece of table configuration information and generate a corresponding table configuration interface to support a corresponding table structure. The data distribution server can read whether each data processing unit receives the table configuration information, if so, the corresponding data processing unit is judged to be full, and if not, the corresponding data processing unit is judged to be empty. The configuration distribution server may distribute the table configuration information to the off-board data processing units.
In application, the configuration distribution server automatically reads the newly added form configuration information when the configuration generation server adds the form configuration information, and distributes the form configuration information according to the load condition of the data processing server, so that the automatic acquisition and distribution of the form configuration information are realized, the performance of the data processing server is fully exerted, and the form export efficiency is improved.
As shown in fig. 5, the data export method based on the big data platform provided in the embodiment of the present application is applied to the configuration generation server, and includes the following steps S501:
step S501, adding table configuration information according to an input instruction of a user, wherein the table configuration information is used for determining a table structure.
In application, the configuration generation server receives the input instruction of the user, which is sent by the demand unit, to add the table configuration information, and can support the user to set the table configuration information according to actual needs, so that different table structures are determined, and the flexibility and convenience for configuring the table structures are improved. The specific configuration content of the table configuration information is the same as that introduced in step S201, and is not described herein again.
In one embodiment, step S501 further includes:
reading the data to be exported of the demand unit, and adding table configuration information according to the table structure of the data to be exported.
In application, the configuration generation server can be connected with the demand unit, read data to be exported of the demand unit, generate configuration content according to a table structure of the data to be exported, package the generated configuration content to generate table configuration information, and realize automatic generation of the table configuration information.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative modules and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed terminal device and method may be implemented in other ways. For example, the above-described terminal device embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and there may be other divisions when actually implementing, for example, a plurality of modules or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (10)
1. A data export method based on a big data platform is characterized by being applied to a data processing server built based on the big data platform, wherein the data processing server is respectively connected with a plurality of demand units, the data processing server is also connected with a configuration distribution server, and the method comprises the following steps:
receiving at least one piece of form configuration information sent by the configuration distribution server, wherein the form configuration information is used for determining a form structure;
packaging each piece of table configuration information to generate a corresponding table configuration interface;
receiving data to be exported and an export instruction sent by the demand unit;
acquiring the content type of the data to be exported;
calling a corresponding table configuration interface according to the content type of the data to be exported and an export instruction, wherein the content type of the data to be exported is matched with the table structure of the corresponding table configuration interface;
and determining the table structure through the corresponding table configuration interface, so that the data to be exported are exported according to the table structure.
2. The data exporting method according to claim 1, wherein said encapsulating each of said table configuration information to generate a corresponding table configuration interface comprises:
packaging each piece of table configuration information to generate a corresponding annotation class;
packaging each annotation class to generate a corresponding annotation method;
and packaging each annotation method to generate a corresponding table configuration interface.
3. The data derivation method of claim 1, wherein the method further comprises:
and recording a table configuration interface corresponding to the content type of each data to be exported, and establishing a table corresponding to the content type and the interface.
4. The data exporting method according to claim 1, before the invoking of the corresponding table configuration interface according to the content type of the data to be exported and the exporting instruction, further comprising:
acquiring the authority level of the export instruction;
determining a corresponding table configuration interface list according to the permission level of the export instruction;
judging whether the content type of the data to be exported and a table configuration interface corresponding to the export instruction are located in a table configuration interface list or not;
when the content type of the data to be exported and a table configuration interface corresponding to the export instruction are not located in a table configuration interface list, the data to be exported is stopped being exported;
and when the content type of the data to be exported and the table configuration interface corresponding to the export instruction are located in a table configuration interface list, executing the calling of the corresponding table configuration interface according to the content type of the data to be exported and the export instruction.
5. A data export method based on a big data platform is applied to a configuration distribution server, the configuration distribution server is respectively connected with a data processing server and a configuration generation server, and the method comprises the following steps:
reading the table configuration information when the table configuration information is newly added to the configuration generation server;
and sending the table configuration information to the data processing server according to the load condition of the data processing server.
6. The data exporting method according to claim 5, wherein the data processing server comprises at least one data processing unit, the configuration distribution server is connected to each of the data processing units, and each data processing unit is configured to receive a piece of form configuration information and generate a corresponding form configuration interface;
the reading and sending the form configuration information to the data processing server according to the load condition of the data processing server includes:
and reading and sending the table configuration information to the unloaded data processing units according to the load condition of each data processing unit.
7. A data export method based on a big data platform is characterized by being applied to a configuration generation server, wherein the configuration generation server is connected with a configuration distribution server;
adding table configuration information according to an input instruction of a user, wherein the table configuration information is used for determining a table structure.
8. A data export system based on a big data platform is characterized by comprising a configuration generation server, a configuration distribution server and a data processing server, wherein the configuration distribution server is respectively connected with the configuration generation server and the data processing server;
the configuration generation server is adapted to perform the steps of the method according to claim 7;
the configuration distribution server is adapted to perform the steps of the method according to claim 5 or 6;
the data processing server is adapted to perform the steps of the method according to any of claims 1 to 4.
9. The system of claim 8, wherein the data processing server includes at least one data processing unit, each of the data processing units being connected to a corresponding one of the demand units.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the data derivation method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111270483.3A CN114003603A (en) | 2021-10-29 | 2021-10-29 | Data export method, system and storage medium based on big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111270483.3A CN114003603A (en) | 2021-10-29 | 2021-10-29 | Data export method, system and storage medium based on big data platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114003603A true CN114003603A (en) | 2022-02-01 |
Family
ID=79925499
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111270483.3A Pending CN114003603A (en) | 2021-10-29 | 2021-10-29 | Data export method, system and storage medium based on big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114003603A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115329733A (en) * | 2022-10-10 | 2022-11-11 | 深圳高灯计算机科技有限公司 | Report form statistical method and device, computer equipment and storage medium |
-
2021
- 2021-10-29 CN CN202111270483.3A patent/CN114003603A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115329733A (en) * | 2022-10-10 | 2022-11-11 | 深圳高灯计算机科技有限公司 | Report form statistical method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112650533B (en) | Interface document generation method and device and terminal equipment | |
| CN111400246B (en) | Asynchronous file import method, device, computer equipment and storage medium | |
| CN113254445B (en) | Real-time data storage method, device, computer equipment and storage medium | |
| CN110825499B (en) | Method and device for realizing security isolation based on unified container cloud platform | |
| CN106326129A (en) | Program abnormity information generating method and device | |
| CN110109983B (en) | Method and device for operating Redis database | |
| CN114995859A (en) | Page hot updating method, device, equipment and storage medium | |
| CN110888972A (en) | Sensitive content identification method and device based on Spark Streaming | |
| CN114003603A (en) | Data export method, system and storage medium based on big data platform | |
| CN115328759A (en) | Form verification method and device | |
| CN109325192B (en) | Advertisement anti-shielding method and device | |
| CN113626223A (en) | Interface calling method and device | |
| CN109062870A (en) | The method and storage medium that server, business datum verify | |
| CN116450134A (en) | Configurable front-end control verification controller generation method and system | |
| CN110673827A (en) | Resource calling method and device based on android system and electronic equipment | |
| CN115934537A (en) | Interface test tool generation method, device, equipment, medium and product | |
| CN112287643B (en) | Message monitoring method, device, equipment and computer readable storage medium | |
| CN114780497A (en) | Batch file processing method, apparatus, computer device, medium, and program product | |
| CN114036180A (en) | Report generation method, device, equipment and storage medium | |
| CN113742235A (en) | Method and device for checking codes | |
| CN113535322A (en) | Form verification method and device | |
| CN111767498A (en) | Method and device for realizing file information sharing | |
| CN116610354A (en) | Parameter verification method, device, computer equipment and storage medium | |
| CN109376023B (en) | Method and equipment for generating calling information and issuing return code group | |
| CN117667668A (en) | Application analysis method, device, equipment and storage medium based on application upgrading |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |