CN114996249A - Data processing method and device, electronic equipment, storage medium and product - Google Patents
Data processing method and device, electronic equipment, storage medium and product Download PDFInfo
- Publication number
- CN114996249A CN114996249A CN202210540644.4A CN202210540644A CN114996249A CN 114996249 A CN114996249 A CN 114996249A CN 202210540644 A CN202210540644 A CN 202210540644A CN 114996249 A CN114996249 A CN 114996249A
- Authority
- CN
- China
- Prior art keywords
- data processing
- data
- component
- model
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The present disclosure relates to a data processing method, apparatus, electronic device, storage medium and product, the method comprising: displaying a preset page; generating a data processing model in response to the combined operations for the target data processing components; responding to an operation instruction aiming at the data processing model, and acquiring target data corresponding to the data processing model; and executing data processing operation corresponding to the target data processing component on the target data according to the connection relation information to obtain a data processing result. By utilizing the embodiment of the disclosure, the labor cost of data processing can be reduced, and the real-time performance and efficiency of data processing are improved.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, a storage medium, and a product.
Background
With the development of data technology, data quality plays a crucial role in practical application scenarios, and therefore, collected data needs to be processed and then analyzed and used.
In the related art, one way is to store historical data through a database, and then perform batch processing on the historical data in the database; the other method is that a professional implementer uses a streaming processing framework to write a corresponding processing process to process data, so that the problem of real-time performance is solved, and meanwhile, the flexibility is kept, but certain requirements are made on the technology of the implementer.
In the related art, the labor cost of data processing is high, and the real-time performance and the efficiency of data processing are low.
Disclosure of Invention
The present disclosure provides a data processing method, apparatus, electronic device, storage medium, and product, so as to at least solve the problems of high labor cost and low real-time performance and efficiency of data processing in the related art. The technical scheme of the disclosure is as follows:
according to a first aspect of the embodiments of the present disclosure, there is provided a data processing method, including:
displaying a preset page, wherein the preset page comprises at least one data processing component;
generating a data processing model in response to a combined operation for a target data processing component, wherein the target data processing component is a data processing component in the at least one data processing component, and the data processing model comprises the target data processing component and connection relation information corresponding to the target data processing component;
responding to an operation instruction aiming at the data processing model, and acquiring target data corresponding to the data processing model;
and executing data processing operation corresponding to the target data processing component on the target data according to the connection relation information to obtain a data processing result.
According to a second aspect of the embodiments of the present disclosure, there is provided a data processing apparatus including:
the page display module is used for displaying a preset page, and the preset page comprises at least one data processing component;
a model generation module, configured to generate a data processing model in response to a combination operation for a target data processing component, where the target data processing component is a data processing component of the at least one data processing component, and the data processing model includes the target data processing component and connection relationship information corresponding to the target data processing component;
the data acquisition module is used for responding to an operation instruction aiming at the data processing model and acquiring target data corresponding to the data processing model;
and the data processing module is used for executing data processing operation corresponding to the target data processing component on the target data according to the connection relation information to obtain a data processing result.
According to a third aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including: a processor; a memory for storing the processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method of any of the first aspects above.
According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium, wherein instructions, when executed by a processor of an electronic device, enable the electronic device to perform the method of any one of the first aspects of the embodiments of the present disclosure.
According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the method of any one of the first aspects of the embodiments of the present disclosure.
The technical scheme provided by the embodiment of the disclosure at least brings the following beneficial effects:
the data processing method comprises the steps of receiving combined operation aiming at data processing components in a preset page, generating a data processing model, obtaining target data corresponding to the data processing model after receiving an operation instruction of the data processing model, carrying out data processing operation corresponding to the data processing components on the target data according to the connection relation in front of the data processing components in the data processing model, and further obtaining a data processing result, wherein technicians can configure a data processing flow in a combined component mode, the labor cost of data processing is reduced, and the real-time performance and the efficiency of data processing are improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure and are not to be construed as limiting the disclosure.
FIG. 1 illustrates a schematic diagram of an application environment in accordance with an exemplary embodiment;
FIG. 2 is a flow chart one of a data processing method according to an exemplary embodiment;
FIG. 3 is a diagram illustrating a preset page diagram I;
FIG. 4 illustrates a schematic diagram of the logical relationship between the source data acquisition component, the data manipulation component, and the data output component;
FIG. 5 illustrates a class diagram of a data manipulation component;
FIG. 6 illustrates a model structure diagram of a data processing model;
FIG. 7 is a flowchart II illustrating a data processing method in accordance with an exemplary embodiment;
FIG. 8 is a diagram illustrating a data processing model;
FIG. 9 is a diagram illustrating an exemplary default page view two;
FIG. 10 illustrates an operational state diagram of a multiple data processing model;
FIG. 11 is a schematic diagram illustrating a data processing flow based on a data processing model;
FIG. 12 is a diagram illustrating an example of a data processing result;
FIG. 13 is a block diagram illustrating a data processing apparatus in accordance with an exemplary embodiment;
FIG. 14 is a block diagram illustrating an electronic device for data processing in accordance with an illustrative embodiment.
Detailed Description
In order to make the technical solutions of the present disclosure better understood by those of ordinary skill in the art, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.
It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in other sequences than those illustrated or described herein. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for presentation, analyzed data, etc.) referred to in the present disclosure are information and data authorized by the user or sufficiently authorized by each party.
Referring to fig. 1, fig. 1 is a schematic diagram illustrating an application environment according to an exemplary embodiment, and as shown in fig. 1, the application environment may include a terminal 100 and a server 200.
The terminal 100 may be used to provide data processing services to any user. Specifically, the terminal 100 may include, but is not limited to, a smart phone, a desktop computer, a tablet computer, a notebook computer, a smart speaker, a digital assistant, an Augmented Reality (AR)/Virtual Reality (VR) device, a smart wearable device, and other types of electronic devices. Alternatively, the terminal 100 is installed with software, such as an application program, for providing the data processing service. Optionally, the operating system running on the electronic device may include, but is not limited to, an android system, an IOS system, linux, windows, and the like.
In an alternative embodiment, the server 200 may provide a background service for the terminal 100. Alternatively, the server 200 provides the target data to the terminal 100. Optionally, the server performs data processing on the target data according to a data processing model configured in the terminal 100, and returns a data processing result to the terminal. Specifically, the server 200 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a CDN (Content Delivery Network), a big data and artificial intelligence platform, and the like.
In addition, it should be noted that fig. 1 shows only one application environment provided by the present disclosure, and in practical applications, other application environments may also be included, for example, more terminals may be included.
In the embodiment of the present specification, the terminal 100 and the server 200 may be directly or indirectly connected through wired or wireless communication, and the disclosure is not limited herein.
FIG. 2 is a flow chart of a data processing method according to an exemplary embodiment, as shown in FIG. 2, the data processing method is used in an electronic device, and includes the following steps (210-240).
And step 210, displaying a preset page.
Optionally, the preset page comprises at least one data processing component. Optionally, the preset page includes a component selection area and a model editing area. Optionally, the component selection area comprises at least one data processing component.
Optionally, the data processing component in the preset page includes a source data obtaining component. The source data acquisition component corresponds to the source data. The source data includes part test data.
Optionally, the data processing component characterizes a data processing operation corresponding to the data processing component. Optionally, the data processing operation includes, but is not limited to, a data checking operation, a data adding operation, a data deleting operation, a data modifying operation, a format converting operation, a type converting operation, a data acquiring operation, and the like.
Optionally, the data processing component in the preset page includes a data operation component and a data output component.
Optionally, the data manipulation component comprises a data checker. Optionally, the data verifier characterizes a data verification operation, and the data processing operation includes a data verification operation.
Optionally, the data manipulation component comprises a data processor. Optionally, the operation corresponding to the data processor includes, but is not limited to, a data adding operation, a data deleting operation, a data modifying operation, a format converting operation, a type converting operation, a data acquiring operation, a data packing operation, a key processing operation, and the like.
In one example, as shown in FIG. 3, a diagram I of a preset page is illustrated. The preset page 30 shown in fig. 3 includes a component selection area 31 and a model editing area 32, the component selection area 31 includes a plurality of data processing components 33 that can be added, and a user can drag the selected data processing components 33 to the model editing area 32. The selected source data acquisition component 34, data manipulation component 35, and data output component 36 are included in the model edit 32. After adding the connecting line between the source data acquisition component 34, the data operation component 35 and the data output component 36, the user can click the model saving button 37, and the data processing model can be generated.
In one example, as shown in FIG. 4, a diagram illustrating the logical relationship between the source data acquisition component, the data manipulation component, and the data output component is illustrated. The source data acquisition component is connected with the data operation component, the information transmission direction is from the source data acquisition component to the data operation component, and the represented flow is based on that after target data acquired by the source data acquisition component is transmitted to the data operation component, data processing operation corresponding to the data processing component is executed on the target data to obtain a data processing result. And then, the data operation component is connected with the data output component, the information transmission direction is from the data operation component to the data output component, and the represented flow is based on the data output component to output the data processing result to the target position.
In brief, data flows from a source data acquisition component (DataIn), is processed and cleaned to be data meeting the conditions through a Processor (data operation component), and is output to a target position through a DataOut (data output component), so that the whole data cleaning process is completed.
In one example, as shown in FIG. 5, a class diagram of a data manipulation component is illustrated. As can be seen from fig. 5, the data manipulation component includes a data checker and a data processor. Wherein the data checker includes a key checker and a value checker. Optionally, the operation corresponding to the key verifier is a key verification operation, and the operation corresponding to the value verifier is a value verifier operation. Optionally, the data processor includes a data populator, a data deleter, a key processor, a data wrapper. Optionally, the data adder characterizes a data addition operation; the data deleter represents data deletion operation; the key processor represents key processing operation; the data wrapper characterizes the data wrapping operation. Alternatively, the data populator may add data by calling a related function.
In one possible embodiment, the data manipulation component has two implementation subclasses, Validator (data checker) and Handler (data Handler), respectively. The Validator (data checker) is responsible for verifying data, and the key verifier and the ValueValidator are arranged in the Validator (data checker), and are distributed to be responsible for verifying keys and values of the data. The Handler (data processor) is responsible for processing data, including adding data, deleting data, modifying data and the like, and corresponds to AddHandler (adding processor), DelHandler (deleting processor) and KeyHandler (key correlation processor), while the WrapHANdler (packaging processor) is used when being opened to the outside, and developers with development ability can develop components meeting the requirements of own industry through the entry.
In an exemplary embodiment, the generating of the at least one data processing component includes: acquiring at least one instruction set, wherein the at least one instruction set is an instruction set corresponding to data processing operation; and packaging the at least one instruction set to generate at least one data processing component.
Optionally, the instruction set is an instruction set input by the target object, for example, instruction sets corresponding to some functions, and the device may implement to execute a corresponding data processing operation by executing the instruction set corresponding to the data processing component.
In a possible implementation manner, the customized program interface information is obtained, the instruction set can be generated based on the program interface information, and then the instruction set is packaged to obtain the data processing component. For some specific scenes, specific data processing components are customized to complete more complex data cleaning work.
In response to the combined operation for the target data processing component, a data processing model is generated, step 220.
Optionally, the target data processing component is a data processing component of the at least one data processing component. Optionally, the target data processing component is a data processing component selected by the target object.
Optionally, the data processing model includes a target data processing component and connection relationship information corresponding to the target data processing component.
Optionally, the target data processing components may be connected by a connection line, and the connection line between the target data processing components represents connection relationship information. The connection relation information includes a connection object and an information transmission direction.
In one example, as shown in FIG. 6, a model structure diagram of a data processing model is illustrated. The data processing model shown in FIG. 6 includes a source data acquisition component, a data manipulation component, and a data output component. The source data acquisition component is connected with the data operation component, the information transmission direction is from the source data acquisition component to the data operation component, and the represented flow is based on that after target data acquired by the source data acquisition component is transmitted to the data operation component, data processing operation corresponding to the data processing component is executed on the target data to obtain a data processing result corresponding to the data processing component. And if the next component corresponding to the data processing component is still the data processing component, continuing to perform data processing operation on the result output by the previous data processing component until the corresponding next component is the data output component. And finally, outputting a data processing result to the target position based on the data output component and the data processing component connected with the data output component.
From the view of example, a Processor (data operation component) can be formed by combining a plurality of sub-Processor (data operation component) examples through different arrangements, a small general processing logic is contained in a single sub-Processor (data operation component), and a user can quickly construct a data cleaning flow for meeting the requirements of a specific scene by orderly combining the sub-processors (data operation components).
In an exemplary embodiment, as shown in fig. 7, the generation process of the data processing model, i.e. the implementation process of the above step 220, may include the following steps (221 to 223), and fig. 7 is a second flowchart of a data processing method according to an exemplary embodiment.
The target object can drag the selected target data processing assembly to the model editing area for display editing by executing assembly dragging operation on the data processing assembly in the assembly selection area.
In step 222, connection relationship information is determined in response to a component connection operation for the target data processing component.
The target object can connect the data processing components by executing component connection operation on the target data processing components in the model editing area, and the device can determine connection relation information according to the connection relation and the information flow direction among the target data processing components.
After adding the connecting lines among the data processing components, a user can click the model saving button of the model editing area, and the device responds to a click instruction aiming at the model saving button and generates a data processing model according to the target data processing component and the connection relation information.
Optionally, the data processing model comprises a source data acquisition component.
Optionally, the data processing model comprises a data manipulation component and a data output component. Optionally, the data manipulation component comprises a data checker. Optionally, the data manipulation component comprises a data processor.
In one example, as shown in FIG. 8, a schematic diagram of a data processing model is illustrated. The data processing model shown in FIG. 8 includes a source data acquisition component, a data manipulation component, and a data output component. The target data can be acquired based on the source data acquisition component, and the data processing result corresponding to the data operation component can be output based on the data output component.
The data processing model shown in FIG. 8 includes the following data manipulation components:
a data checker: "target field" data.results.measures "exists;
a data processor: an "acquisition field" time _ sec ";
a data processor: "acquisition field" time _ hour ";
a data processor: an "acquisition field" time _ day ";
a data processor: "character string field" data. results. measures. value "is converted into numerical type";
a data processor: "character string field" data. results. measures. normal "is converted into numerical type";
a data processor: "character string field" data. results. measurement. lower _ LIMIT "is converted into a number type;
a data processor: "character string field" data. results. measures. upper _ LIMIT "is converted into a digital type;
a data processor: "character string field" data. results. measures. vresult "is converted into boolean type";
a data processor: "all field names are converted to lowercase".
Accordingly, the data processing model shown in FIG. 8 involves the following operational steps:
1. judging whether the data has a field of 'DATA.RESULTS.MEASUREMENTS', if so, continuing, and if not, skipping;
2. the value of the TIME field is converted into millisecond number (accurate to second) and is assigned to the field TIME _ sec, so that the computer operation is facilitated;
3. the value of the TIME field is converted into the number of milliseconds (accurate to the hour) and is assigned to the field TIME _ hour, so that the computer operation is facilitated;
4. the value of the TIME field is converted into millisecond number (accurate to day) and is assigned to the field TIME _ day, so that the computer operation is facilitated;
5. converting the string field "data.results.measures.value" to a numeric type;
6. converting the string field "data. results. measures. normal" to a numeric type;
7. converting the character string field "data. results. measurement. lower _ LIMIT" into a digital type (such as a floating point type);
8. converting the string field "data. results. measures. upper _ LIMIT" to a digital type (such as a floating point type);
9. converting the string field "data
10. All field names are converted to lower case.
The middle part in fig. 8 is a logic combination of each data operation component, and the functions from top to bottom correspond to the operation steps 1 to 10 respectively.
If the processing steps corresponding to the data processing model shown in fig. 8 are realized by writing codes, from development to testing to online operation, several hours are needed at the fastest, and the data processing model provided by the embodiment of the disclosure can be completed only in about several minutes, so that the requirements on the technical level of users are reduced.
According to the embodiment of the application, the instruction sets corresponding to basic function functions or specific logics are packaged into the data processing assemblies, the data processing assemblies can be connected at will, the sequence can be modified at will, meanwhile, a simple and easy-to-use operation interface is provided, the labor cost of data processing is reduced, and the real-time performance and the efficiency of data processing are improved.
And step 230, responding to the operation instruction aiming at the data processing model, and acquiring target data corresponding to the data processing model.
Optionally, the target data includes historical measurement data and real-time streaming data. Optionally, the historical measurement data or the real-time streaming data includes measurement data of the component, such as measurement data of dimensions, parameters, and the like.
In an exemplary embodiment, the preset page includes a model selection area, the model selection area includes at least one data processing model and at least one operation option corresponding to the data processing model, and the operation option includes a start option and a close option.
In one example, as shown in FIG. 9, a diagram two of a preset page is exemplarily shown. The preset page 90 shown in fig. 9 includes a model selection area 91 including a plurality of selectable data processing models 92. Each data processing model 92 has corresponding operating options 93, and a user can control the starting and stopping of the data processing model 92 through the operating options 93. The model selection area further includes a full start button 94 and a full stop button 95, and the user can start all the data processing models 92 by one key through the full start button 94 and stop all the data processing models 92 by one key through the full stop button 95.
Correspondingly, the triggering mode of the operation instruction comprises the following steps: and generating the running instruction in response to the selection operation of the starting option corresponding to the data processing model. In addition, the terminal can also respond to the selection operation of the closing option corresponding to the data processing model to generate a running stopping instruction; in response to the stop execution instruction, execution of the data processing operation is stopped.
In one example, as shown in FIG. 10, an operational state diagram of a multiple data processing model is illustrated. The data processing model shown in fig. 10 includes model 1, model 2, and model 3. In this case, model 1 is in a stopped state, and both models 2 and 3 are in an operating state.
Through the interactive inlet for managing the multiple data processing models, provided by the embodiment of the application, the starting and stopping of the single or multiple data processing models can be respectively edited and controlled, and the implementation personnel can conveniently process multiple different business requirements at the same time.
Optionally, the data processing model comprises a source data acquisition component. Accordingly, target data is obtained based on the source data obtaining component in response to the execution instructions for the data processing model.
The target data comprises at least one of real-time measurement data or historical measurement data corresponding to the source data acquisition component.
Optionally, the target data is collected according to a preset period based on the source data obtaining component.
In one possible embodiment, the manner of acquiring the target data includes: the inside data transfer that adopts the mode of wholesale data circulation of procedure, the washing of native support real-time data is handled, provides historical data backtracking subassembly simultaneously, can handle historical data, has solved the problem of data real-time nature, has kept historical data processing's ability simultaneously.
And 240, executing data processing operation corresponding to the target data processing component on the target data according to the connection relation information to obtain a data processing result.
For any target data processing component, such as a data operation component, determining a last data processing component corresponding to the data operation component according to the connection relation information; taking the output data corresponding to the previous data processing assembly as the input data corresponding to the target data processing assembly; executing data processing operation corresponding to the data operation component on the input data to obtain output data corresponding to the data operation component; the data processing results can be generated based on the output data.
In an exemplary embodiment, the data processing model includes a data manipulation component and a data output component. Accordingly, as shown in FIG. 7, the implementation of step 240 includes the following steps (241-242).
And 241, executing data processing operation corresponding to the data operation component on the target data according to the connection relation information, and generating a data processing result.
Optionally, the data manipulation component comprises a data checker. Correspondingly, according to the connection relation information, data verification operation corresponding to the data verifier is performed on the target data, and a data verification result is obtained, wherein the data processing result comprises the data verification result.
Optionally, the data manipulation component comprises a data processor. Correspondingly, according to the connection relation information, the data processing operation corresponding to the data processor is executed on the target data, and a data processing result is generated.
The data processing operation corresponding to the data processor comprises at least one of a data adding operation, a data deleting operation, a data modifying operation, a format conversion operation, a type conversion operation or a data acquiring operation.
And 242, outputting a data processing result based on the data output component.
In one example, as shown in FIG. 11, a data processing flow diagram based on a data processing model is illustrated. And starting the corresponding data processing flow after starting the data processing model. Target data is obtained based on the source data obtaining component.
Judging whether the next data operation assembly is a data processor or not;
if the next data operation component is a data processor, performing data processing operation on the input data based on the data processor to obtain output data; judging whether the data list needs to be modified or not; if not, starting to execute the step of judging whether the next data operation assembly is the data processor or not; if yes, circularly updating the data list, and starting to execute the step of judging whether the next data operation assembly is the data processor or not.
If the next data operation component is not the data processor, the next data operation component is a data checker, and data check operation is performed on input data based on the data checker to judge whether the data is correct; if the data operation component is correct, the step of judging whether the next data operation component is the data processor is started to be executed; if not, continuing the subsequent flow.
And under the condition that all the data operation components are traversed, outputting a data processing result based on the data output component.
In one example, as shown in FIG. 12, a diagram illustrating the results of data processing is shown. Through the data processing model shown in fig. 8, after the corresponding data processing operation is performed on the data in part a of fig. 12, the obtained data processing result is shown in part B.
To sum up, according to the technical scheme provided by the embodiment of the application, the data processing model can be generated by receiving the combined operation of the data processing components in the preset page, the target data corresponding to the data processing model can be acquired after the operation instruction of the data processing model is received, and the data processing operation corresponding to the data processing components is performed on the target data according to the connection relation before the data processing components in the data processing model, so that the data processing result is obtained, a technician can configure the data processing flow through the form of the combined components, the labor cost of data processing is reduced, and the real-time performance and the efficiency of data processing are improved.
In some practical applications, an implementer can log in a preset page through a browser, drag a component through a mouse to construct a data cleaning process, and click a single component to edit and modify component attributes. And combining the edited components together in a certain logic arrangement to complete a model. The whole process can be completed in a short time of several minutes. Finally, the implementer clicks the operation button to start the data cleaning process, so that the real-time data cleaning or historical data cleaning work is completed, the labor cost of data processing is reduced, and the real-time performance and the efficiency of data processing are improved.
FIG. 13 is a block diagram illustrating a data processing apparatus according to an example embodiment. Referring to fig. 13, the apparatus 1300 includes: a page display module 1310, a data acquisition module 1330, and a data processing module 1340.
A page display module 1310 configured to display a preset page, where the preset page includes at least one data processing component;
a model generating module 1320, configured to generate a data processing model in response to a combination operation for a target data processing component, where the target data processing component is a data processing component of the at least one data processing component, and the data processing model includes the target data processing component and connection relationship information corresponding to the target data processing component;
a data obtaining module 1330, configured to, in response to the operation instruction for the data processing model, obtain target data corresponding to the data processing model;
the data processing module 1340 is configured to execute, according to the connection relationship information, a data processing operation corresponding to the target data processing component on the target data, so as to obtain a data processing result.
In an exemplary embodiment, the preset page includes a component selection area and a model editing area, the component selection area includes the at least one data processing component, and the model generation module 1320 includes:
the component display unit is used for responding to the component dragging operation received by the component selection area and displaying the target data processing component in the model editing area;
a connection information determination unit configured to determine the connection relationship information in response to a component connection operation for the target data processing component;
and the model generating unit is used for generating the data processing model based on the target data processing assembly and the connection relation information.
In an exemplary embodiment, the data processing model includes a source data acquisition component, and the data acquisition module 1330 is specifically configured to acquire the target data based on the source data acquisition component in response to an execution instruction for the data processing model;
the target data comprises at least one of real-time measurement data or historical measurement data corresponding to the source data acquisition assembly.
In an exemplary embodiment, the data processing model includes a data manipulation component and a data output component, and the data processing module 1340 includes:
a result generating unit, configured to execute, according to the connection relationship information, a data processing operation corresponding to the data operation component on the target data, and generate the data processing result;
and the result output unit is used for outputting the data processing result based on the data output component.
In an exemplary embodiment, the data operation component includes a data checker, and the result generation unit includes:
and the verification result generating subunit is configured to execute a data verification operation corresponding to the data verifier on the target data according to the connection relationship information, so as to obtain a data verification result, and the data processing result includes the data verification result.
In an exemplary embodiment, the data manipulation component includes a data processor, and the result generation unit further includes:
a processing result generating subunit, configured to execute, according to the connection relationship information, a data processing operation corresponding to the data processor on the target data, and generate the data processing result;
the data processing operation corresponding to the data processor comprises at least one of a data adding operation, a data deleting operation, a data modifying operation, a format conversion operation, a type conversion operation or a data acquiring operation.
In an exemplary embodiment, the apparatus 1300 further comprises:
the instruction set acquisition module is used for acquiring at least one instruction set, wherein the at least one instruction set refers to an instruction set corresponding to the data processing operation;
and the component generating module is used for packaging the at least one instruction set and generating the at least one data processing component.
In an exemplary embodiment, the preset page includes a model selection area, the model selection area includes at least one data processing model and operation options corresponding to the at least one data processing model, the operation options include a start option and a close option, the apparatus 1300 further includes:
the operation instruction generating module is used for responding to the selection operation of the starting option corresponding to the data processing model and generating the operation instruction;
the device further comprises:
the stop instruction generating module is used for responding to the selection operation of the closing option corresponding to the data processing model and generating a stop operation instruction;
the data processing module 1340 is further configured to stop executing the data processing operation in response to the stop operation instruction.
To sum up, according to the technical scheme provided by the embodiment of the application, the data processing model can be generated by receiving the combined operation of the data processing components in the preset page, the target data corresponding to the data processing model can be acquired after the operation instruction of the data processing model is received, and the data processing operation corresponding to the data processing components is performed on the target data according to the connection relation before the data processing components in the data processing model, so that the data processing result is obtained, a technician can configure the data processing flow through the form of the combined components, the labor cost of data processing is reduced, and the real-time performance and the efficiency of data processing are improved.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Fig. 14 is a block diagram illustrating an electronic device for data processing, which may be a terminal, according to an exemplary embodiment, an internal structure of which may be as shown in fig. 14. The electronic device comprises a processor, a memory, a network interface, a display screen and an input device which are connected through a system bus. Wherein the processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic equipment comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the electronic device is used for connecting and communicating with an external terminal through a network. The computer program is executed by a processor to implement a data processing method. The display screen of the electronic equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the electronic equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 14 is merely a block diagram of some of the structures associated with the disclosed aspects and does not constitute a limitation on the electronic devices to which the disclosed aspects apply, as a particular electronic device may include more or less components than those shown, or combine certain components, or have a different arrangement of components.
In an exemplary embodiment, there is also provided an electronic device including: a processor; a memory for storing the processor-executable instructions; wherein the processor is configured to execute the instructions to implement a data processing method as in embodiments of the present disclosure.
In an exemplary embodiment, there is also provided a computer-readable storage medium, in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform a data processing method in an embodiment of the present disclosure.
In an exemplary embodiment, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the data processing method in the embodiments of the present disclosure.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), Rambus (Rambus) direct RAM (RDRAM), direct bused dynamic RAM (DRDRAM), and bused dynamic RAM (RDRAM).
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (12)
1. A method of data processing, comprising:
displaying a preset page, wherein the preset page comprises at least one data processing component;
generating a data processing model in response to a combined operation aiming at a target data processing component, wherein the target data processing component is a data processing component in the at least one data processing component, and the data processing model comprises the target data processing component and connection relation information corresponding to the target data processing component;
responding to an operation instruction aiming at the data processing model, and acquiring target data corresponding to the data processing model;
and executing data processing operation corresponding to the target data processing component on the target data according to the connection relation information to obtain a data processing result.
2. The data processing method of claim 1, wherein the preset page includes a component selection area and a model editing area, the component selection area includes the at least one data processing component, and the generating a data processing model in response to a combined operation for a target data processing component includes:
displaying the target data processing component in the model editing area in response to the component dragging operation received by the component selection area;
determining the connection relationship information in response to a component connection operation for the target data processing component;
and generating the data processing model based on the target data processing component and the connection relation information.
3. The data processing method of claim 1, wherein the data processing model comprises a source data acquisition component, and the acquiring target data corresponding to the data processing model in response to the execution instruction for the data processing model comprises:
in response to an execution instruction for the data processing model, acquiring the target data based on the source data acquisition component;
the target data comprises at least one of real-time measurement data or historical measurement data corresponding to the source data acquisition component.
4. The data processing method according to claim 1, wherein the data processing model includes a data operation component and a data output component, and the performing, according to the connection relationship information, a data processing operation corresponding to the target data processing component on the target data to obtain a data processing result includes:
according to the connection relation information, performing data processing operation corresponding to the data operation component on the target data to generate a data processing result;
and outputting the data processing result based on the data output component.
5. The data processing method according to claim 4, wherein the data operation component includes a data checker, and the performing, according to the connection relationship information, the data processing operation corresponding to the data operation component on the target data to generate the data processing result includes:
and executing data verification operation corresponding to the data verifier on the target data according to the connection relation information to obtain a data verification result, wherein the data processing result comprises the data verification result.
6. The data processing method according to claim 4, wherein the data operation component includes a data processor, and the performing, according to the connection relationship information, a data processing operation corresponding to the data operation component on the target data to generate the data processing result includes:
executing data processing operation corresponding to the data processor on the target data according to the connection relation information to generate a data processing result;
the data processing operation corresponding to the data processor comprises at least one of a data adding operation, a data deleting operation, a data modifying operation, a format conversion operation, a type conversion operation or a data acquiring operation.
7. The data processing method of claim 1, wherein the method further comprises:
acquiring at least one instruction set, wherein the at least one instruction set refers to an instruction set corresponding to the data processing operation;
and performing encapsulation processing on the at least one instruction set to generate the at least one data processing component.
8. The data processing method according to claim 1, wherein the preset page includes a model selection area, the model selection area includes at least one data processing model and operation options corresponding to the at least one data processing model, the operation options include a start option and a close option, and before the target data corresponding to the data processing model is obtained in response to the execution instruction for the data processing model, the method further includes:
responding to the selection operation of the starting option corresponding to the data processing model, and generating the running instruction;
the method further comprises the following steps:
responding to the selection operation of the closing option corresponding to the data processing model, and generating a stop operation instruction;
stopping execution of the data processing operation in response to the stop-run instruction.
9. A data processing apparatus, comprising:
the page display module is used for displaying a preset page, and the preset page comprises at least one data processing component;
a model generation module, configured to generate a data processing model in response to a combined operation for a target data processing component, where the target data processing component is a data processing component of the at least one data processing component, and the data processing model includes the target data processing component and connection relationship information corresponding to the target data processing component;
the data acquisition module is used for responding to an operation instruction aiming at the data processing model and acquiring target data corresponding to the data processing model;
and the data processing module is used for executing data processing operation corresponding to the target data processing component on the target data according to the connection relation information to obtain a data processing result.
10. An electronic device, comprising:
a processor;
a memory for storing the processor-executable instructions;
wherein the processor is configured to execute the instructions to implement the data processing method of any one of claims 1 to 8.
11. A computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the data processing method of any one of claims 1 to 8.
12. A computer program product comprising computer instructions, characterized in that the computer instructions, when executed by a processor, implement the data processing method of any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210540644.4A CN114996249A (en) | 2022-05-17 | 2022-05-17 | Data processing method and device, electronic equipment, storage medium and product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210540644.4A CN114996249A (en) | 2022-05-17 | 2022-05-17 | Data processing method and device, electronic equipment, storage medium and product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114996249A true CN114996249A (en) | 2022-09-02 |
Family
ID=83026294
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210540644.4A Pending CN114996249A (en) | 2022-05-17 | 2022-05-17 | Data processing method and device, electronic equipment, storage medium and product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114996249A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015056942A1 (en) * | 2013-10-14 | 2015-04-23 | 김경철 | Gui sharing system and gui editing device |
| CN112749194A (en) * | 2020-06-03 | 2021-05-04 | 腾讯科技（深圳）有限公司 | Visualized data processing method and device, electronic equipment and readable storage medium |
| CN112783632A (en) * | 2021-02-07 | 2021-05-11 | 树根互联股份有限公司 | Stream calculation system, method, electronic device, and readable storage medium |
| CN113590447A (en) * | 2021-08-02 | 2021-11-02 | 北京百度网讯科技有限公司 | Buried point processing method and device |
-
2022
- 2022-05-17 CN CN202210540644.4A patent/CN114996249A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015056942A1 (en) * | 2013-10-14 | 2015-04-23 | 김경철 | Gui sharing system and gui editing device |
| CN112749194A (en) * | 2020-06-03 | 2021-05-04 | 腾讯科技（深圳）有限公司 | Visualized data processing method and device, electronic equipment and readable storage medium |
| CN112783632A (en) * | 2021-02-07 | 2021-05-11 | 树根互联股份有限公司 | Stream calculation system, method, electronic device, and readable storage medium |
| CN113590447A (en) * | 2021-08-02 | 2021-11-02 | 北京百度网讯科技有限公司 | Buried point processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108415832B (en) | Interface automation test method, device, equipment and storage medium | |
| US20220050769A1 (en) | Program testing method and apparatus, computer device, and storage medium | |
| CN108446116A (en) | Generation method, device, computer equipment and the storage medium of the application program page | |
| CN109388561A (en) | Interface testing case generation method, device, computer equipment and storage medium | |
| CN111061526B (en) | Automatic test method, device, computer equipment and storage medium | |
| CN108388515B (en) | Test data generation method, device, equipment and computer readable storage medium | |
| CN107133174A (en) | Test case code automatically generating device and method | |
| CN106648556B (en) | Method and device for front-end and back-end integrated development test | |
| CN110069297A (en) | Abnormality eliminating method, device, computer equipment and storage medium based on Spring MVC | |
| CN111930617B (en) | Automatic test method and device based on data objectification | |
| CN110119352A (en) | UI adjustment method, device, computer equipment and the storage medium of application program | |
| CN109871312B (en) | Interface testing method, device, equipment and readable storage medium | |
| CN112486828A (en) | Test case generation method and device, computer equipment and storage medium | |
| CN111597110A (en) | Page testing method and device, electronic equipment and storage medium | |
| CN110334303A (en) | Form validation method, device, computer equipment and storage medium | |
| CN110232018A (en) | Interface test method, device, computer equipment | |
| CN107357721B (en) | Method and device for testing system | |
| CN112650689A (en) | Test method, test device, electronic equipment and storage medium | |
| CN112561690A (en) | Method, system, equipment and storage medium for testing credit card staging service interface | |
| CN114996249A (en) | Data processing method and device, electronic equipment, storage medium and product | |
| CN109614188A (en) | A kind of page online help method, apparatus, computer equipment and storage medium | |
| CN112667490A (en) | Performance test method and device based on hook function | |
| CN114637672A (en) | Automatic data testing method and device, computer equipment and storage medium | |
| CN113672222A (en) | Application program interface management device and construction method thereof | |
| CN113608995A (en) | Number making method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 215011 No. 2, Kunlunshan Road, high tech Zone, Suzhou, Jiangsu Applicant after: Suzhou Jiaqishi Technology Co.,Ltd. Address before: 215011 No. 2, Kunlunshan Road, high tech Zone, Suzhou, Jiangsu Applicant before: SUZHOU JIAQISHI INFORMATION SCIENCE & TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information |