CN115794764A - Data processing method and device, electronic equipment and readable storage medium - Google Patents
Data processing method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN115794764A CN115794764A CN202211412371.1A CN202211412371A CN115794764A CN 115794764 A CN115794764 A CN 115794764A CN 202211412371 A CN202211412371 A CN 202211412371A CN 115794764 A CN115794764 A CN 115794764A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- target
- data processing
- processing engine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application provides a data processing method, a data processing device, an electronic device and a readable storage medium. The method comprises the following steps: acquiring a target data processing engine determined by a user based on service requirements and at least one target service processing content; the target data processing engine is one of a plurality of data processing engines in the electronic device that are modular; acquiring a plurality of data to be processed through the target data processing engine, and processing each data to be processed according to the target service processing content to acquire a plurality of processed data; and outputting each processed data through the target data processing engine when each to-be-processed data is processed. By adopting the method, the data processing system is convenient to maintain.
Description
Technical Field
The present disclosure relates to internet data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a readable storage medium.
Background
The internet data processing technology mainly comprises the steps of data acquisition, data analysis, data processing and analysis from the perspective of platform design, and the mining efficiency of internet data is improved by optimizing the overall process of internet data processing. The data analysis module is a key component of internet data processing, and a data processing method is provided for improving the data processing efficiency of the data analysis module.
In the current data processing method, a single calculation engine is mainly used in the data processing method, for example, a data processing system only uses a batch calculation engine to deal with a batch processing scenario, or only uses a stream processing calculation engine to deal with a stream processing scenario, and when a service of the data processing system needs real-time or batch processing, the entire system needs to be switched, which is inconvenient for maintaining the data processing system.
Disclosure of Invention
The application provides a data processing method, a data processing device, an electronic device and a readable storage medium, which are used for solving the technical problem that a system is inconvenient to maintain in the prior art.
In a first aspect, the present application provides a data processing method, including:
acquiring a target data processing engine determined by a user based on business requirements and at least one target business processing content, wherein the target data processing engine is one of a plurality of modularized data processing engines in the electronic equipment;
acquiring a plurality of data to be processed through the target data processing engine, and processing each data to be processed according to the target service processing content to acquire a plurality of processed data;
and outputting each processed data through the target data processing engine when each to-be-processed data is processed.
In one embodiment, the obtaining of the target data processing engine determined by the user based on the business requirement and at least one target business processing content includes:
generating a corresponding identification signal in response to the operation of the user, wherein the identification signal comprises an identification of a target data processing engine and target business processing content;
and determining the target data processing engine selected by the user and at least one target business processing content according to the identification signal.
In an embodiment, the processing each of the to-be-processed data according to the target service processing content to obtain a plurality of processed data includes:
acquiring a target information code table corresponding to the target service processing content through the target data processing engine, wherein the target information code table comprises screening conditions for screening each piece of data to be processed, and the screening conditions are determined based on the service requirements;
and screening the data to be processed according to the screening conditions to obtain a plurality of processed data.
In one embodiment, the screening condition includes a target domain name; the screening of each to-be-processed data according to the screening conditions to obtain a plurality of processed data includes:
and determining any data to be processed consistent with the target domain name as processed data meeting the service requirement so as to obtain a plurality of processed data.
In one embodiment, before the obtaining of the target data processing engine determined by the user based on the business requirement and at least one target business processing content, the method further comprises:
separating each business processing content from each data processing engine, and solidifying each business processing content so that the plurality of business processing contents can be respectively called by each data processing engine.
In one embodiment, when each piece of the to-be-processed data is processed, outputting each piece of the processed data by the target data processing engine includes:
and when all the data to be processed are processed, transmitting all the processed data to a distributed file system through the target data processing engine, and storing the processed data in a corresponding storage path.
In one embodiment, the plurality of data processing engines includes at least a batch data processing engine and a stream data processing engine.
In a second aspect, the present application provides a data processing apparatus, the apparatus comprising:
the engine determining module is used for acquiring a target data processing engine determined by a user based on business requirements and at least one target business processing content; the target data processing engine is one of a plurality of data processing engines in the electronic device that are modular;
the data processing module is used for acquiring a plurality of data to be processed through the target data processing engine and processing each data to be processed according to the target service processing content to acquire a plurality of processed data;
and the data output module is used for outputting each processed data through the target data processing engine when each to-be-processed data is processed.
In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer execution instructions;
the processor executes computer-executable instructions stored by the memory to implement the method of the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon computer-executable instructions for implementing the method according to the first aspect when executed by a processor.
According to the data processing method, the data processing device, the electronic equipment and the readable storage medium, a target data processing engine determined by a user based on business requirements and at least one target business processing content are obtained; the target data processing engine is one of a plurality of data processing engines in the electronic device that are modular; acquiring a plurality of data to be processed through the target data processing engine, and processing each data to be processed according to the target service processing content to acquire a plurality of processed data; and outputting each processed data through the target data processing engine when each to-be-processed data is processed. In the application, the target data processing engine for processing each to-be-processed data according to the target service processing content is obtained by selecting one of the plurality of data processing engines based on the service requirement by the user. Namely, the target data processing engine can flexibly select according to the service requirement, and then process each data to be processed according to the target service processing content. Therefore, the whole data processing system has certain flexibility when the data processing engine is adopted to process data. When the service of the data processing system needs real-time or batch processing, the whole system does not need to be switched, thereby facilitating the maintenance of the data processing system.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
Fig. 1 is an application scenario diagram for implementing the data processing method according to the embodiment of the present application;
FIG. 2 is a schematic flowchart illustrating a method for implementing data processing according to an embodiment of the present application;
FIG. 3 is a schematic flow chart illustrating a method for implementing data processing according to another embodiment of the present application;
FIG. 4 is a schematic structural diagram of a data processing method implemented by the present application;
fig. 5 is a schematic structural diagram of an electronic device for implementing the data processing method.
With the above figures, there are shown specific embodiments of the present application, which will be described in more detail below. These drawings and written description are not intended to limit the scope of the inventive concepts in any manner, but rather to illustrate the inventive concepts to those skilled in the art by reference to specific embodiments.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
For a clear understanding of the technical solutions of the present application, a detailed description of the prior art solutions will be given first.
In a conventional manner, in an existing data processing method, a single calculation engine is mainly used in the data processing method, for example, a data processing system only uses a batch calculation engine to deal with a batch processing scenario, or only uses a stream processing calculation engine to deal with a stream processing scenario, and when a service of the data processing system needs real-time or batch processing, the entire system needs to be switched, which is inconvenient for maintaining the data processing system.
Therefore, in order to solve the technical problems in the prior art, the inventors found through creative research that maintenance of a data processing system is facilitated, and therefore, a plurality of data processing engines are modularized, and then, when data processing is required, a user performs corresponding operations based on business requirements, so that an electronic device obtains a corresponding target data processing engine and at least one target business processing content. And then, acquiring a plurality of data to be processed by a target data processing engine, processing each data to be processed according to the target service processing content, acquiring each processed data and outputting the processed data. The target data processing engine for processing each data to be processed according to the target service processing content is obtained by selecting one of a plurality of data processing engines based on service requirements by a user. Namely, the target data processing engine can flexibly select according to the service requirement and process each data to be processed according to the target service processing content. Therefore, the whole data processing system has certain flexibility when the data processing engine is adopted to process data. When the service of the data processing system needs real-time or batch processing, the whole system does not need to be switched, thereby facilitating the maintenance of the data processing system.
As shown in fig. 1, in an application scenario of the data processing method provided in the embodiment of the present application, a corresponding network architecture in the application scenario includes an electronic device 10, and the electronic device 10 obtains a target data processing engine determined by a user based on a service requirement, and at least one target service processing content. And then acquiring a plurality of data to be processed through the target data processing engine, and processing each data to be processed according to the target service processing content to acquire a plurality of processed data. And finally, outputting the processed data through the target data processing engine when the data to be processed are processed.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. These several specific embodiments may be combined with each other below, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Fig. 2 is a data processing method according to an embodiment of the present application, and as shown in fig. 2, an execution main body of the data processing method according to the embodiment is an electronic device. The data processing method provided in this embodiment includes the following steps:
The data processing engine is an engine capable of processing large-scale data. The data processing engine at least comprises a batch data processing engine and a stream data processing engine, wherein the batch data processing engine comprises a spark engine, and the stream data processing engine comprises a flash engine. In this embodiment, each batch data processing engine and each stream data processing engine are modularized, and flexible switching between batch data processing and stream data processing can be achieved. The target data processing engine is one of a plurality of data processing engines that are modularized in an electronic device.
The target service processing content refers to the content of processing data matched with the service requirement. In order to realize modularization of a plurality of data processing engines, a control part and a business part related to a data processing process can be separated firstly. The control part can be understood as a data processing process which is shared by the data processing engines to the business part. For the data analysis field of the internet, the service processing contents of the internet data can be divided into at least three categories: information identification, information filtering and information extraction. Here, the internet data refers to behavior data of a user accessing the internet, for example, the user xx accesses the xx website at the xx time.
The target service processing required for different service requirements is different, only information identification may be required, and information identification, information filtering and the like may also be required, that is, for the three types of service processing contents, different combination forms may be provided according to different service requirements.
The target data processing engine and at least one target business processing content are determined by a user according to business requirements, for example, if the business requirements are high in timeliness requirements on input/output data, the target data processing engine selected by the user is a streaming data processing engine; and if the service requirement is that only information identification is needed, the target service processing content selected by the user is the information identification.
And 102, acquiring a plurality of pieces of data to be processed through the target data processing engine, and processing each piece of data to be processed according to the target service processing content to acquire a plurality of pieces of processed data.
The data to be processed is service data to be processed, and can be obtained from a distributed file system (HDFS) through a target data processing engine.
And after the target data processing engine acquires a plurality of pieces of data to be processed, processing each piece of data to be processed according to the target service processing content. After the data to be processed is processed, the obtained data is called processed data.
For example, if the target service processing content is the information identifier, each piece of data to be processed is processed according to the target service processing content. Namely, information identification processing is performed on each piece of data to be processed. The information identifies the processed data, typically tagged with access information, e.g., if the user accesses a web page for xx cars, then the processed data is "potential user of xx cars".
And 103, outputting each processed data through the target data processing engine when each to-be-processed data is processed.
When the target data processing engine finishes processing each piece of data to be processed (namely, when each piece of data to be processed is finished), each piece of processed data is output through the target data processing engine.
In the method, a target data processing engine determined by a user based on business requirements and at least one target business processing content are obtained; the target data processing engine is one of a plurality of data processing engines in the electronic device that are modular; acquiring a plurality of data to be processed through the target data processing engine, and processing each data to be processed according to the target service processing content to acquire a plurality of processed data; and outputting each processed data through the target data processing engine when each to-be-processed data is processed. In the application, the target data processing engine for processing each to-be-processed data according to the target service processing content is obtained by selecting one of the plurality of data processing engines based on the service requirement by the user. Namely, the target data processing engine can flexibly select according to the service requirement, and then process each data to be processed according to the target service processing content. Therefore, the whole data processing system has certain flexibility when the data processing engine is adopted to process data. When the service of the data processing system needs real-time or batch processing, the whole system does not need to be switched, thereby facilitating the maintenance of the data processing system.
As an alternative implementation, as shown in fig. 3, in this embodiment, step 101 includes the following steps:
The electronic device may include an operation panel and/or a voice collecting device, where the operation panel may be used for a user to perform touch/click operations, and the voice collecting device may collect and recognize a voice uttered by the user. The user can determine the target data processing engine and/or at least one target business processing content by means of clicking operation and/or voice input.
After the user operation determines the target data processing engine and the target business processing content, the electronic equipment generates a corresponding identification signal in response to the user operation. The identification signal includes an identification of the targeted data processing engine and the targeted business process content.
The identification signal comprises the identification of the target data processing engine and the target business processing content, so that the electronic equipment can determine the target data processing engine selected by the user and at least one target business processing content according to the two items of information in the identification information.
Illustratively, the identifiers of the multiple data processing engines are A1 and A2, the identifiers of the multiple service processing contents are B1, B2 and B3, and the generated identification signal includes A1, B1 and B2, that is, it can be determined that the target data processing engine selected by the user is the data processing engine corresponding to the identifier of A1, and the target service processing contents are the service processing contents corresponding to the identifiers of B1 and B2, respectively.
In this embodiment, a corresponding identification signal is generated in response to the operation of the user, where the identification signal includes an identification of a target data processing engine and a target service processing content; and determining the target data processing engine selected by the user and at least one target business processing content according to the identification signal. Since the identification signal generated in response to the user operation has the identification of the target data processing engine and the target processing service content, the target data processing engine and the at least one target processing service content selected by the user can be accurately determined based on the identification signal.
As an optional implementation manner, in this embodiment, the step 102 includes the following steps:
step 301, obtaining, by the target data processing engine, a target information code table corresponding to the target service processing content.
The target information code table is set based on the service requirement and is used for screening the information code table of the data. The target information code table comprises screening conditions for screening the data to be processed, and the screening conditions are determined based on service requirements. Therefore, according to the target information code table, the data to be processed can be screened, and the data meeting the service requirements can be screened out.
Each service processing content has an information code table corresponding to it. For example, if the target service processing content is an information identifier, the corresponding target information code table is an information identifier table, the screening condition in the information identifier table includes at least one target domain name, that is, at least one domain name is preset, and when the domain name in any data to be processed is consistent with the at least one target domain name, the data to be processed is determined to be processed data meeting the service requirement, and a tag of the access information is printed, so as to obtain tag data of the corresponding access information.
The information identification code also comprises access information to be acquired, the access information to be acquired at least comprises a user agent (user agent) and a webpage address (url), and the user agent can be understood as agent information of a device terminal used by a user when the user accesses a webpage. In the foregoing process of tagging the access information, the access information is determined according to the access information to be obtained in the information identification code. Specifically, according to the access information to be acquired in the information identification code, the user agent corresponding to any data to be processed and the webpage address are acquired.
Step 302, screening each of the data to be processed according to the screening condition to obtain a plurality of processed data.
And the data to be processed is screened according to the screening conditions in the target information code table, so that a plurality of processed data meeting the service requirements can be obtained.
In this embodiment, a target information code table corresponding to the target service processing content is obtained by the target data processing engine, where the target information code table includes a screening condition for screening each piece of data to be processed, and the screening condition is determined based on the service requirement; and screening the data to be processed according to the screening conditions to obtain a plurality of processed data. Since the plurality of processed data are obtained based on the screening condition determined in the target information code table and the screening condition determined based on the service requirement, the plurality of processed data meeting the service requirement can be obtained by screening each to-be-processed data based on the screening condition.
As an optional implementation manner, in this embodiment, before step 101, the method further includes: separating each business processing content from each data processing engine, and solidifying each business processing content so that the plurality of business processing contents can be respectively called by each data processing engine.
The control part is separated from the service part, that is, each service processing content is separated from each data processing engine. After the business processing contents are solidified and the control part and the business part are separated, the business processing contents can be flexibly called by each data processing engine without adjusting the whole data processing system.
In this embodiment, each of the service processing contents is separated from each of the data processing engines, and each of the service processing contents is solidified, so that the plurality of service processing contents can be respectively called by each of the data processing engines, which can simplify maintenance of a data processing system and effectively improve reusability of a control part and a service part.
As an optional implementation manner, in this embodiment, step 103 includes: and when the data to be processed are processed, transmitting the processed data to a distributed file system through the target data processing engine and storing the processed data in a corresponding storage path.
Specifically, the process of outputting each processed data is to output each processed data to a corresponding storage path in the distributed file system through the target data processing engine for storage. When a subsequent user needs to use the processed data, sending a data acquisition request to the electronic equipment, wherein the data acquisition request comprises a processed data identifier; and acquiring corresponding processed data from the storage path of the distributed file system according to the processed data identifier.
In this embodiment, when each piece of to-be-processed data is processed, each piece of processed data is transmitted to the distributed file system through the target data processing engine and stored in the corresponding storage path. And when the processed data are output, the processed data are stored in the distributed file system, so that the subsequent users can read and use the processed data.
In one embodiment, the Spark engine executes the Spark program by calling a Spark parse builder, wherein Spark, when initiated, invokes a Spark context initialization module to initialize Spark context and Spark streamingcontext according to a selected schema. And constructors are mainly divided into a batch executor and a stream executor, wherein the two executors use a Spark engine based on a Spark executor, but can be used by distinguishing different scenes as the data processing scenes change. If the data is transmitted in batch for batch processing, selecting a batch processing executor; and if the data real-time transmission needs real-time processing, selecting a stream processing executor. Wherein, spark executor mainly includes three parts: rule reading, data analysis and custom writing.
And defining two modes of shunting processing and batch processing of the data writing-out module. And in the user-defined writing-out module, multi-path data writing-out can be carried out according to different services.
The written data volume of the batch processing data is large, so that the condition that the number of small files is too large can be avoided by carrying out partition statistics on the written data and redistributing the partitions to be written to the HDFS according to the statistical number.
And the data volume written out is correspondingly less due to the limited data volume of each batch in the flow processing calculation. In order to avoid the problem of too many small files, data writing out can be assisted by two tools, namely batch locking and output result accumulator. First, the batch lock can implement additional writing of multiple batches of data. In addition, the output result accumulator can be used for re-partitioning data in one batch, thereby reducing the number of small files. Specifically, the output result is in the form of a key value pair rdd (i.e., pair rdd), and the accumulator counting operation is performed on the same key value data in a batch, after the results of multiple batches are accumulated, the results are subjected to key averaging to specify new partitions of data with different key values, so that the data with the same key value are aggregated in the same partition as much as possible.
Fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application, and as shown in fig. 4, if the data processing apparatus 40 provided in this embodiment is located in an electronic device, the data processing apparatus 40 provided in this embodiment includes: an engine determination module 41, a data processing module 42 and a data output module 43. Wherein the content of the first and second substances,
an engine determining module 41, configured to obtain a target data processing engine determined by a user based on a service requirement, and at least one target service processing content; the target data processing engine is one of a plurality of data processing engines in the electronic device that are modular;
a data processing module 42, configured to obtain multiple pieces of data to be processed through the target data processing engine, and process each piece of data to be processed according to the target service processing content to obtain multiple pieces of processed data;
a data output module 43, configured to output each processed data through the target data processing engine when each to-be-processed data is processed.
Optionally, the engine determining module 41 is specifically configured to: generating a corresponding identification signal in response to the operation of the user, wherein the identification signal comprises an identification of a target data processing engine and target business processing content; and determining the target data processing engine selected by the user and at least one target business processing content according to the identification signal.
Optionally, the data processing module 42 is specifically configured to: acquiring a target information code table corresponding to the target service processing content through the target data processing engine, wherein the target information code table comprises screening conditions for screening each piece of data to be processed, and the screening conditions are determined based on the service requirements; and screening the data to be processed according to the screening conditions to obtain a plurality of processed data.
Optionally, the screening condition includes a target domain name; the data processing module 42, configured to, when screening each piece of data to be processed according to the screening condition to obtain a plurality of pieces of processed data, specifically: and determining any data to be processed consistent with the target domain name as processed data meeting the service requirement so as to obtain a plurality of processed data.
Optionally, the system further includes a service and control separation module, configured to: separating each business processing content from each data processing engine, and solidifying each business processing content so that the plurality of business processing contents can be respectively called by each data processing engine.
Optionally, the data output module 43 is specifically configured to: and when the data to be processed are processed, transmitting the processed data to a distributed file system through the target data processing engine and storing the processed data in a corresponding storage path.
Optionally, the plurality of data processing engines includes at least a batch data processing engine and a stream data processing engine.
Fig. 5 is a block diagram illustrating an electronic device according to an example embodiment, which may be the electronic device shown in fig. 5, including: a memory 51, a processor 52; memory 51 is a memory for storing processor-executable instructions; the processor 52 is used to run a computer program or instructions to implement the data processing method provided by any of the above embodiments.
The memory 51 stores programs. In particular, the program may include program code comprising computer operating instructions. The memory 51 may comprise high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.
The processor 52 may be a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement the embodiments of the present disclosure.
Alternatively, in a specific implementation, if the memory 51 and the processor 52 are implemented independently, the memory 51 and the processor 52 may be connected to each other through a bus 53 and perform communication with each other. The bus 53 may be an Industry Standard Architecture (ISA) bus 53, a Peripheral Component Interconnect (PCI) bus 53, an Extended ISA (EISA) bus 53, or the like. The bus 53 may be divided into an address bus 53, a data bus 53, a control bus 53, and the like. For ease of illustration, only one thick line is shown in fig. 5, but does not indicate only one bus 53 or one type of bus 53.
Alternatively, in a specific implementation, if the memory 51 and the processor 52 are integrated on a chip, the memory 51 and the processor 52 may complete the same communication through an internal interface.
A non-transitory computer-readable storage medium, in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform the data processing method of the electronic device.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.
Claims (10)
1. A data processing method is applied to an electronic device, and the method comprises the following steps:
acquiring a target data processing engine determined by a user based on service requirements and at least one target service processing content; the target data processing engine is one of a plurality of data processing engines in the electronic device that are modularized;
acquiring a plurality of data to be processed through the target data processing engine, and processing each data to be processed according to the target service processing content to acquire a plurality of processed data;
and outputting each processed data through the target data processing engine when each to-be-processed data is processed.
2. The method of claim 1, wherein the obtaining of the target data processing engine determined by the user based on the business requirement and the at least one target business processing content comprises:
generating a corresponding identification signal in response to the operation of the user, wherein the identification signal comprises an identification of a target data processing engine and target business processing content;
and determining the target data processing engine selected by the user and at least one target business processing content according to the identification signal.
3. The method according to claim 1, wherein said processing each of the to-be-processed data according to the target service processing content to obtain a plurality of processed data comprises:
acquiring a target information code table corresponding to the target service processing content through the target data processing engine, wherein the target information code table comprises screening conditions for screening each piece of data to be processed, and the screening conditions are determined based on the service requirements;
and screening the data to be processed according to the screening conditions to obtain a plurality of processed data.
4. The method of claim 3, wherein the screening condition comprises a target domain name; the screening of each to-be-processed data according to the screening conditions to obtain a plurality of processed data includes:
and determining any data to be processed consistent with the target domain name as processed data meeting the service requirement so as to obtain a plurality of processed data.
5. The method of claim 4, further comprising, before the obtaining the target data processing engine determined by the user based on the business requirement and the at least one target business processing content:
and separating each business processing content from each data processing engine, and solidifying each business processing content so that the plurality of business processing contents can be respectively called by each data processing engine.
6. The method of claim 3, wherein outputting, by the target data processing engine, each of the processed data when each of the to-be-processed data is processed comprises:
and when all the data to be processed are processed, transmitting all the processed data to a distributed file system through the target data processing engine, and storing the processed data in a corresponding storage path.
7. The method of any of claims 1-6, wherein the plurality of data processing engines comprises at least a batch data processing engine and a stream data processing engine.
8. A data processing apparatus, characterized in that the apparatus comprises:
the engine determining module is used for acquiring a target data processing engine determined by a user based on business requirements and at least one target business processing content; the target data processing engine is one of a plurality of data processing engines in an electronic device that are modular;
the data processing module is used for acquiring a plurality of data to be processed through the target data processing engine and processing each data to be processed according to the target service processing content to acquire a plurality of processed data;
and the data output module is used for outputting each processed data through the target data processing engine when each to-be-processed data is processed.
9. An electronic device, comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored by the memory to implement the method of any of claims 1-7.
10. A computer-readable storage medium having computer-executable instructions stored therein, which when executed by a processor, are configured to implement the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211412371.1A CN115794764A (en) | 2022-11-11 | 2022-11-11 | Data processing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211412371.1A CN115794764A (en) | 2022-11-11 | 2022-11-11 | Data processing method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115794764A true CN115794764A (en) | 2023-03-14 |
Family
ID=85436970
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211412371.1A Pending CN115794764A (en) | 2022-11-11 | 2022-11-11 | Data processing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115794764A (en) |
-
2022
- 2022-11-11 CN CN202211412371.1A patent/CN115794764A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210081950A1 (en) | Method and apparatus for identifying identity information | |
| KR101661000B1 (en) | Systems and methods to enable identification of different data sets | |
| CN110928851B (en) | Method, device and equipment for processing log information and storage medium | |
| CN114185708A (en) | Data analysis method and device based on distributed link tracking and electronic equipment | |
| CN110162512B (en) | Log retrieval method, device and storage medium | |
| CN110795756A (en) | Data desensitization method and device, computer equipment and computer readable storage medium | |
| CN113067883A (en) | Data transmission method and device, computer equipment and storage medium | |
| CN110780870A (en) | Service execution method, device, equipment and storage medium | |
| CN107295082A (en) | Running software processing method, apparatus and system | |
| US20200204688A1 (en) | Picture book sharing method and apparatus and system using the same | |
| CN107092553A (en) | A kind of method, device and computer system for setting up request call chain between process | |
| CN104219230A (en) | Method and device for identifying malicious websites | |
| CN107748772B (en) | Trademark identification method and device | |
| CN111881094A (en) | Method, device, terminal and storage medium for extracting key information in log | |
| CN111047434A (en) | Operation record generation method and device, computer equipment and storage medium | |
| CN114125015A (en) | Data acquisition method and system | |
| GB2521637A (en) | Messaging digest | |
| CN111198806B (en) | Service call data statistical analysis method and system based on service open platform | |
| CN117093619A (en) | Rule engine processing method and device, electronic equipment and storage medium | |
| CN115794764A (en) | Data processing method and device, electronic equipment and readable storage medium | |
| CN105893286A (en) | Data processing method and apparatus | |
| CN114168557A (en) | Processing method and device for access log, computer equipment and storage medium | |
| CN113204706B (en) | Data screening and extracting method and system based on MapReduce | |
| WO2021129849A1 (en) | Log processing method, apparatus and device, and storage medium | |
| CN112508526B (en) | Data auditing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |