CN117472356A

CN117472356A - Big data processing method and device

Info

Publication number: CN117472356A
Application number: CN202311491824.9A
Authority: CN
Inventors: 胡文涛; 李桂
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-11-09
Filing date: 2023-11-09
Publication date: 2024-01-30

Abstract

The embodiment of the application provides a big data processing method and device, relates to the field of big data processing, and can also be applied to the financial field, wherein the method comprises the following steps: acquiring data resources of a current big data service cloud in a data exchange platform, wherein the data resources comprise extraction, aggregate data resource definition and a corresponding rule base; displaying the data resources to a user through a front-end interaction platform based on a document objectification model for editing by the user; receiving rule information edited by a user on the data resource, and updating the rule information into independently deployed bloom filters through a mapping function; the method and the device can flexibly customize data processing, edit and adjust the data processing rules in real time, reduce production risks and improve response speed.

Description

Big data processing method and device

Technical Field

The application relates to the field of big data processing, and also can be applied to the field of finance, in particular to a big data processing method and device.

Background

Current data warehouses typically employ large data processing layering mechanisms that often pre-write data processing rules and methods in the system, which are not easily altered, any alteration requiring a series of cumbersome processes such as putting out requirements, writing new rules, implementing rule changes, and issuing new versions.

This results in the general problem of insufficient flexibility in the current flow. During use, the user's requirements in terms of the need for data processing may change or wish to alter the way or flow of data processing of the extraction layer. However, due to the fixity of the prior art, the modification needs to go through a complicated flow and a lot of work, such as waiting for the release of a new version, which may cause a delay in processing data. Thus, existing data warehouse data processing approaches are difficult to meet the rapidly changing needs of users due to the lack of immediate data processing flexibility.

Disclosure of Invention

Aiming at the problems in the prior art, the application provides a big data processing method and device, which can flexibly customize data processing, edit and adjust data processing rules in real time, reduce production risk and improve response speed.

In order to solve at least one of the above problems, the present application provides the following technical solutions:

according to a first aspect of an embodiment of the present application, the present application provides a big data processing method, including:

acquiring data resources of a current big data service cloud in a data exchange platform, wherein the data resources comprise extraction, aggregate data resource definition and a corresponding rule base;

Displaying the data resources to a user through a front-end interaction platform based on a document objectification model for editing by the user;

and receiving rule information edited by a user on the data resource, and updating the rule information into the independently deployed bloom filter through a mapping function.

According to any embodiment of the present application, the obtaining a data resource of a current big data service cloud in a data exchange platform includes:

and responding to the update of the data interaction platform, and caching the data resources of the updated big data service cloud.

According to any embodiment of the present application, the document-based objectification model presents the data resource to a user through a front-end interaction platform, for the user to edit, including:

nesting the data resources to a front-end interaction platform page through a document objectification model;

and carrying out identity verification on the user to be accessed, and displaying the data resource through the front-end interaction platform page under the condition that the user to be accessed has access rights.

According to any embodiment of the present application, the receiving rule information edited by the user on the data resource includes:

Responding to receiving a selection instruction of a user for the position of the extraction element page, and displaying a corresponding editing sub-interface;

and receiving an element naming and/or extraction rule editing result of the extraction element based on the editing sub-interface by a user, and storing the editing result in a view form to a front-end custom view database.

According to any embodiment of the present application, after receiving an element naming and/or extraction rule editing result of the extraction element by the user, the method further includes:

and carrying out injection attack detection on the editing result through the structured query language, and storing the editing result to a front-end custom-made attempt database in a view mode under the condition that the detection result meets the preset requirement.

According to any embodiment of the present application, before the receiving rule information edited by the user on the data resource, the method further includes:

searching the data elements to be edited in a bloom server, and determining whether the data elements to be edited are in a custom view database;

acquiring and running a custom view of the data element to be edited from the front-end interaction platform in response to the data element to be edited being in the custom view database;

And acquiring the aggregate data resource of the data element to be edited through the big data service cloud, and displaying the aggregate data resource to a user through a front-end interaction platform.

According to a second aspect of embodiments of the present application, there is provided a big data processing apparatus, comprising:

the rule extraction module is used for: acquiring data resources of a current big data service cloud in a data exchange platform, wherein the data resources comprise extraction, aggregate data resource definition and a corresponding rule base;

front end display module for: displaying the data resources to a user through a front-end interaction platform based on a document objectification model for editing by the user;

a rule editing module for: and receiving rule information edited by a user on the data resource, and updating the rule information into the independently deployed bloom filter through a mapping function.

According to any embodiment of the present application, the rule extraction module is specifically configured to:

According to any embodiment of the present application, the front end display module includes:

a data display unit for: nesting the data resources to a front-end interaction platform page through a document objectification model;

An identity verification unit for: and carrying out identity verification on the user to be accessed, and displaying the data resource through the front-end interaction platform page under the condition that the user to be accessed has access rights.

According to any embodiment of the present application, the rule editing module includes:

an instruction receiving unit configured to: responding to receiving a selection instruction of a user for the position of the extraction element page, and displaying a corresponding editing sub-interface;

a result storage unit configured to: and receiving an element naming and/or extraction rule editing result of the extraction element based on the editing sub-interface by a user, and storing the editing result in a view form to a front-end custom view database.

According to any embodiment of the present application, after receiving an element naming and/or extraction rule editing result of the extraction element by the user, the attack detection module is further configured to:

According to any embodiment of the present application, before the receiving rule information edited by the user on the data resource, the method further includes a retrieving module, configured to:

According to a third aspect of embodiments of the present application, there is provided an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the big data processing method when executing the program.

According to a fourth aspect of embodiments of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the big data processing method.

According to a fifth aspect of embodiments of the present application, there is provided a computer program product comprising computer programs/instructions which, when executed by a processor, implement the steps of the big data processing method.

As can be seen from the above technical solutions, the present application provides a big data processing method and apparatus, by acquiring data resources of a current big data service cloud in a data exchange platform, where the data resources include extraction, aggregated data resource definition and a corresponding rule base; displaying the data resources to a user through a front-end interaction platform based on a document objectification model for editing by the user; and receiving rule information edited by a user on the data resource, and updating the rule information into a bloom filter which is independently deployed through a mapping function, so that the data processing can be flexibly customized, the data processing rule can be edited and adjusted in real time, the production risk is reduced, and the response speed is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to the above drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a big data processing method according to an embodiment of the present application;

FIG. 2 is a multi-level variable rule data processing block diagram of a big data processing method in an embodiment of the present application;

FIG. 3 is a second flow chart of the big data processing method according to the embodiment of the present application;

FIG. 4 is a diagram illustrating an example DOM tree structure of a big data processing method according to an embodiment of the present application;

FIG. 5 is a third flow chart of the big data processing method according to the embodiment of the present application;

FIG. 6 is a flow chart of a big data processing method according to an embodiment of the present application;

FIG. 7 is one of the block diagrams of a big data processing apparatus in an embodiment of the present application;

fig. 8 is a schematic structural diagram of an electronic device in an embodiment of the present application.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

The data acquisition, storage, use, processing and the like in the technical scheme meet the relevant regulations of national laws and regulations.

The application provides a big data processing method and device, which flexibly customize data processing, edit and adjust data processing rules in real time, reduce production risk and improve response speed.

In order to flexibly customize data processing, edit and adjust data processing rules in real time, reduce production risk and increase response speed, the present application provides an embodiment of a big data processing method, referring to fig. 1, the big data processing method specifically includes the following contents:

step S101: and acquiring data resources of the current big data service cloud in the data exchange platform, wherein the data resources comprise extraction, aggregate data resource definition and a corresponding rule base.

The data exchange platform is used for acquiring data resources in the current big data service cloud, and the data resources comprise two main aspects:

the extraction data resource definition characterizes the definition of various data elements, tables, files and related information involved in the data processing layer in the big data service cloud, including the structure, format, attributes of the data, and rules how to process, etc., which determine how to extract, process and process the data from the original data source.

The aggregate data resource definition characterizes a definition of data resources for aggregate operations in a large data service cloud. Aggregation is the integration of information from multiple data sources to generate more valuable data, with the definitions described above relating to the aggregation, association, manner of aggregation, and other relevant rules and operations of the data.

Furthermore, the definitions of the data resources described above are associated with their corresponding rule base. The rule base contains processing rules applied to the data resources, including rules of data cleaning, conversion, processing and other operations.

In an optional embodiment, the acquiring the data resource of the current big data service cloud in the data exchange platform includes:

Preferably: in order to improve the flexibility and user experience of data processing, a variable rule processor is introduced into a big data service cloud, and extraction and aggregation data resource definition and an extraction rule base are obtained from a data exchange platform. The definition of the data resources and the rule base are then provided to the custom user front-end processor for use with the customized extraction layer data elements.

Typically, the big data service cloud will directly perform the resource definition and rule base processing, typically batch, timed or long-time-triggered operations. This approach is not well suited to quickly meet the user's custom needs due to lack of real-time.

By the NOS cache function of the variable rule processor. And each time a new version is released, the latest resource definition and rule library are acquired from the data interaction platform so as to ensure the timeliness and accuracy of the data, so that the related service platform can access the data resources in real time, and faster, flexible and personalized data processing experience is provided for the user.

Step S102: and displaying the data resources to a user through a front-end interaction platform based on a document objectification model for editing by the user.

Fig. 2 shows a multi-level variable rule data processing block diagram provided herein.

And displaying the data resources to the user in a DOM-based manner through the front-end interaction platform. The user may view, edit, or perform operations associated with the data resources on the front-end interface to better understand and use the resources.

The front-end interaction platform characterizes an interface for a user to interact with the system, and the user can interact with the system on the platform to view data and execute various operations.

The Document Object Model (DOM) is one way to represent and manipulate web pages or application structures. The HTML Document is taken as a root node, other nodes are taken as child nodes, and a tree-shaped data structure representation is organized. The front page can easily change the content, structure and style by invoking the DOM interface like js.

Elements (e.g., text, images, forms, etc.) of a web page or application are represented as a tree structure, each element having corresponding properties and methods. Allowing developers to programmatically access and modify the content of the page and respond to user interactions.

By presenting the content of the data asset to the user, various aspects of the data asset are displayed, typically in a visual manner, in text, graphics, etc., enabling the user to intuitively understand and manipulate the data.

In an alternative embodiment, as shown in fig. 3, the document-based objectification model presents the data resource to a user through a front-end interaction platform for the user to edit, including:

step S102A: nesting the data resources to a front-end interaction platform page through a document objectification model;

step S102B: and carrying out identity verification on the user to be accessed, and displaying the data resource through the front-end interaction platform page under the condition that the user to be accessed has access rights.

Wherein, a new functional module, namely a custom user front-end processor, can be introduced into the front-end processing of a specific service system, so that a user with specific authority (usually a person with specific authority in the interior) is allowed to customize data resources and acquire rules.

Users with access rights are typically specific personnel within an organization that have rights to customize data resources. Such users typically have special requirements for data processing and rule setting of the business system.

Illustratively, in banking systems, most banking employees, such as banking teller, customer service representative, etc., belong to common employees. The system may be used to find customer information, process transactions and provide basic banking services without generally having the right to modify data processing rules in the system.

While data specialists within banks are responsible for analyzing large amounts of transaction data in order to better understand customer behavior and risk factors, a higher level of authority is required to access and query the data, but data processing rules are also typically not self-modifying.

In addition, there are custom users in banks, typically advanced analysts or risk management specialists. Such users have special requirements for data processing and rule settings, as they need to adjust the way they process data according to the latest market trends, regulatory changes or specific risk situations. For example, they may need to create new data processing rules to better monitor abnormal transactions, or they may need to customize data reports to meet regulatory requirements.

The custom user has high-level authority of the system, and can autonomously manage and modify the data processing rules to meet special requirements through the front-end custom view database without waiting for the update of the whole system.

Data resource definitions and rules are typically complex structured data. Through a Document Object Model (DOM), the above data may be nested in the pages of the front-end interaction platform to present a user-friendly interface. The elements of the data resource are displayed on the front page in a tree structure mode, so that a user can easily browse and edit the data resource.

Illustratively, the functions of the DOM tree processing and other functional modules include:

and acquiring the current extraction/aggregation data resource definition and rule base comprising data processing rules by interacting with the data exchange platform. The rules specify how the different data elements are processed, including filtering, computing, aggregating, etc.

Once the data resource definitions and rule base are obtained, the functional module organizes the information into a tree structure, the DOM tree. Each data element is a node in the DOM tree, and rules and definition information related to the data element are nested inside the node, so that the structure of the data resource is clearly visible, and the operation of a user is facilitated.

Through the front-end interaction platform, the user can browse the DOM tree, click on the node to view the data resource definition and rules, and then edit as required.

FIG. 4 shows an example DOM tree structure disclosed in the present application:

the example shown in fig. 4 contains two extraction elements, labeled "extraction element 1" and "extraction element 2", respectively. Each extraction element includes the following key information:

element name: the name or identifier characterizing the extraction element.

Defining data resources: the definition is obtained from NOS in real time, including the unique identifier (id) of the element, the data type, whether the element contains sensitive information, the data update period and other important information.

Extraction rule: rules indicating how data is extracted from the data resources. The rules may be complex SQL query statements or other data processing methods.

Associated aggregation elements: characterizing the polymeric element associated with the extraction element. The above-mentioned association relationship is also obtained in real time from NOS.

When a user wishes to edit an extraction element, the operation can be performed through the front-end interface. The user's edit request typically triggers a mouse click event to pop up an edit sub-page. The page will display the relevant editing options, allowing the user to modify the element names and rules.

When the user opens the editing sub-page, the system will extract relevant data resource definitions, rules and other information from the NOS in real time to populate the editing sub-page. Ensuring that the information seen by the user is up to date for editing.

In addition, the architecture and design of the system are not only suitable for modifying the existing extraction elements, but also for directly adding the extraction elements. The bottom structure and design of the system are not required to be modified on a large scale, and the requirement of newly added elements can be met by only slightly adjusting front-end page processing generally without complex system modification.

According to the method and the device, the custom user front-end processor is introduced, so that a user with authority can access the definition and rules of the data resources of the extraction layer through the front-end interface, the user can customize the data resources in a visual mode, and the flexibility and user experience of data processing are improved.

Step S103: and receiving rule information edited by a user on the data resource, and updating the rule information into the independently deployed bloom filter through a mapping function.

The bloom filter is a functional mechanism that retrieves whether an element is in a set, and consists of a very long binary vector and a series of random mapping functions for optimizing space efficiency and query time.

Once the user edits the rule information, the information may be transformed and updated into a separately deployed bloom filter via a mapping function. The bloom filter is a data structure used to determine whether an element exists in a collection, typically for efficient retrieval of the element. In this application, bloom filters are used to store edited rule information independent of other data stores.

As can be seen from the above description, in the big data processing method provided in the embodiments of the present application, by adding the variable rule module in the big data service cloud, a higher level of flexibility is introduced in the data processing architecture by a technical manner, so that the user is closer to the final requirement, thereby meeting the finer data processing requirement. The user can customize the data processing rules without waiting for a new system version to be released.

According to the method and the system, the DOM tree is constructed by extracting the information of the related data elements from the data resource definition and extraction aggregation rules, so that a user can more intuitively know the structures of the data resources and the rules, and the data resources and the rules can be visually and conveniently edited.

In addition, in the application, the extraction rule of the specific data element can be directly customized, and the extraction rule is usually realized through dynamic SQL, and the traditional flow of 'demand to edition' is not needed, so that the extraction rule has higher response speed and flexibility in the aspect of data processing. This helps to avoid potential production risks, since rule changes do not cause modification of the entire system, but only need to affect specific elements.

Finally, the method ensures that the rule information can be efficiently updated into the bloom filter by designing a mapping function mechanism for quick verification and retrieval.

In an embodiment of the big data processing method of the present application, referring to fig. 5, the receiving rule information edited by the user on the data resource may further specifically include the following:

step S103A: and responding to receiving a selection instruction of a user for the position of the extraction element page, and displaying a corresponding editing sub-interface.

Step S103B: and receiving an element naming and/or extraction rule editing result of the extraction element based on the editing sub-interface by a user, and storing the editing result in a view form to a front-end custom view database.

The user decides to edit a certain extraction element by selecting a specific position on the page. The system responds to the user selected position instruction to display an editing sub-interface corresponding to the selected position.

The interface is typically the interface that the user uses to edit the extraction elements of the data resource, and may include various editing options and tools for modifying the naming of the elements and defining the extraction rules.

The user may perform editing operations on the selected extraction element on the editing sub-interface, including modifying the name of the element and defining extraction rules. The user may enter a new name or edit rule on the edit sub-interface to meet his particular needs.

Illustratively, after receiving the element naming and/or extraction rule editing result of the extraction element by the user, the method further comprises:

The user modifies the naming of the extraction layer data elements, extracts the rules, and sends the customized rules (usually complex SQL statement combinations) to the dynamic SQL security module, and the rules are stored as views after the checking pass and put into the front-end customized view database. The editing result is stored in the front-end custom view database in the form of a view.

In an embodiment of the big data processing method of the present application, referring to fig. 6, before the rule information edited by the receiving user on the data resource, the following may further be specifically included:

step S104: searching the data elements to be edited in a bloom server, and determining whether the data elements to be edited are in a custom view database;

step S105: acquiring and running a custom view of the data element to be edited from the front-end interaction platform in response to the data element to be edited being in the custom view database;

Step S106: and acquiring the aggregate data resource of the data element to be edited through the big data service cloud, and displaying the aggregate data resource to a user through a front-end interaction platform.

When the user front-end accesses the extraction layer data elements, a bloom filter check is first performed to detect whether a particular data element is present in the custom view data.

If the bloom filter detects that the element is present in the custom view data, it indicates that the element has been custom made by a user and can be found in the custom view, a rule or machining method that characterizes the element that the user has edited.

If the element is present in the custom view data, the system will continue to pass the request to the custom user front-end processor.

The front-end processor may run corresponding custom views including custom rules for the elements by the user, typically complex SQL queries or other processing methods, which may be run in real-time to process the data according to the user's needs.

Complex SQL statements in custom views require access to the data elements of the aggregation layer to obtain the required data. The system accomplishes this by way of an aggregation layer data element access interface on the cloud.

Finally, the front-end service processor returns the operation result to the front end of the general user for the user to check or further process. The result is typically data after the data has been processed according to custom rules for the user.

At present, the big data processing function of the data warehouse on the cloud is to store the result as an extraction layer data element after the operation of the data elements of the aggregation layer according to the rule. The front end or other system of a particular business system may then directly obtain the values of the elements described above.

In existing architectures, the big data service cloud alone provides one aggregation layer data element access interface for accessing the path of the data resources of the aggregation layer. The relevant rules and data processing operations are completed by the front-end service processor.

In this application, considering that the service cloud is used by multiple systems, and the customized user usually belongs to a specific service system, the optimized architecture no longer needs to calculate and store extraction layer data elements of the specific service system on the cloud. Instead, the cloud only needs to provide an interface that enables the custom user to access existing aggregation layer data elements.

By the optimization method, data redundancy can be reduced, and the calculated result is not required to be stored as a new extraction layer data element. The same data does not need to be repeatedly stored among different systems, so that the data storage cost and the maintenance complexity are reduced.

Furthermore, this optimization also reduces the security risk of running on the cloud, since the rules and machining operations of the custom user are all performed on the respective front-end business processor, since the complex SQL of the specific business system is no longer running on the cloud.

If the data element to be edited is not in the custom view database, the system will continue to operate according to the original processing flow, i.e. the front-end service processor will directly access the extraction layer data on the big data service cloud.

In particular, this is because the system employs bloom filters to determine whether a data element exists in the custom view database. Although the false recognition rate of the bloom filter is very low, i.e., the probability of false positives is very small, in rare cases false positives may occur. This means that there is little likelihood that the bloom filter may erroneously identify as present a data element that is not actually present in the custom view database.

In order to cope with the situation, the system still provides an original processing flow, allows the front-end service processor to directly access the extraction layer data on the cloud, ensures the robustness and the reliability of the system, and can normally process the request of a user even if false positive occurs, so as to avoid the problem of data access.

In order to flexibly customize data processing, edit and adjust data processing rules in real time, reduce production risk and increase response speed, the present application provides an embodiment of a big data processing device for implementing all or part of the content of the big data processing method, see fig. 7, where the big data processing device specifically includes:

a rule extraction module 1101, configured to: acquiring data resources of a current big data service cloud in a data exchange platform, wherein the data resources comprise extraction, aggregate data resource definition and a corresponding rule base;

a front end display module 1102, configured to: displaying the data resources to a user through a front-end interaction platform based on a document objectification model for editing by the user;

a rule editing module 1103 for: and receiving rule information edited by a user on the data resource, and updating the rule information into the independently deployed bloom filter through a mapping function.

In order to flexibly customize data processing, edit and adjust data processing rules in real time, reduce production risk and increase response speed from a hardware aspect, the application provides an embodiment of an electronic device for implementing all or part of contents in the big data processing method, where the electronic device specifically includes the following contents:

a processor (processor), a memory (memory), a communication interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete communication with each other through the bus; the communication interface is used for realizing information transmission between the big data processing device and related equipment such as a core service system, a user terminal, a related database and the like; the logic controller may be a desktop computer, a tablet computer, a mobile terminal, etc., and the embodiment is not limited thereto. In this embodiment, the logic controller may refer to the embodiment of the big data processing method in the embodiment and the embodiment of the big data processing apparatus, and the contents thereof are incorporated herein, and the repetition is omitted.

It is understood that the user terminal may include a smart phone, a tablet electronic device, a network set top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Wherein, intelligent wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..

In practical applications, part of the big data processing method may be performed on the electronic device side as described above, or all operations may be performed in the client device. Specifically, the selection may be made according to the processing capability of the client device, and restrictions of the use scenario of the user. The present application is not limited in this regard. If all operations are performed in the client device, the client device may further include a processor.

The client device may have a communication module (i.e. a communication unit) and may be connected to a remote server in a communication manner, so as to implement data transmission with the server. The server may include a server on the side of the task scheduling center, and in other implementations may include a server of an intermediate platform, such as a server of a third party server platform having a communication link with the task scheduling center server. The server may include a single computer device, a server cluster formed by a plurality of servers, or a server structure of a distributed device.

Fig. 8 is a schematic block diagram of a system configuration of an electronic device 9600 of an embodiment of the present application. As shown in fig. 8, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 8 is exemplary; other types of structures may also be used in addition to or in place of the structures to implement telecommunications functions or other functions.

In one embodiment, the big data processing method functions may be integrated into the central processor 9100. The central processor 9100 may be configured to perform the following control:

step S101: acquiring data resources of a current big data service cloud in a data exchange platform, wherein the data resources comprise extraction, aggregate data resource definition and a corresponding rule base

Step S102: displaying the data resource to a user through a front-end interaction platform based on a document objectification model for the user to edit

From the above description, the electronic device provided by the embodiment of the application flexibly customizes data processing, edits and adjusts data processing rules in real time, reduces production risk and improves response speed.

In another embodiment, the big data processing apparatus may be configured separately from the central processor 9100, for example, the big data processing apparatus may be configured as a chip connected to the central processor 9100, and the big data processing method function is implemented by control of the central processor.

As shown in fig. 8, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 need not include all of the components shown in fig. 8; in addition, the electronic device 9600 may further include components not shown in fig. 8, and reference may be made to the related art.

As shown in fig. 8, the central processor 9100, sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, which central processor 9100 receives inputs and controls the operation of the various components of the electronic device 9600.

The memory 9140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information about failure may be stored, and a program for executing the information may be stored. And the central processor 9100 can execute the program stored in the memory 9140 to realize information storage or processing, and the like.

The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, but not limited to, an LCD display.

The memory 9140 may be a solid state memory such as Read Only Memory (ROM), random Access Memory (RAM), SIM card, etc. But also a memory which holds information even when powered down, can be selectively erased and provided with further data, an example of which is sometimes referred to as EPROM or the like. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 storing application programs and function programs or a flow for executing operations of the electronic device 9600 by the central processor 9100.

The memory 9140 may also include a data store 9143, the data store 9143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, address book applications, etc.).

The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. A communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, as in the case of conventional mobile communication terminals.

Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, etc., may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and to receive audio input from the microphone 9132 to implement usual telecommunications functions. The audio processor 9130 can include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100 so that sound can be recorded locally through the microphone 9132 and sound stored locally can be played through the speaker 9131.

The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps in the big data processing method in which the execution subject is a server or a client in the above embodiments, the computer-readable storage medium storing a computer program thereon, the computer program implementing all the steps in the big data processing method in which the execution subject is a server or a client in the above embodiments when executed by a processor, for example, the processor implementing the following steps when executing the computer program:

As can be seen from the above description, the computer readable storage medium provided in the embodiments of the present application flexibly customizes data processing, edits and adjusts data processing rules in real time, reduces production risk, and increases response speed.

The embodiments of the present application also provide a computer program product capable of implementing all the steps in the big data processing method in which the execution subject in the above embodiments is a server or a client, where the computer program/instructions implement the steps of the big data processing method when executed by a processor, for example, the computer program/instructions implement the steps of:

As can be seen from the above description, the computer program product provided by the embodiments of the present application flexibly customizes data processing, edits and adjusts data processing rules in real time, reduces production risk and increases response speed.

It will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. The computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The principles and embodiments of the present invention have been described in detail with reference to specific examples, which are provided to facilitate understanding of the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims

1. A method of big data processing, the method comprising:

and receiving rule information edited by the user on the data resource, and updating the rule information into independently deployed bloom filters through a mapping function.

2. The big data processing method according to claim 1, wherein the obtaining the data resource of the current big data service cloud in the data exchange platform includes:

and responding to the update of the data exchange platform, and caching the data resources of the updated big data service cloud.

3. The big data processing method according to claim 1, wherein the presenting the data resource to the user through the front-end interaction platform based on the document objectification model for editing by the user comprises:

4. The big data processing method according to claim 1, wherein the receiving rule information edited by the user for the data resource includes:

5. The big data processing method according to claim 4, further comprising, after receiving an element naming and/or extraction rule editing result of the extraction element by a user:

6. The big data processing method according to claim 4, further comprising, before said receiving rule information edited by the user on the data resource:

7. A big data processing apparatus, the apparatus comprising:

8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the big data processing method of any of claims 1 to 6 when the program is executed by the processor.

9. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the big data processing method of any of claims 1 to 6.

10. A computer program product comprising computer programs/instructions which, when executed by a processor, implement the steps of the big data processing method of any of claims 1 to 6.