CN110019025B

CN110019025B - Stream data processing method and device

Info

Publication number: CN110019025B
Application number: CN201710597929.0A
Authority: CN
Inventors: 徐萌; 段云峰; 魏瑗珍; 黄岩; 尚晶; 刘虹; 徐海勇
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Information Technology Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Information Technology Co Ltd
Priority date: 2017-07-20
Filing date: 2017-07-20
Publication date: 2021-12-21
Anticipated expiration: 2037-07-20
Also published as: CN110019025A

Abstract

The invention discloses a stream data processing method and a stream data processing device, wherein the method comprises the following steps: receiving a first metadata establishing instruction, responding to the first metadata establishing instruction, and generating first metadata according to a preset metadata establishing model; loading first streaming data corresponding to the first metadata according to the first metadata; and receiving a data processing instruction of the first stream data, determining data processing operation corresponding to the data processing instruction, and realizing the data processing operation of the first stream data according to the first metadata.

Description

Stream data processing method and device

Technical Field

The present invention relates to the field of data processing, and in particular, to a method and an apparatus for processing stream data.

Background

The streaming data is a group of data sequences which arrive sequentially, massively, rapidly and continuously, and more fields relate to the streaming data, such as the fields of network monitoring, sensor networks, aerospace, meteorological measurement and control or financial services, so that the operations executed on the streaming data are more and more frequent, and the processing of the streaming data is more and more emphasized.

However, there is no method for performing unified management and processing on stream data in the existing database, which results in inconvenient use of stream data.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a method and an apparatus for processing stream data, which can improve the processing efficiency of the stream data.

In order to achieve the purpose, the technical scheme of the invention is realized as follows:

the embodiment of the invention information and defines the business logic operation corresponding to the flow data;

and generating the first metadata according to the data structure, the source address information and the service logic operation.

In the above method, the loading first streaming data corresponding to the first metadata according to the first metadata includes:

and loading the first streaming data sent by the source address corresponding to the source address information.

In the above method, the receiving a data processing instruction for the first streaming data, determining a data processing operation corresponding to the data processing instruction, and implementing the data processing operation for the first streaming data according to the first metadata includes:

receiving the subscription and distribution instruction sent by a subscription and distribution module, wherein the subscription and distribution instruction carries at least one piece of address information to be sent and preset sending time corresponding to the subscription and distribution module;

determining a current system time according to the first metadata;

and when the current system time reaches the preset sending time, sending the first stream data to the at least one address information to be sent, and storing the at least one address information to be sent.

receiving a service logic instruction for the first-flow data, wherein the service logic instruction carries a trigger condition and the service logic operation;

and when the first metadata meets the trigger condition, realizing the business logic operation on the first flow data.

In the above method, the business logic operation includes a filtering operation, an association operation and an aggregation operation.

receiving a deletion instruction of the first streaming data, wherein the deletion instruction carries a preset storage time and a deletion strategy, and the deletion strategy comprises deleting the first streaming data or deleting any one of the storage data corresponding to the first streaming data before the preset storage time;

there is provided a stream data processing method, the method including:

receiving a first metadata establishing instruction, responding to the first metadata establishing instruction, and generating first metadata according to a preset metadata establishing model;

loading first streaming data corresponding to the first metadata according to the first metadata;

receiving a data processing instruction of the first stream data, determining a data processing operation corresponding to the data processing instruction, and implementing the data processing operation of the first stream data according to the first metadata.

In the above method, the generating the first metadata according to the first metadata establishment instruction includes:

setting a data structure describing stream data, recording a source address of the stream data, based on preset constraint information

Determining the actual storage duration of the first streaming data according to the first metadata;

and when the actual storage duration is equal to the preset storage duration, carrying out corresponding deletion according to the deletion strategy.

In the above method, after the loading the first streaming data corresponding to the first metadata, the method further includes:

determining first information to be detected corresponding to preset information to be detected from the first streaming data;

and when the first information to be detected is not matched with the preset information to be detected, generating alarm information.

An embodiment of the present invention provides a stream data processing apparatus, including:

a receiving unit, configured to receive a first metadata establishment instruction;

the generating unit is used for responding to the first metadata establishing instruction and generating first metadata according to a preset metadata establishing model;

the loading unit is used for loading first streaming data corresponding to the first metadata according to the first metadata;

and the data processing unit is used for receiving a data processing instruction of the first stream data, determining a data processing operation corresponding to the data processing instruction, and realizing the data processing operation of the first stream data according to the first metadata.

In the above-mentioned stream data processing apparatus, the stream data processing apparatus further includes: a setting unit;

the setting unit is used for setting a data structure describing stream data based on preset constraint information, recording source address information of the stream data and defining business logic operation corresponding to the stream data;

the generating unit is specifically configured to generate the first metadata according to the data structure, the source address information, and the service logic operation.

The loading unit is specifically configured to load the first streaming data sent by the source address corresponding to the source address information.

In the above-described streaming data processing apparatus, the data processing instruction includes a subscription distribution instruction, and the streaming data processing apparatus further includes: a determining unit, a transmitting unit and a storing unit;

the receiving unit is further configured to receive the subscription and distribution instruction sent by the subscription and distribution module, where the subscription and distribution instruction carries at least one piece of address information to be sent and preset sending time corresponding to the subscription and distribution module;

the determining unit is used for determining the current system time according to the first metadata;

the sending unit is configured to send the first stream data to the at least one to-be-sent address information when the current system time reaches the preset sending time;

the storage unit is configured to store the at least one to-be-sent address information.

In the above-mentioned stream data processing apparatus, the data processing instruction includes a service logic instruction, and the stream data processing apparatus further includes: a service logic unit;

the receiving unit is further configured to receive a service logic instruction for the first-flow data, where the service logic instruction carries a trigger condition and the service logic operation;

the service logic unit is further configured to implement the service logic operation on the first streaming data when the first metadata meets the trigger condition.

In the above-described stream data processing apparatus, the service logic operation includes a filtering operation, an association operation, and an aggregation operation.

In the above-mentioned stream data processing apparatus, the stream data processing apparatus further includes: a deletion unit;

the receiving unit is further configured to receive a deletion instruction for the first streaming data, where the deletion instruction carries a predetermined storage duration and a deletion policy, and the deletion policy includes deleting the first streaming data or deleting any one of the storage data corresponding to the first streaming data before the predetermined storage duration;

the determining unit is further configured to determine an actual storage duration of the first streaming data according to the first metadata;

and the deleting unit is used for deleting the actual storage duration according to the deleting strategy when the actual storage duration is equal to the preset storage duration.

In the method, the determining unit is configured to determine first information to be detected corresponding to preset information to be detected from the first streaming data;

the generating unit is used for generating alarm information when the first information to be detected is not matched with the preset information to be detected.

An embodiment of the present invention provides a stream data processing apparatus, where the stream data processing apparatus includes at least one stream data processing interface, and the stream data processing apparatus includes: memory, a receiver, a processor, and a computer program stored on the memory and executable on the processor, the receiver relying on the processor to perform operations, the programs when executed by the processor performing the following operations:

the receiver is used for receiving a first metadata establishing instruction and receiving a data processing instruction of the first stream data;

the processor is used for responding to a first metadata establishing instruction and generating first metadata according to a preset metadata establishing model; loading first streaming data corresponding to the first metadata according to the first metadata; and determining data processing operation corresponding to the data processing instruction, and realizing the data processing operation on the first streaming data according to the first metadata.

An embodiment of the present invention provides a computer-readable storage medium, having a computer program stored thereon, for use in a stream data processing apparatus, where the computer program, when executed by a processor, implements any of the stream data processing methods described above.

The method and the device for processing stream data provided by the embodiment of the invention can comprise the following steps: receiving a first metadata establishing instruction, responding to the first metadata establishing instruction, and generating first metadata according to a preset metadata establishing model; loading first streaming data corresponding to the first metadata according to the first metadata; and receiving a data processing instruction of the first stream data, determining data processing operation corresponding to the data processing instruction, and realizing the data processing operation of the first stream data according to the first metadata. By adopting the method, the first metadata is generated by establishing the model according to the preset metadata, the corresponding first-stream data is loaded according to the first metadata, and the data processing operation is carried out on the first-stream data according to the first metadata.

Drawings

Fig. 1 is a first flowchart of a stream data processing method according to an embodiment of the present invention;

FIG. 2 is a 4-level structure of an exemplary default metadata modeling provided by an embodiment of the present invention;

FIG. 3 is a block diagram illustrating an overall architecture of an exemplary big data stream data processing platform according to an embodiment of the present invention;

fig. 4 is a diagram illustrating a connection relationship between an exemplary stream data processing apparatus and other modules according to an embodiment of the present invention;

fig. 5 is a second flowchart of a stream data processing method according to an embodiment of the present invention;

fig. 6 is a first schematic structural diagram of a stream data processing apparatus according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of a stream data processing apparatus according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a stream data processing apparatus according to a third embodiment of the present invention;

fig. 9 is a fourth schematic structural diagram of a stream data processing apparatus according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of a stream data processing apparatus according to an embodiment of the present invention;

fig. 11 is a sixth schematic structural diagram of a stream data processing apparatus according to an embodiment of the present invention.

Detailed Description

The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

In the following description, suffixes such as "module", "component", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no specific meaning in themselves. Thus, "module" and "component" may be used in a mixture.

The terminal may be implemented in various forms. For example, the terminal described in the present invention may include a mobile terminal such as a mobile phone, a smart phone, a notebook computer, a digital broadcast receiver, a Personal Digital Assistant (PDA), a tablet computer (PAD), a Portable Multimedia Player (PMP), a navigation device, etc., and a stationary terminal such as a digital TV, a desktop computer, etc. In the following, it is assumed that the terminal is a mobile terminal. However, it will be understood by those skilled in the art that the configuration according to the embodiment of the present invention can be applied to a fixed type terminal in addition to elements particularly used for moving purposes.

Example one

The present embodiment provides a stream data processing method, as shown in fig. 1, the method including:

s101, receiving a first metadata establishing instruction, responding to the first metadata establishing instruction, and establishing a model according to preset metadata to generate first metadata.

The embodiment of the invention provides a stream data processing method which is suitable for a scene of uniformly processing stream data.

In the embodiment of the invention, the streaming data processing device is preset with a metadata establishment model, and when the streaming data processing device receives a first metadata establishment instruction, the streaming data processing device generates first metadata according to the preset metadata establishment model.

In this embodiment of the present invention, the preset metadata establishment model may include a 4-layer architecture, as shown in fig. 2, where the four-layer architecture is a data structure meta model, a data quality meta model, a transmission storage meta model, and a service logic meta model.

The meta model of the data structure records data structure information of the stream data, which may include field name, field type, length, default value, service description, and the like.

In addition, the format of the structure information may be: a field name; a field type; a length; a default value; and (5) service description.

Illustratively, for signaling example, the data structure meta model is as shown in table one:

watch 1

And the data quality meta-model is used for defining data type conversion, field conversion, merging and the like. Specifically, the definition may be performed by a binary array (field: operation), for example, for the sex field, (sex: 01 female), (sex: 02 male).

And also for defining field constraints, wherein constraints on a field may include primary key constraints, unique constraints, check constraints, and default constraints.

And (3) primary key constraint: the primary key column data is required to be unique and not allowed to be null.

Unique constraint: the column is required to be unique, allowed to be null, but only one null value can occur.

Checking the constraint: a certain list of value range limitations, format limitations, etc. For example, where a column records an age, then it may be constrained that the column value must be in the range of 30-50. For another example, when a certain column of records is a mailbox, the content of the column must have @.

Default constraints are as follows: a default value for a column. For example, a data in the stream data is frequently reused and may be set as a default value.

In addition, the constraints on the stream data can also be recorded using a binary set (field name, constraint).

And the method is also used for defining the check on the data quality of the stream, and the data quality of the stream data can comprise the integrity, consistency, accuracy, timeliness and the like of the data. For example, when the streaming data is a mobile phone number, the mobile phone number is required to be 11 bits, which cannot be less than 11 bits, nor more than 11 bits. As another example, it is required that the filling rate of a certain field of the stream data must reach 70% or more. For another example, the stream data reaches a frequency of 10 stream data per minute, and when the amount of the stream data is checked, the stream data processing apparatus loads the stream data into the preset metadata creation model to generate metadata corresponding to the stream data.

And transmitting the storage meta model for recording an input position, a storage position, an output position and/or a predetermined storage time of the stream data. Storing a transport information layer for locating streaming data, such as recording a source load Internet Protocol (IP) address and port number of a source device providing the streaming data; a protocol type; a storage period of the stream data; destination information of the stream data, i.e., a destination IP address and a port number of a device receiving the stream data, etc.

In addition, the format of the transmission storage meta-model can adopt a table two format.

Watch two

Alternatively, the format of the transport storage meta-model may be: (source load address: IP and port) (protocol type) (memory cycle) (destination address 1: IP and port) (destination address 2: IP and port).

Illustratively, the contents of the transport storage meta-model are: (Source load addresses: 106.2.232.165 and 20) (SDTP) (5 minutes) (destination addresses 1: 106.2.233.166 and 30) (destination addresses 2: 06.2.232.167 and 40).

It should be noted that, when a consumer needs to subscribe to a certain data stream, the IP address and port number of the subscribing terminal and the time for starting subscription can be defined by the subscribing terminal on the storage transmission information layer of the stream data processing device; when the time to start subscription is reached, the streaming data is transmitted to the subscribing terminal corresponding to the IP address and the port number.

And the business logic meta model is used for defining business logic events executed on the stream data and business logic conditions for executing the business logic events. The business logic events include, for example, filtering, association, aggregation, and the like.

And (3) filtering: the processing logic is set for a certain field. For example: in service, when a certain user arrives at a certain position, a reminding short message is triggered. In the stream processing logic, the stream data with protocol type 00 may be filtered, and when the personal location information meets a certain range, a certain action (such as sending a short message) may be triggered if the condition is met, such as: for certain stream data, only data of gender type male is retained, other data is not retained, and the like.

And (3) association: when stream data from different sources, such as weather information and traffic information, is summarized, association operation is performed according to fields of geographic positions, that is, stream data from area a of which the weather information and the traffic information are both from can be selected, for example, a weather information table contains area fields, the traffic information also contains area fields, and the area fields can be subjected to join operation in logic; or, for the streaming data of different protocols loaded into the streaming data processing device, performing association operation, for example, when a mobile phone makes a call, each network element generates logs, and a mobile phone number between the logs can be used as an association field, so that the whole call flow analysis can be performed after association.

Polymerization: certain aggregation is performed before and after the streaming data, and a certain field is aggregated, for example, the number of messages of a certain protocol type is calculated, the real-time kpi type is obtained, and the number of calls per minute, the number of short messages and the like can be counted.

In addition, there is a need for supporting business logic information describing languages and interpretation engines, such as Event Processing Language (EPL), which includes SELECT, FROM, WHERE, GROUP BY, hold, and ORDER BY clauses, while replacing table with Event stream as data source, and can implement join, filtering, and aggregation.

Further, based on the 4-layer architecture, the user may also add elements such as a data structure type or a service event type, specifically, select the elements according to actual situations, and the embodiment of the present invention is not limited specifically.

In the embodiment of the present invention, data structure information corresponding to the data structure meta model and a source address network interconnection Protocol (IP, Internet Protocol), a port number, and the like in the transmission storage meta model need to be filled in manually or in a file manner, and the selection is specifically performed according to an actual situation.

And S102, loading first streaming data corresponding to the first metadata according to the first metadata.

After the stream data processing apparatus generates the first metadata according to the preset metadata building model, the stream data processing apparatus loads the first stream data corresponding to the first metadata.

In the embodiment of the invention, the stream data processing device loads the first stream data sent by the source address corresponding to the source address information.

Further, after the stream data processing device loads the first stream data sent by the source address corresponding to the source address information, the stream data processing device performs data quality inspection on the first stream data, and when the quality inspection on the first stream data is failed, the stream data processing device generates alarm information, wherein the message of the first stream data comprises a message header and a message body. The message header indicates a certain protocol and can be parsed to a message body according to the protocol. The message body in turn includes public information and private information. Illustratively, the signaling in the Mobile communication device is taken as an example, wherein the common information represents general information of each message, such as length, home province, home city, roaming province, roaming city, roaming type, interface information, message identifier, International Mobile Subscriber identity Number (IMSI), and the like. The interface information includes process type, service flow starting time, service flow ending time, position longitude and latitude, flow state and the like.

In the embodiment of the present invention, the first-flow data may be a dynamic data set such as network data or service support data, which is specifically selected according to an actual situation, and the embodiment of the present invention is not specifically limited.

Further, the message of the stream data is in a fixed length format or an indefinite length format, and if the message is in the fixed length format, the length needs to be defined in the first character.

S103, receiving a data processing instruction of the first-stream data, determining a data processing operation corresponding to the data processing instruction, and realizing the data processing operation of the first-stream data according to the first metadata.

After the stream data processing apparatus generates the first metadata corresponding to the first stream data, the stream data processing model receives a data processing instruction for the first stream data, and completes a data processing operation for the first stream data based on the first metadata.

In the embodiment of the present invention, the setting of the data processing instruction may be set for the device in a unified manner, or may also be set for the first-stream data by the user in a single manner, which is specifically selected according to the actual situation, and the embodiment of the present invention is not limited specifically.

In the embodiment of the present invention, the data processing instruction may be a service logic instruction such as a filtering instruction, a correlation instruction, and an aggregation instruction defined in the service logic meta model, or a subscription distribution instruction, a deletion instruction, and the like set by the user, which is specifically selected according to an actual situation, and the embodiment of the present invention is not specifically limited.

In the embodiment of the invention, when the data processing instruction is a service logic instruction, the stream data processing device determines the trigger condition and the specific service logic operation corresponding to the service logic instruction, and when the first metadata corresponding to the first stream data is successfully matched with the trigger condition, the stream data processing device realizes the service logic operation on the first stream data.

Illustratively, when a user needs to boot up, a balance is queried, at this time, a trigger condition of a service logic instruction is boot up, a service logic operation is to query the balance, and a NAS field in the data structure meta-model is 1, which indicates that the boot up is successful, and the user may filter and judge the field, and at this time, the user writes "SELECT from S1where NAS" is "1" through an Event Processing Language (EPL) to send the service logic instruction to the stream data Processing apparatus.

In the embodiment of the invention, when the data processing instruction is a deletion instruction, the stream data processing device determines the preset storage time length and the deletion strategy corresponding to the deletion instruction, determines the actual storage time length of the first stream data according to the first metadata, and executes the deletion strategy on the first stream data when the actual storage time length is equal to the preset storage time length.

In the embodiment of the present invention, the deletion policy includes deleting the first flow data, stopping accessing the first flow data, or deleting the stored data corresponding to the first flow data before a predetermined storage time.

Illustratively, the user sends the streaming data processing apparatus to delete the stored data 3 months ago corresponding to the first streaming data, at which time the streaming data processing apparatus determines that the actual storage time period of the first streaming data is 5 months, and at which time the streaming data processing apparatus determines the stored data 3 months ago from the current time and deletes it.

Illustratively, the service logic information is "day 4 and 21 in 2017, and stream data 1 is deleted", then the service logic condition is day 4 and 21 in 2017, and the service logic event is deletion of stream data 1, that is, the access to stream data 1 is stopped at day 21 in 4 and 21 in 2017.

In the embodiment of the invention, when the data processing instruction is a subscription distribution instruction, the stream data processing device determines at least one piece of address information to be sent and preset sending time corresponding to the subscription distribution instruction, the stream data processing device determines the current system time according to the first metadata, and when the current system time is successfully matched with the preset sending time, the stream data processing device sends the first stream data to the at least one piece of address information to be sent and stores the at least one piece of address information to be sent.

Illustratively, when a user needs to export first stream data in the stream data processing device, the user selects the first stream data and the sending time, and at this time, the stream data processing device records an IP address and a port number corresponding to the user in the transmission storage meta model, and sends the first stream data to the IP address corresponding to the user when the sending time arrives.

In the embodiment of the present invention, the stream data processing apparatus is integrated on a big stream data processing platform, as shown in fig. 3, the big stream data processing platform obtains stream data from the data source 1, the data source 2, …, and the data source n, and distributes the stream data to the consumer 1, the consumer 2, …, and the consumer n, and the big stream data processing platform further includes message pair queue management, subscription distribution management, and message processing.

In the embodiment of the present invention, the stream data processing apparatus implements processing of stream data by interacting with subscription distribution management, message queue management, and message processing, and the connection relationship is as shown in fig. 4, specifically, when a user subscribes to first stream data, the stream data processing apparatus provides destination IP and port number combination information to the subscription distribution module according to the transmission storage meta-model, and the stream data processing apparatus sends the first stream data to the destination IP through subscription distribution management; the stream data processing device provides storage information of first stream data to message queue management according to the transmission storage meta model, wherein the storage information comprises a storage period and a storage mode; the stream data processing means provides quality metadata, a data structure and service logic to the message processing according to the service logic meta-model, the data quality meta-model and the data structure meta-model.

In the embodiment of the invention, when a user adds, deletes, changes and checks the metadata information, the metadata maintenance module in the stream data processing device can detect whether the service logic of the metadata information conflicts or not.

Illustratively, data operation metadata is predefined to be a male field 01 and a female field 02; when the revision is changed to 01 for female, and when the revision is changed to 02 for male, conflict detection can prompt that the revision is inconsistent with the previous logic; or when the constraint is defined, a field is already defined as a unique constraint, and the definition as a key value constraint can not be repeated.

It can be understood that, in the embodiment of the present invention, the stream data processing apparatus generates the first metadata according to the preset metadata establishment model, loads the corresponding first stream data according to the first metadata, and performs the data processing operation on the first stream data according to the first metadata, at this time, the stream data processing apparatus can perform the data processing operation on a plurality of pieces of stream data at the same time, thereby implementing uniform processing on the stream data and improving the efficiency of processing the stream data.

Example two

An embodiment of the present invention provides a stream data processing method, as shown in fig. 5, where the method may include:

s201, the stream data processing device receives a first metadata establishing instruction, responds to the first metadata establishing instruction, sets a data structure describing stream data, records source address information of the stream data and defines business logic operation corresponding to the stream data based on preset constraint information.

The stream data processing method provided by the embodiment of the invention is suitable for a scene of uniformly processing stream data.

In the embodiment of the invention, the stream data processing device is preset with a metadata establishment model, and when the stream data processing device receives a first metadata establishment instruction, the stream data processing device sets a data structure describing stream data based on preset constraint information in the preset metadata establishment model and records source address information of the stream data.

watch 1

Watch two

And (3) association: when stream data from different sources, such as weather information and traffic information, are summarized, association operation is performed according to fields of geographic positions, that is, stream data from an area A of which the weather information and the traffic information are both from can be selected, for example, a weather information table contains a region field, a traffic information page contains a region field, and join operation can be performed on the region field in logic; or, for the streaming data of different protocols loaded into the streaming data processing device, performing association operation, for example, when a mobile phone makes a call, each network element generates logs, and a mobile phone number between the logs can be used as an association field, so that the whole call flow analysis can be performed after association.

In the embodiment of the invention, when the data structure information is inconsistent with the set constraint information, the stream data processing device sends a prompt to inform a user that the constraint information is inconsistent to be stored and the user is required to change.

S202, the stream data processing device generates first metadata according to the data structure, the source address information and the service logic operation.

After the stream data processing apparatus sets the data structure and the source address information, the stream data processing apparatus generates first metadata based on the data structure and the source address information.

In the embodiment of the invention, the stream data processing device generates the first metadata according to the data structure and the source address information, so that the stream data processing device acquires the corresponding first stream data according to the source address information and manages the first stream data.

In the embodiment of the present invention, the first metadata may be stored in a file or a database. Specifically, spacers or line breaks may be used to distinguish the information of each layer, and spacers or line breaks may also be used to distinguish the information of each layer from each other.

S203, the stream data processing device loads the first stream data sent by the source address corresponding to the source address information.

After the stream data processing apparatus generates the first metadata, the stream data processing apparatus acquires the corresponding first stream data according to the first metadata.

In the embodiment of the invention, the stream data processing device determines the source address from the transmission storage meta-model and loads the first stream data transmitted from the source address.

S204, the stream data processing device determines first information to be detected corresponding to preset information to be detected from the first stream data.

After the stream data processing device acquires the first stream data, the stream data processing device determines first information to be detected corresponding to preset information to be detected from the first stream data to check the quality of the first stream data.

In the embodiment of the invention, the preset information to be detected and the corresponding preset quality standard are arranged in the stream data processing device, and after the stream data processing device acquires the first stream data, the first information to be detected corresponding to the preset information to be detected is determined from the first stream data.

In the embodiment of the present invention, a user may manually set the preset information to be detected, where the preset information to be detected may be field filling rate or frequency, and the preset information to be detected is specifically selected according to an actual situation, and the embodiment of the present invention is not specifically limited.

S205, when the first information to be detected is not matched with the preset information to be detected, the stream data processing device generates alarm information.

And the stream data processing device matches the first information to be detected with the preset information to be detected, and loads the first stream data when the matching fails.

In the embodiment of the invention, the stream data processing device matches the first information to be detected with the preset information to be detected to judge whether the first information to be detected meets the preset quality standard corresponding to the preset information to be detected.

Illustratively, the preset information to be detected set by the stream data processing apparatus is a filling rate of the field a, the preset quality standard is 70%, and when the filling rate of the field a in the first stream data is 60%, the stream data processing apparatus generates the alarm information.

In the embodiment of the present invention, the message of the first stream data includes a message header and a message body. The message header indicates a certain protocol and can be parsed to a message body according to the protocol. The message body in turn includes public information and private information. Illustratively, the signaling in the Mobile communication device is taken as an example, wherein the common information represents general information of each message, such as length, home province, home city, roaming province, roaming city, roaming type, interface information, message identifier, International Mobile Subscriber identity Number (IMSI), and the like. The interface information includes process type, service flow starting time, service flow ending time, position longitude and latitude, flow state and the like.

S206, the stream data processing device receives a data processing instruction of the first stream data, determines a data processing operation corresponding to the data processing instruction, and realizes the data processing operation of the first stream data according to the first metadata.

After the stream data processing apparatus obtains the first metadata corresponding to the first stream data, the stream data processing apparatus receives a data processing instruction for the first stream data, and completes a data processing operation for the first stream data according to the first metadata.

In the embodiment of the invention, a user sends a data processing instruction to the stream data processing device, and after receiving the data processing instruction, the stream data processing device realizes data processing operation on the first stream data according to the first metadata.

In the embodiment of the present invention, the service logic operation includes operations of filtering, associating, aggregating, and the like, which can embody the service logic of the stream data, and the selection is specifically performed according to an actual situation, and the embodiment of the present invention is not specifically limited.

In the embodiment of the invention, when the data processing instruction is a subscription distribution instruction, the stream data processing device determines at least one piece of address information to be sent and preset sending time corresponding to the subscription distribution instruction, the stream data processing device determines the current system time according to the first metadata, and when the current system time reaches the preset sending time, the stream data processing device sends the first stream data to the at least one piece of address information to be sent and stores the at least one piece of address information to be sent.

In the embodiment of the present invention, the stream data processing apparatus implements processing of stream data by interacting with subscription distribution management, message queue management, and message processing, and the connection relationship is as shown in fig. 4, specifically, when a user subscribes to first stream data, the stream data processing apparatus provides destination IP and port number combination information to the subscription distribution module according to the transmission storage meta-model, and the stream data processing apparatus sends the first stream data to the destination IP through subscription distribution processing; the stream data processing device provides storage information of first stream data to the message queue processing according to the transmission storage meta model, wherein the storage information comprises a storage period and a storage mode; the stream data processing means provides quality metadata, a data structure and service logic to the message processing according to the service logic meta-model, the data quality meta-model and the data structure meta-model.

Step S204-step S205 and step S206 are two parallel steps after step S203, which are specifically selected according to actual situations, and the embodiment of the present invention is not limited specifically.

EXAMPLE III

An embodiment of the present invention provides a stream data processing apparatus 1, as shown in fig. 6, the stream data processing apparatus 1 including:

a receiving unit 10, configured to receive a first metadata establishment instruction.

And the generating unit 11 is configured to respond to the first metadata establishing instruction and generate first metadata according to a preset metadata establishing model.

And a loading unit 12, configured to load first streaming data corresponding to the first metadata according to the first metadata.

The data processing unit 13 is configured to receive a data processing instruction for the first streaming data, determine a data processing operation corresponding to the data processing instruction, and implement the data processing operation for the first streaming data according to the first metadata.

Optionally, based on fig. 6, as shown in fig. 7, the stream data processing apparatus 1 further includes: a unit 14 is provided.

The setting unit 14 is configured to set a data structure describing stream data, record source address information of the stream data, and define a service logic operation corresponding to the stream data based on preset constraint information.

The generating unit 11 is specifically configured to generate the first metadata according to the data structure, the source address information, and the service logic operation.

Optionally, the loading unit 12 is specifically configured to load the first streaming data sent by the source address corresponding to the source address information.

Optionally, based on fig. 7, as shown in fig. 8, the data processing instruction includes a subscription distribution instruction, and the streaming data processing apparatus 1 further includes: a determination unit 15, a transmission unit 16 and a storage unit 17.

The receiving unit 10 is further configured to receive the subscription and distribution instruction sent by the subscription and distribution module, where the subscription and distribution instruction carries at least one piece of address information to be sent and preset sending time corresponding to the subscription and distribution module.

The determining unit 15 is further configured to determine a current system time according to the first metadata.

The sending unit 16 is configured to send the first stream data to the at least one address information to be sent when the current system time reaches the preset sending time.

The storage unit 17 is configured to store the at least one address information to be sent.

Optionally, based on fig. 8, as shown in fig. 9, the data processing instruction includes a service logic instruction, and the stream data processing apparatus 1 further includes: a service logic unit 18.

The receiving unit 10 is further configured to receive a service logic instruction for the first-flow data, where the service logic instruction carries a trigger condition and the service logic operation.

The service logic unit 18 is further configured to implement the service logic operation on the first streaming data when the first metadata meets the trigger condition.

Optionally, the business logic operation includes a filtering operation, an association operation, and an aggregation operation.

Optionally, based on fig. 8, as shown in fig. 10, the stream data processing apparatus 1 further includes: and a deletion unit 19.

The receiving unit 10 is further configured to receive a deletion instruction for the first streaming data, where the deletion instruction carries a predetermined storage duration and a deletion policy, and the deletion policy includes deleting the first streaming data or deleting any one of the storage data corresponding to the first streaming data before the predetermined storage duration.

The determining unit 15 is further configured to determine an actual storage duration of the first streaming data according to the first metadata.

And the deleting unit 19 is configured to, when the actual storage duration is equal to the preset storage duration, perform corresponding deletion according to the deletion policy.

Optionally, the determining unit 15 is configured to determine, from the first streaming data, first to-be-detected information corresponding to preset to-be-detected information.

The generating unit 11 is configured to generate warning information when the first information to be detected is not matched with the preset information to be detected.

In practical applications, based on the same inventive concept of the first and second embodiments, as shown in fig. 11, the stream data processing apparatus 1 may include: a communication interface 110, a transmitter 111, a receiver 112, a memory 113, a processor 114, and a bus 115;

the generating Unit 11, the loading Unit 12, the data Processing Unit 13, the setting Unit 14, the determining Unit 15, the storing Unit 17, the service Logic Unit 18, and the deleting Unit 19 may be implemented by a Processor 114 located on the stream data Processing apparatus 1, and the Processor 114 may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understood that, for different apparatuses, the electronic device for implementing the above-mentioned processor function may be other, embodiments of the present invention are not limited in particular, the sending unit 16 may be implemented by the sender 111, the receiving unit 10 may be implemented by the receiver 112, the stream data processing apparatus 1 may further include a memory 113, the memory 113 and the sender 111 may be connected to the processor 114, wherein the memory 113 is configured to store executable program codes, the program codes include computer operation instructions, and the memory 113 may include a high-speed RAM memory and may further include a non-volatile memory, for example, at least one disk memory.

114 may be implemented on a cluster basis using distributed technology to maintain data distribution and processing performance.

The bus 115 is used for connecting the communication interface 110, the transmitter 111, the receiver 112, the processor 114 and the memory 113 and the intercommunication among these devices;

the communication interface 110 is configured to perform data transmission with an external network element;

the memory 111 is used for storing instructions and data;

the processor 114 executes the instructions to: responding to a first metadata establishing instruction, and establishing a model according to preset metadata to generate first metadata; loading first streaming data corresponding to the first metadata according to the first metadata; and determining data processing operation corresponding to the data processing instruction, and realizing the data processing operation on the first streaming data according to the first metadata.

The receiver 112 is configured to receive a first metadata creation instruction and receive a data processing instruction for the first stream data.

The transmitter 111 is configured to send the first stream data to the at least one address information to be sent when the current system time reaches the preset sending time.

In practical applications, the Memory 113 may be a volatile Memory (volatile Memory), such as a Random-Access Memory (RAM); or a non-volatile Memory (non-volatile Memory), such as a Read-Only Memory (ROM), a flash Memory (flash Memory), a Hard Disk (HDD), or a Solid-State Drive (SSD); or a combination of the above types of memories and provides instructions and data to the processor 114.

In addition, each functional module in this embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware or a form of a software functional module.

Based on the understanding that the technical solution of the present embodiment essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, and include several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the method of the present embodiment. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

The embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, for use in the stream data processing apparatus 1, the computer program implementing the method according to the first and second embodiments when executed by the processor 114.

Specifically, the program instructions corresponding to a stream data processing method in the present embodiment may be stored on a storage medium such as an optical disc, a hard disk, a usb disk, or the like, and when the computer program instructions corresponding to a stream data processing method in the storage medium are read or executed by an electronic device, the method includes the steps of:

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.

Claims

1. A method of stream data processing, the method comprising:

receiving a first metadata establishing instruction, responding to the first metadata establishing instruction, and generating first metadata according to a preset metadata establishing model; the preset metadata establishing model comprises a data structure meta model, a data quality meta model, a transmission storage meta model and a service logic meta model; the data structure meta-model is used for recording data structure information of the stream data; the data quality meta-model is used for defining data type conversion, field conversion, merging, field constraint and checking the quality of the streaming data; the transmission storage meta-model is used for recording an input position, a storage position, an output position and/or preset storage duration of the stream data; the service logic meta-model is used for defining service logic events executed on the stream data and service logic conditions for executing the service logic events;

receiving a service logic instruction for the first-flow data, wherein the service logic instruction carries a trigger condition and service logic operation;

2. The method according to claim 1, wherein the generating the first metadata according to the preset metadata building model comprises:

setting a data structure describing the streaming data based on preset constraint information, recording source address information of the streaming data, and defining the business logic operation corresponding to the streaming data;

3. The method according to claim 2, wherein the loading the first streaming data corresponding to the first metadata according to the first metadata comprises:

4. The method of claim 1, further comprising:

determining a current system time according to the first metadata;

5. The method of claim 1, wherein the business logic operations comprise filtering operations, association operations, and aggregation operations.

6. The method of claim 1, further comprising:

receiving a deletion instruction of the first flow data, wherein the deletion instruction carries a preset storage time length and a deletion strategy, and the deletion strategy comprises any one of deleting the first flow data, stopping accessing the first flow data, or deleting the storage data corresponding to the first flow data before the preset storage time length;

7. The method according to claim 1, wherein after loading the first streaming data corresponding to the first metadata, the method further comprises:

8. A stream data processing apparatus, characterized by comprising:

the generating unit is used for responding to the first metadata establishing instruction and generating first metadata according to a preset metadata establishing model; the preset metadata establishing model comprises a data structure meta model, a data quality meta model, a transmission storage meta model and a service logic meta model; the data structure meta-model is used for recording data structure information of the stream data; the data quality meta-model is used for defining data type conversion, field conversion, merging, field constraint and checking the quality of the streaming data; the transmission storage meta-model is used for recording an input position, a storage position, an output position and/or preset storage duration of the stream data; the service logic meta-model is used for defining service logic events executed on the stream data and service logic conditions for executing the service logic events;

the receiving unit is further configured to receive a service logic instruction for the first-flow data, where the service logic instruction carries a trigger condition and a service logic operation;

and the service logic unit is used for realizing the service logic operation on the first flow data when the first metadata meets the trigger condition.

9. The stream-data processing apparatus according to claim 8, characterized by further comprising: a setting unit;

the setting unit is used for setting a data structure describing the stream data, recording source address information of the stream data and defining the business logic operation corresponding to the stream data based on preset constraint information;

10. The stream data processing apparatus according to claim 9,

11. The streaming data processing apparatus according to claim 8, wherein the data processing instruction includes a subscription distribution instruction, the streaming data processing apparatus further comprising: a determining unit, a transmitting unit and a storing unit;

12. The streaming data processing apparatus of claim 8, wherein the business logic operations comprise a filtering operation, an association operation, and an aggregation operation.

13. The stream-data processing apparatus according to claim 11, characterized by further comprising: a deletion unit;

the receiving unit is further configured to receive a deletion instruction for the first streaming data, where the deletion instruction carries a preset storage duration and a deletion policy, and the deletion policy includes deleting the first streaming data or deleting any one of the storage data corresponding to the first streaming data before the preset storage duration;

14. The stream data processing apparatus according to claim 11,

the determining unit is used for determining first information to be detected corresponding to preset information to be detected from the first streaming data;

15. A stream data processing apparatus including at least one stream data processing interface, characterized by comprising: memory, a receiver, a processor, and a computer program stored on the memory and executable on the processor, the receiver relying on the processor to perform operations, the programs when executed by the processor performing the following operations:

the receiver is used for receiving a first metadata establishing instruction and receiving a data processing instruction of first stream data;

the processor is used for responding to the first metadata establishing instruction and generating first metadata according to a preset metadata establishing model; loading the first streaming data corresponding to the first metadata according to the first metadata; and determining data processing operation corresponding to the data processing instruction, and realizing the data processing operation on the first streaming data according to the first metadata.

16. A computer-readable storage medium, on which a computer program is stored, for application on a stream data processing apparatus, characterized in that the computer program, when being executed by a processor, implements the method of any one of claims 1 to 7.