CN111104425A

CN111104425A - Data processing method and device

Info

Publication number: CN111104425A
Application number: CN201911104858.1A
Authority: CN
Inventors: 李阳; 王兴国; 邱孝兵; 高雪峰
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2019-11-13
Filing date: 2019-11-13
Publication date: 2020-05-05

Abstract

The embodiment of the invention discloses a data processing method and a device, wherein the data processing method comprises the following steps: acquiring a query request for the streaming data in the first service system in the second service system, wherein the query request comprises a first query condition; converting the first query condition into a second query condition corresponding to a storage structure according to the query request; generating the flow query data according to the second query condition; converting the pipelining query data into first pipelining data of a general structure; and outputting the first running water data at the second service system. The embodiment of the invention can solve the problems of universality and expansibility among different service systems, simultaneously reduce the operation cost and improve the query efficiency by acquiring the query condition of the streaming data in the first service system in the second service system and enabling the second service system to query the streaming data in the first service system through the format converter.

Description

Data processing method and device

Technical Field

The present invention relates to the field of communications technologies, and in particular, to a data processing method and apparatus.

Background

Under the rapid development of the internet, the types of services are more and more, and when a certain service is in operation, corresponding pipeline data is generated and needs to be stored.

The existing flow technical scheme is designed and realized based on a memory, a flow format is designed for an object, and then the increased checking operation on a database is added, however, under the condition of the rapid development of the current network, the types of services are not single, so that the design is respectively carried out for each single service, the cost is greatly increased, and the operation efficiency is low.

Disclosure of Invention

The application provides a data processing method and a data processing device, which can solve the problems of universality and expansibility among different service systems, reduce the operation cost and improve the query efficiency by acquiring the query conditions of the streaming data in the first service system in the second service system and enabling the second service system to query the streaming data in the first service system through a format converter.

In a first aspect, the present application provides a data processing method, including:

acquiring a query request for the streaming data in the first service system in the second service system, wherein the query request comprises a first query condition;

converting the first query condition into a second query condition corresponding to a storage structure according to the query request;

generating the flow query data according to the second query condition;

converting the pipelining query data into first pipelining data of a general structure;

and outputting the first running water data at the second service system.

In some embodiments of the present application, the method further comprises:

acquiring a first storage request for storing first streaming data to a memory corresponding to a second service system, wherein the first storage request comprises a first storage condition;

converting the first flow data into corresponding flow data in a first storage format;

and storing the pipeline data in the first storage format into a memory corresponding to the first service system according to the first storage condition.

In some embodiments of the present application, the method further comprises:

acquiring a second storage request for storing the first streaming data to a storage corresponding to a second service system, wherein the second storage request comprises a first storage condition;

converting the first flow data into corresponding flow data in a second storage format;

and storing the pipeline data in the second storage format into a memory corresponding to the second service system according to the second storage condition.

In some embodiments of the present application, the storing the pipeline data in the second storage format into a memory corresponding to the second service system according to the second storage condition includes:

counting the target data volume of the first flowing water data;

and when the target data volume reaches a preset data volume, asynchronously storing the first running water data according to the second storage condition.

In some embodiments of the present application, said asynchronously storing the first streaming data according to the second storage condition includes:

sending the first flow data to a preset message queue;

and when the message queue is in turn to process the first streaming data, reading the first streaming data from the message queue according to the second storage condition and storing the first streaming data into a corresponding storage of the second service system.

In some embodiments of the present application, after obtaining a second storage condition for storing the first streaming data in a memory corresponding to the second service system, the method further includes:

repeatedly comparing the pipeline type of the pipeline data with the pipeline type of the stored pipeline data;

when the comparison result is no repetition, defining the flow type of the flow data as enumeration information;

and converting the enumeration information into a query field, and outputting the query field to a terminal for display.

In some embodiments of the present application, the method further comprises:

while generating the pipeline data;

generating a new block according to the pipeline data;

adding the new tile to the chain of tiles.

In a second aspect, the present application provides a data processing apparatus, the apparatus comprising:

the system comprises a first acquisition unit, a second acquisition unit and a first processing unit, wherein the first acquisition unit is used for acquiring an inquiry request for the streaming data in the first service system in the second service system, and the inquiry request comprises a first inquiry condition;

the first conversion unit is used for converting the first query condition into a second query condition corresponding to the storage structure according to the query request;

the first generating unit is used for generating the pipelining query data according to the second query condition;

the second conversion unit is used for converting the pipelining query data into first pipelining data with a general structure;

and the first output unit is used for outputting the first running water data in the second service system.

In some embodiments of the present application, the apparatus further comprises:

the second acquisition unit is used for acquiring a first storage request for storing the first running water data into a memory corresponding to a second service system;

the third conversion unit is used for converting the first flow data into corresponding flow data in a first storage format;

and the first storage unit is used for storing the pipeline data in the first storage format into a memory corresponding to the first service system according to the first storage condition.

the third acquisition unit is used for acquiring a second storage request for storing the first streaming data to a storage corresponding to the second service system;

the fourth conversion unit is used for converting the first flow data into corresponding flow data in a second storage format;

and the second storage unit is used for storing the pipeline data in the second storage format into a memory corresponding to the second service system according to the second storage condition.

the statistical unit is used for counting the target data volume of the first flowing water data;

and the third storage unit is used for asynchronously storing the first streaming data according to the second storage condition when the target data volume reaches a preset data volume.

In a third aspect, the present application provides a server comprising a memory and a processor, the memory having stored therein a computer program that, when executed by the processor, causes the processor to perform the steps of:

generating the flow query data according to the second query condition;

and outputting the first running water data at the second service system.

In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program to be loaded by a processor for performing the steps of the data processing method of any one of the first aspect.

In the embodiment of the application, a query request for the flowing water data in the first service system is acquired in the second service system, wherein the query request comprises a first query condition; converting the first query condition into a second query condition corresponding to a storage structure according to the query request; generating the flow query data according to the second query condition; converting the pipelining query data into first pipelining data of a general structure; and outputting the first running water data at the second service system. The embodiment of the invention can solve the problems of universality and expansibility among different service systems, simultaneously reduce the operation cost and improve the query efficiency by acquiring the query condition of the streaming data in the first service system in the second service system and enabling the second service system to query the streaming data in the first service system through the format converter.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic view of a scenario of a service system provided in an embodiment of the present invention;

FIG. 2 is an alternative structure diagram of the distributed system applied to the blockchain system according to the embodiment of the present invention;

FIG. 3 is an alternative block structure according to an embodiment of the present invention;

FIG. 4 is a flow chart of an embodiment of a data processing method according to an embodiment of the present invention;

FIG. 5 is a flow chart of another embodiment of a data processing method according to an embodiment of the present invention;

FIG. 6 is a flow chart of another embodiment of a data processing method according to an embodiment of the present invention;

FIG. 7 is a flow chart of another embodiment of a data processing method according to an embodiment of the present invention;

FIG. 8 is a flow chart of another embodiment of a data processing method according to an embodiment of the present invention;

FIG. 9 is a flow chart of a data query process provided by an embodiment of the invention;

FIG. 10 is a diagram of a second service system architecture provided by an embodiment of the present invention;

FIG. 11 is a block diagram of a data processing apparatus according to an embodiment of the present invention;

fig. 12 is a schematic structural diagram of a server provided in the embodiment of the present invention;

fig. 13 is a schematic structural diagram of a terminal according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

In the description that follows, specific embodiments of the present invention are described with reference to steps and symbols executed by one or more computers, unless otherwise indicated. Accordingly, these steps and operations will be referred to, several times, as being performed by a computer, the computer performing operations involving a processing unit of the computer in electronic signals representing data in a structured form. This operation transforms the data or maintains it at locations in the computer's memory system, which may be reconfigured or otherwise altered in a manner well known to those skilled in the art. The data maintains a data structure that is a physical location of the memory that has particular characteristics defined by the data format. However, while the principles of the invention have been described in the foregoing context, it is not intended to be limited to the specific form set forth herein, but rather to hardware implementations that will enable those of ordinary skill in the art to practice the principles of the invention.

The term "module" or "unit" as used herein may be considered a software object executing on the computing system. The various components, modules, engines, and services described herein may be viewed as objects implemented on the computing system. The apparatus and method described herein are preferably implemented in software, but may also be implemented in hardware, and are within the scope of the present invention.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.

To facilitate an understanding of the embodiments of the present invention, a few basic concepts that will be introduced in the description of the embodiments of the present invention will be introduced first:

flowing water: a particular journal record, such as a journal of bank transaction bills.

The flow data: the term of software engineering is used to record a data structure of a pipeline to build a pipeline structure.

The stream object: software engineering terminology, which means a concrete data of a pipeline structure, is used for storing a concrete piece of pipeline data, wherein an object refers to a main object of a pipeline record, such as bank transfer pipeline, and the object refers to deposit of a user. For example, the object further includes: users, articles, comments, etc., a particular streamlet is referred to as a user streamlet (including user registration, user attention, user approval, user reporting, etc.), an article streamlet, a comment streamlet, etc.

The type of running water: the structure for managing the pipeline data of various objects realizes the storage of the pipeline of specific objects into a specific memory.

A service system: the system is used for recording the operation records of the users, wherein the business systems can be various business systems, such as a banking business system, a logistics business system and the like, the first business system and the second business system can be different business systems, such as a banking business system, a logistics business system and the same business system, and business systems of different versions, such as a banking business system 1.0 and a banking business system 2.0 and the like.

A memory: the system for storing the pipeline objects is an abstract object, and the specific memory comprises mysql memory, mongodb memory, elastic search memory and the like.

Storing the route: the routing of network engineering refers to a network-wide process for determining an end-to-end path when a packet is from a source to a destination, the routing in the application refers to a path selection relation in which flow data is stored in a specific memory, and logic for recording the storage relation is called routing. Because the storage space of a table is limited, when the stored data is increased to a certain condition, the problems of slow query, even incapability of writing and the like can be caused, so that in the process of storing and reading quality inspection, a layer of storage route management module is added, the data can be aggregated according to a certain condition, for example, the data of the same user is aggregated into the same table for convenient query, but for most of the running data, the table needs to be divided according to time, the management is called as routing, and the table is positioned according to the query condition, or the stored table is positioned according to the running data.

Artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.

The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Computer Vision technology (CV) Computer Vision is a science for researching how to make a machine "see", and further refers to that a camera and a Computer are used to replace human eyes to perform machine Vision such as identification, tracking and measurement on a target, and further image processing is performed, so that the Computer processing becomes an image more suitable for human eyes to observe or transmitted to an instrument to detect. As a scientific discipline, computer vision research-related theories and techniques attempt to build artificial intelligence systems that can capture information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D technologies, virtual reality, augmented reality, synchronous positioning, map construction, and other technologies, and also include common biometric technologies such as face recognition and fingerprint recognition.

The scheme provided by the embodiment of the application can be a data processing method related to artificial intelligence, namely, the embodiment of the application provides a data processing method based on artificial intelligence, and the method can comprise the following steps: acquiring a first query condition of the running water data in the first service system in the second service system; determining a corresponding memory according to the first query condition; converting the first query condition into a second query condition corresponding to the storage structure; generating the flow query data according to the second query condition; converting the pipelining query data into first pipelining data of a general structure; and outputting the first running water data at the second service system.

The embodiment of the invention provides a data processing method and device.

Referring to fig. 1, fig. 1 is a schematic view of a scenario of a service system according to an embodiment of the present invention, where the service system may include a server and a terminal, where the terminal and the server are connected through a network, and a data processing device is integrated in a server, where the server may be a work platform server (i.e., a server loaded with a work platform), for example, a first service system in fig. 1 includes a first server 101 and a first terminal 102, and a second service system includes a second server 103 and a second terminal 104, where the first server may be accessed by the first terminal, the second server may be accessed by the first terminal, and the first server may be accessed by the second terminal, and the second server may also be accessed by the second terminal, where the first terminal may be one terminal or more than one terminal. The server in the embodiment of the invention is mainly used for acquiring the query request of the flowing water data in the first service system in the second service system, wherein the query request comprises a first query condition; converting the first query condition into a second query condition corresponding to the storage structure according to the query request; generating the flow query data according to the second query condition; converting the pipelining query data into first pipelining data of a general structure; and outputting the first running water data at the second service system.

In this embodiment of the present invention, the first server 101/the second server 103 may be independent servers, or may be a server network or a server cluster composed of servers, for example, the servers described in this embodiment of the present invention include, but are not limited to, a computer, a network host, a single network server, multiple network server sets, or a cloud server composed of multiple servers. Among them, the cloud server is constituted by a large number of computers or web servers based on cloud computing (CloudComputing). In the embodiment of the present invention, the server and the terminal may implement communication through any communication manner, including but not limited to mobile communication based on the third Generation Partnership Project (3 GPP), Long Term Evolution (LTE), Worldwide Interoperability for microwave access (WiMAX), or computer network communication based on the TCP/IP Protocol Suite (TCP/IP), User Datagram Protocol (UDP) Protocol, and the like.

It is to be understood that the first terminal 102/the second terminal 104 used in the embodiments of the present invention may be understood as terminal devices, which include both receiving and transmitting hardware, i.e., devices having receiving and transmitting hardware capable of performing two-way communication over a two-way communication link. Such a terminal device may include: a cellular or other communication device having a single line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific terminal may be a desktop terminal or a mobile terminal, and the terminal may be one of a mobile phone, a tablet computer, a notebook computer, and the like.

Those skilled in the art will understand that the application environment shown in fig. 1 is only one application scenario related to the present application, and does not constitute a limitation on the application scenario of the present application, and that other application environments may further include more or less servers than those shown in fig. 1, or a server network connection relationship, for example, only 2 servers and 2 terminals are shown in fig. 1, and it is understood that the service system may further include one or more other servers, or/and one or more terminals connected to a server network, and is not limited herein.

In some embodiments of the present invention, the business system may be an enterprise office system, such as an enterprise banking account system, and taking the server 101 as an example, the business system may further specifically include an enterprise banking account server, an enterprise office configuration management server, and a Web management server, where an enterprise user or a developer may access the Web management server by using a Web browsing terminal to configure field configuration information on the enterprise office platform configuration management server, and set and store enterprise user information of enterprise employees of the enterprise office platform in an enterprise office platform contact server.

In addition, as shown in fig. 1, the business system may further include a memory, where the memory includes a first memory 105 and a second memory 106, where the first memory 105 corresponds to the first business system, and the second memory corresponds to the second business system, and the memory is used to store data, such as a business pipeline database, where the business pipeline database stores pipeline object data, and correspondingly, the business pipeline database may also be divided into multiple types of data, such as an application database, a file database, or a picture database.

It should be noted that the scenario diagram of the service system shown in fig. 1 is only an example, and the service system and the scenario described in the embodiment of the present invention are for more clearly illustrating the technical solution of the embodiment of the present invention, and do not form a limitation on the technical solution provided in the embodiment of the present invention.

The business system related to the embodiment of the invention can be a distributed system formed by connecting a plurality of nodes (any type of computing equipment in an access network, such as electronic equipment, a server and the like) in a network communication mode.

Taking a distributed system as an example of a blockchain system, referring to fig. 2, fig. 2 is an optional structural schematic diagram of the application of the distributed system to the blockchain system provided by the embodiment of the present invention, and the system is formed by a plurality of nodes (computing devices in any form in an access network, such as servers and user terminals) and terminals, a Peer-to-Peer (P2P, Peer to Peer) network is formed between the nodes, and the P2P Protocol is an application layer Protocol operating on a Transmission Control Protocol (TCP). In a distributed system, any machine, such as a server or a terminal, can join to become a node, and the node comprises a hardware layer, a middle layer, an operating system layer and an application layer. In the embodiment of the invention, the electronic device and the server are respectively one node in a block chain system.

Referring to the functions of each node in the blockchain system shown in fig. 2, the functions involved include:

1) routing, a basic function that a node has, is used to support communication between nodes.

Besides the routing function, the node may also have the following functions:

2) the application is used for being deployed in a block chain, realizing specific services according to actual service requirements, recording data related to the realization functions to form recording data, carrying a digital signature in the recording data to represent a source of task data, and sending the recording data to other nodes in the block chain system, so that the other nodes add the recording data to a temporary block when the source and integrity of the recording data are verified successfully.

For example, the services implemented by the application include:

2.1) wallet, for providing the function of transaction of electronic money, including initiating transaction (i.e. sending the transaction record of current transaction to other nodes in the blockchain system, after the other nodes are successfully verified, storing the record data of transaction in the temporary blocks of the blockchain as the response of confirming the transaction is valid; of course, the wallet also supports the querying of the remaining electronic money in the electronic money address;

and 2.2) sharing the account book, wherein the shared account book is used for providing functions of operations such as storage, query and modification of account data, record data of the operations on the account data are sent to other nodes in the block chain system, and after the other nodes verify the validity, the record data are stored in a temporary block as a response for acknowledging that the account data are valid, and confirmation can be sent to the node initiating the operations.

2.3) Intelligent contracts, computerized agreements, which can enforce the terms of a contract, implemented by codes deployed on a shared ledger for execution when certain conditions are met, for completing automated transactions according to actual business requirement codes, such as querying the logistics status of goods purchased by a buyer, transferring the buyer's electronic money to the merchant's address after the buyer signs for the goods; of course, smart contracts are not limited to executing contracts for trading, but may also execute contracts that process received information.

3) And the Block chain comprises a series of blocks (blocks) which are mutually connected according to the generated chronological order, new blocks cannot be removed once being added into the Block chain, and recorded data submitted by nodes in the Block chain system are recorded in the blocks.

Referring to fig. 3, fig. 3 is an optional schematic diagram of a Block Structure (Block Structure) according to an embodiment of the present invention, where each Block includes a hash value of a transaction record stored in the Block (hash value of the Block) and a hash value of a previous Block, and the blocks are connected by the hash values to form a Block chain. The block may include information such as a time stamp at the time of block generation. A block chain (Blockchain), which is essentially a decentralized database, is a string of data blocks associated by using cryptography, and each data block contains related information for verifying the validity (anti-counterfeiting) of the information and generating a next block.

It should be noted that in the embodiment of the present invention, the pipeline data and the query data may be stored in blocks in the block chain.

The following is a detailed description of specific embodiments.

In the present embodiment, description will be made from the viewpoint of a data processing apparatus, which may be specifically integrated in a server.

The invention provides a data processing method, which comprises the following steps: the method comprises the following steps: acquiring a query request for the streaming data in the first service system in the second service system, wherein the query request comprises a first query condition; converting the first query condition into a second query condition corresponding to the storage structure according to the query request; generating the flow query data according to the second query condition; converting the pipelining query data into first pipelining data of a general structure; and outputting the first running water data at the second service system.

Referring to fig. 4, fig. 4 is a schematic flow chart of an embodiment of a data processing method according to the embodiment of the present invention, where the data processing method includes:

401. and acquiring an inquiry request for the streaming data in the first service system in the second service system, wherein the inquiry request comprises a first inquiry condition.

The first query condition includes a pipelining type and an object of the pipelining data, specifically, the object refers to a main object of a pipelining record, such as bank transfer pipelining, where the object refers to deposit of a user, and the transfer pipelining is a pipelining type.

402. And converting the first query condition into a second query condition corresponding to the storage structure according to the query request.

When data query is performed on the system, the corresponding memory is determined according to the first query condition, and the first query condition needs to be converted into a second query condition corresponding to the storage structure.

403. And generating the pipeline query data according to the second query condition.

And reading the target data in the corresponding memory through the second query condition so as to generate corresponding pipeline query data.

404. The pipelined query data is converted into first pipelined data of a generic structure.

When the pipeline query data is queried, the format converter converts the pipeline query data into first pipeline data with a universal structure, wherein the universal structure is a universal data format of the pipeline data in the second service system, for example, the universal structure is a pipeline structure of a service pipeline model (OperteLogModel), and the service pipeline model (OperteLogModel) is a universal pipeline structure, wherein fields mainly included in the general structure are divided into five types, namely a related attribute of an operated object, an index attribute of the operated object, an operation related attribute, a related attribute of an operator and an extended attribute.

405. And outputting the first running water data at the second service system.

The general first flow data conforms to the output format of the second service system, so that the first flow data can be output by the second service system.

In summary, in the embodiment of the present application, a query request for the streaming data in the first service system is obtained in the second service system, where the query request includes a first query condition; converting the first query condition into a second query condition corresponding to a storage structure according to the query request; generating the flow query data according to the second query condition; converting the pipelining query data into first pipelining data of a general structure; and outputting the first running water data at the second service system. The embodiment of the invention can solve the problems of universality and expansibility among different service systems, simultaneously reduce the operation cost and improve the query efficiency by acquiring the query condition of the streaming data in the first service system in the second service system and enabling the second service system to query the streaming data in the first service system through the format converter.

In some embodiments of the present application, the data processing method further includes: and acquiring a first storage request for storing the first streaming data to a storage corresponding to the second service system, wherein the first storage request comprises a first storage condition. And converting the first flow data into corresponding flow data in a first storage format. And storing the pipeline data in the first storage format into a memory corresponding to the first service system according to the first storage condition.

Referring to fig. 5, fig. 5 is a schematic flow chart of another embodiment of a data processing method according to an embodiment of the present invention, where the data processing method further includes:

501. and acquiring a first storage request for storing the first streaming data to a storage corresponding to the second service system, wherein the first storage request comprises a first storage condition.

502. And converting the first flow data into corresponding flow data in a first storage format.

503. And storing the pipeline data in the first storage format into a memory corresponding to the first service system according to the first storage condition.

In some embodiments of the present application, the data processing method further includes: acquiring a second storage request for storing the first streaming data to a storage corresponding to a second service system, wherein the second storage request comprises a first storage condition; converting the first flow data into corresponding flow data in a second storage format; and storing the pipeline data in the second storage format into a memory corresponding to the second service system according to the second storage condition.

Referring to fig. 6, fig. 6 is a schematic flow chart of another embodiment of a data processing method according to an embodiment of the present invention, where the data processing method further includes:

601. and acquiring a second storage request for storing the first streaming data to a storage corresponding to a second service system, wherein the second storage request comprises a first storage condition.

602. And converting the first flow data into corresponding flow data in a second storage format.

603. And storing the pipeline data in the second storage format into a memory corresponding to the second service system according to the second storage condition.

In some embodiments of the present application, the storing the pipeline data in the second storage format into a memory corresponding to the second service system according to the second storage condition includes: counting the target data volume of the first flowing water data; and when the target data volume reaches a preset data volume, asynchronously storing the first running water data according to the second storage condition.

Specifically, the storage mode includes synchronous storage and asynchronous storage, where the synchronous storage: the method is characterized in that data is directly saved into an associated memory at the time of calling, and then a saving result is returned. Asynchronous saving: in contrast to synchronous save, when a save is invoked, the system does not save the data directly, but pushes the data into a message queue, and the asynchronous save service writes the data into the associated memory by reading the message queue. Such saving can reduce the time consumption of the system, with the disadvantage that the data just saved cannot be immediately queried.

In some embodiments of the present application, the asynchronously storing the first streaming data according to the second storage condition includes: sending the first flow data to a preset message queue; and when the message queue is in turn to process the first flow data, reading the first flow data from the message queue according to the second storage condition and storing the first flow data into a corresponding storage of the second service system.

In some embodiments of the present application, after obtaining a second storage condition for storing the first streaming data in a memory corresponding to the second service system, the method further includes: repeatedly comparing the pipeline type of the pipeline data with the pipeline type of the stored pipeline data; when the result of the comparison is no repetition, defining the flow type of the flow data as enumeration information; and converting the enumeration information into a query field, and outputting the query field to a terminal for display.

Referring to fig. 7, it is a schematic flow chart of a data processing method according to another embodiment of the present invention, after obtaining a second storage condition for storing first streaming data in a memory corresponding to a second service system, the method further includes:

701. the pipeline type of the pipeline data is repeatedly compared with the pipeline type of the stored pipeline data.

702. When the result of the comparison is no duplication, the pipeline type of the pipeline data is defined as enumeration information.

703. And converting the enumeration information into a query field, and outputting the query field to a terminal for display.

Referring to fig. 8, fig. 8 is a schematic flow chart of another embodiment of the data processing method in the embodiment of the present invention, after obtaining a second storage condition for storing the first streaming data in the memory corresponding to the second service system, the method further includes:

801. repeatedly comparing the pipeline type of the pipeline data with the pipeline type of the stored pipeline data;

specifically, when a new streaming type is added, the streaming type needs to be repeatedly compared with existing streaming data types to prevent a chaotic phenomenon, and due to the addition of the new streaming type, for subsequent convenient query, the new streaming type can be set as enumeration information.

802. When the result of the comparison is no repetition, defining the flow type of the flow data as enumeration information;

according to the example in step 801, when the comparison structure between the praise running water data type and the original running water data type is no duplication, the praise running water data type is defined as the corresponding enumeration information.

803. And converting the enumeration information into a query field, and outputting the query field to a terminal for display.

In some embodiments of the present application, the data processing method further includes: while generating the pipeline data; generating a new block according to the pipeline data; the new tile is added to the chain of tiles.

The following describes a data query process in the data processing method according to an embodiment of the present invention with reference to a specific application scenario.

Referring to fig. 9, fig. 9 is a schematic flow chart of a data query process according to an embodiment of the present invention, where the flow chart includes:

901. the first terminal receives a query request of a user A for acquiring the bank account running data in the first business system in the second business system, wherein the query request comprises a first query condition.

Specifically, suppose that a user B stores ten thousand yuan in a chinese bank in a certain region of shenzhen city and records through a first service system of the bank, and a chinese bank in a region of hubei province of the user a needs to confirm storage information of the user B through a second service system of a local bank, where the first service system is a banking account pipelining service system, and the second service system is a pipelining service system in which the first service system is upgraded, and the second service system can query a banking card consumption record pipelining and personal transfer information of the user of the bank in addition to a fund access pipelining of the first service system, where the query request includes a pipelining type and an object, and the object is a banking account.

Specifically, referring to fig. 10, fig. 10 is a schematic diagram of a second service system architecture provided in the embodiment of the present invention, where the second service system includes a presentation layer 1001, an abstraction layer 1002, and a storage layer 1003, and the presentation layer provides a unified data structure and a query interface service for the outside; the abstraction layer realizes abstraction of bottom storage, and realizes conversion of general flowing water and a concrete storage structure through functions of routing, data conversion and the like; the storage layer is an instance of various stored data.

Step 901 is performed in a presentation layer in the second business system, where the presentation layer mainly provides external services, including a write stream interface, a read stream interface, and an Application Programming Interface (API) call service, and maintains enumeration information of an enumeration field, and through the enumeration information, business stream query with prompt may be implemented, specifically, in the API layer, data query is implemented.

The interface calling bottom layer is realized based on API calling, and some field mapping functions are packaged at the same time so as to realize the processing of special fields. The concrete query process is implemented at an abstraction layer.

Wherein, the specific calling relationship is as follows: the service flow model (OperateLogModel) is a general flow structure, wherein the fields mainly comprise five types of fields including the related attribute of the operated object, the index attribute of the operated object, the related attribute of the operation, the related attribute of the operator and the extended attribute.

Wherein, the operation object representation related field comprises type (type), object (object), object id (object id); the operated object index field is a function supported for supporting multi-middle dimension query, and comprises index1-5, the operator related field mainly comprises an operator, an operation platform, an operation sub-platform and the like, and the operation related field mainly comprises an operation (operation), a sub-operation (super), operation time, a message, remarks, priority, effectiveness and the like. The service flow query (OperateLogCriteria) is a general query structure, and the main purpose of the query structure is to realize the query function of the relevant fields of the operation objects, and the supported comparison functions include equal to, greater than, less than, fuzzy query and the like.

Executing the functions of relevant establishment and service flow inquiry through a centralized service flow management module; the task management module is used for managing related services of the background, including related functions of asynchronously storing the flow, exporting the service flow in batches and the like; the Http interface controller provides a network interface of Http protocol, and is a portal for front-end or other service interaction.

902. And the second business system generates the running data of the target bank account according to the query request.

Wherein, the step 902 is performed in an abstraction layer of the second service system, after the abstraction layer receives the query request from the presentation layer, the abstraction layer directs the new pipelining object data to a memory corresponding to the first service system by adding a corresponding routing rule, and converts the first query condition into a second query condition corresponding to a storage structure by using a format converter, and performs mutual conversion and query conversion on the bank account pipelining data in the first service system and the target bank account pipelining data of the second service system, so as to implement the access of the second service system to the first service system, specifically, the storage structure is converted into a general pipelining structure and the conversion of query fields, for example, an operator field in a relational database is a user, but a field of a general pipelining object is an operator, so that some format conversion functions need to be performed, and here, the function is implemented, at the time of query, the fields in the query object are mapped.

The abstraction layer is responsible for the matching of abstracted, pipelined data to concrete memory, the most important of which are the storage structure field mapping and table routing functions.

903. And the second business system outputs the running data of the target bank account.

Specifically, after the running data of the target bank account is generated, the running data of the target bank account is transmitted to a presentation layer of the second business system, and the running data of the target bank account is output.

In order to better implement the data processing method provided by the embodiment of the present invention, an embodiment of the present invention further provides a device based on the data processing method. The terms are the same as those in the data processing method, and details of implementation can be referred to the description in the method embodiment.

Referring to fig. 11, fig. 11 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention, where the data processing apparatus 1100 may include a first obtaining unit 1101, a first converting unit 1102, a first generating unit 1103, a second converting unit 1104, and a first output unit 1105, where:

a first obtaining unit 1101, configured to obtain, in a second service system, an inquiry request for the streaming data in the first service system, where the inquiry request includes a first inquiry condition.

A first conversion unit 1102, configured to convert the first query condition into a second query condition corresponding to the storage structure according to the query request.

A first generating unit 1103, configured to generate pipelined query data according to the second query condition.

A second conversion unit 1104, configured to convert the pipeline query data into the first pipeline data of the general structure.

A first output unit 1105, configured to output the first streaming data in the second service system.

and the second acquisition unit is used for acquiring a first storage request for storing the first streaming data to a memory corresponding to the second service system.

And the third conversion unit is used for converting the first flow data into corresponding flow data in the first storage format.

and the third acquisition unit is used for acquiring a second storage request for storing the first streaming data to a storage corresponding to the second service system.

And the fourth conversion unit is used for converting the first flow data into corresponding flow data in the second storage format.

and the statistical unit is used for counting the target data volume of the first flowing water data.

In some embodiments of the present application, the third storage unit is specifically configured to:

and sending the first flow data to a preset message queue.

And when the message queue is in turn to process the first flow data, reading the first flow data from the message queue according to the second storage condition and storing the first flow data into a corresponding storage of the second service system.

In some embodiments of the present application, after obtaining a second storage condition for storing the first streaming data in a memory corresponding to the second service system, the apparatus further includes:

and the comparison unit is used for repeatedly comparing the pipeline type of the pipeline data with the pipeline type of the stored pipeline data.

And the defining unit is used for defining the flow type of the flow data as enumeration information when the comparison result is no repetition.

And a fifth conversion unit, configured to convert the enumeration information into a query field.

And the second output unit is used for outputting the query field to the terminal for display.

In some embodiments of the present application, the apparatus further comprises a blockchain unit, the blockchain unit being configured to:

while generating the pipeline data;

generating a new block according to the pipeline data;

the new tile is added to the chain of tiles.

An embodiment of the present invention further provides a server, as shown in fig. 12, which shows a schematic structural diagram of the server according to the embodiment of the present invention, specifically:

the server may include components such as a processor 1201 of one or more processing cores, memory 1202 of one or more computer-readable storage media, a power supply 1203, and an input unit 1204. Those skilled in the art will appreciate that the server architecture shown in FIG. 12 is not meant to be limiting, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:

the processor 1201 is a control center of the server, connects various parts of the entire server using various interfaces and lines, and performs various functions of the server and processes data by operating or executing software programs and/or modules stored in the memory 1202 and calling data stored in the memory 1202, thereby performing overall monitoring of the server. Optionally, the processor 1201 may include one or more processing cores; preferably, the processor 1201 may integrate an application processor, which mainly handles operating storage media, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communication. It will be appreciated that the modem processor described above may not be integrated into the processor 1201.

The memory 1202 may be used to store software programs and modules, and the processor 1201 executes various functional applications and data processing by operating the software programs and modules stored in the memory 1202. The memory 1202 may mainly include a storage program area and a storage data area, wherein the storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for operating a storage medium, at least one function, and the like; the storage data area may store data created according to the use of the server, and the like. Further, the memory 1202 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 1202 may also include a memory controller to provide the processor 1201 with access to the memory 1202.

The server further comprises a power supply 1203 for supplying power to each component, and preferably, the power supply 1203 may be logically connected to the processor 1201 through a power management storage medium, so as to realize functions of managing charging, discharging, power consumption management and the like through the power management storage medium. The power supply 1203 may also include any component of one or more of a dc or ac power source, a rechargeable storage medium, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

The server may further include an input unit 1204, and the input unit 1204 may be used to receive input numeric or character information and generate a keyboard, mouse, joystick, optical or trackball signal input in relation to user settings and function control.

Although not shown, the server may further include a display unit and the like, which will not be described in detail herein. Specifically, in this embodiment, the processor 1201 in the server loads the executable file corresponding to the process of one or more application programs into the memory 1202 according to the following instructions, and the processor 1201 runs the application programs stored in the memory 1202, thereby implementing various functions as follows:

acquiring a query request for the streaming data in the first service system in the second service system, wherein the query request comprises a first query condition; converting the first query condition into a second query condition corresponding to the storage structure according to the query request; generating the flow query data according to the second query condition; converting the pipelining query data into first pipelining data of a general structure; and outputting the first running water data at the second service system. In the above embodiments, the descriptions of the embodiments have respective emphasis, and parts that are not described in detail in a certain embodiment may refer to the above detailed description of the data processing method, and are not described herein again.

Accordingly, as shown in fig. 13, the terminal may include a Radio Frequency (RF) circuit 1301, a memory 1302 including one or more computer-readable storage media, an input unit 1303, a display unit 1304, a sensor 1305, an audio circuit 1306, a wireless fidelity (WiFi) module 1307, a processor 1308 including one or more processing cores, a power supply 1309, and other components. Those skilled in the art will appreciate that the terminal structure shown in fig. 13 is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components. Wherein:

the RF circuit 1301 may be used for receiving and transmitting signals during information transmission and reception or during a call, and in particular, receive downlink information from a base station and then send the received downlink information to one or more processors 1308 for processing; in addition, data relating to uplink is transmitted to the base station. In general, the RF circuit 1301 includes, but is not limited to, an antenna, at least one Amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, RF circuit 1301 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Message Service (SMS), and the like.

The memory 1302 may be used to store software programs and modules, and the processor 1308 may execute various functional applications and data processing by operating the software programs and modules stored in the memory 1302. The memory 1302 may mainly include a storage program area and a storage data area, wherein the storage program area may store an application program (such as a sound playing function, an image playing function, etc.) required for operating the storage medium, at least one function, and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the terminal, etc. Further, the memory 1302 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Correspondingly, the memory 1302 may further include a memory controller to provide the processor 1308 and the input unit 1303 with access to the memory 1302, and in this embodiment, the memory may store the relationship chain data of the user and the attribute information of the user.

The input unit 1303 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. Specifically, in a particular embodiment, the input unit 1303 may include a touch-sensitive surface as well as other input devices. The touch-sensitive surface, also referred to as a touch display screen or a touch pad, may collect touch operations by a user (e.g., operations by a user on or near the touch-sensitive surface using a finger, a stylus, or any other suitable object or attachment) thereon or nearby, and drive the corresponding connection device according to a predetermined program. Alternatively, the touch sensitive surface may comprise two parts, a touch detection means and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, and sends the touch point coordinates to the processor 1308, where the touch controller can receive and execute commands sent by the processor 1308. In addition, touch sensitive surfaces may be implemented using various types of resistive, capacitive, infrared, and surface acoustic waves. The input unit 1303 may include other input devices in addition to the touch-sensitive surface. In particular, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

The display unit 1304 may be used to display information input by or provided to a user and various graphical user interfaces of the terminal, which may be made up of graphics, text, icons, video, and any combination thereof. The display unit 1304 may include a display panel, and optionally, the display panel may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, the touch-sensitive surface may overlay the display panel, and when a touch operation is detected on or near the touch-sensitive surface, the touch operation is communicated to the processor 1308 for determining the type of touch event, and the processor 1308 then provides a corresponding visual output on the display panel according to the type of touch event. Although in FIG. 13 the touch sensitive surface and the display panel are two separate components to implement input and output functions, in some embodiments the touch sensitive surface may be integrated with the display panel to implement input and output functions.

The terminal may also include at least one sensor 1305, such as light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor that may adjust the brightness of the display panel according to the brightness of ambient light, and a proximity sensor that may turn off the display panel and/or the backlight when the terminal is moved to the ear. As one of the motion sensors, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally, three axes), can detect the magnitude and direction of gravity when the mobile phone is stationary, and can be used for applications of recognizing the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer and tapping), and the like; as for other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which can be configured in the terminal, detailed description is omitted here.

Audio circuitry 1306, a speaker, and a microphone may provide an audio interface between a user and a terminal. The audio circuit 1306 can transmit the electrical signal converted from the received audio data to a speaker, and the electrical signal is converted into a sound signal by the speaker to be output; on the other hand, the microphone converts a collected sound signal into an electric signal, converts the electric signal into audio data after being received by the audio circuit 1306, and then processes the audio data by the audio data output processor 1308, and then sends the audio data to, for example, another terminal via the RF circuit 1301, or outputs the audio data to the memory 1302 for further processing. The audio circuit 1306 may also include an earbud jack to provide peripheral headset communication with the terminal.

WiFi belongs to a short-distance wireless transmission technology, and the terminal can help a user to send and receive e-mails, browse webpages, access streaming media and the like through the WiFi module 1307, and provides wireless broadband internet access for the user. Although fig. 13 shows the WiFi module 1307, it is understood that it does not belong to the essential constitution of the terminal, and can be omitted entirely as needed within the scope not changing the essence of the invention.

The processor 1308 is a control center of the terminal, connects various parts of the entire handset by various interfaces and lines, and performs various functions of the terminal and processes data by operating or executing software programs and/or modules stored in the memory 1302 and calling data stored in the memory 1302, thereby integrally monitoring the handset. Optionally, processor 1308 may include one or more processing cores; preferably, the processor 1308 may integrate an application processor, which mainly handles operations of storage media, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It is to be appreciated that the modem processor described above may not be integrated into processor 1308.

The terminal also includes a power supply 1309 (e.g., a battery) for powering the various components, which may preferably be logically connected to the processor 1308 via a power management storage medium that provides functionality to manage charging, discharging, and power consumption management. The power supply 1309 can also include any component or components of one or more dc or ac power supplies, rechargeable storage media, power failure detection circuitry, a power converter or inverter, a power status indicator, and the like.

Although not shown, the terminal may further include a camera, a bluetooth module, and the like, which will not be described herein. Specifically, in this embodiment, the processor 1308 in the terminal loads the executable file corresponding to the process of one or more application programs into the memory 1302 according to the following instructions, and the processor 1308 runs the application programs stored in the memory 1302, thereby implementing various functions.

In the above embodiments, the descriptions of the embodiments have respective emphasis, and parts that are not described in detail in a certain embodiment may refer to the above detailed description of the data processing method, and are not described herein again.

It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.

To this end, the present invention provides a storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the data processing methods provided by the embodiments of the present invention. For example, the instructions may perform the steps of:

acquiring a query request for the streaming data in the first service system in the second service system, wherein the query request comprises a first query condition; converting the first query condition into a second query condition corresponding to the storage structure according to the query request; generating the flow query data according to the second query condition; converting the pipelining query data into first pipelining data of a general structure; and outputting the first running water data at the second service system.

Since the instructions stored in the storage medium can execute the steps in any data processing method provided in the embodiment of the present invention, the beneficial effects that can be achieved by any data processing method provided in the embodiment of the present invention can be achieved, which are detailed in the foregoing embodiments and will not be described herein again.

Embodiments of the present invention provide another storage medium, in which a plurality of instructions are stored, where the instructions can be loaded by a processor to execute the steps in any one of the data processing methods provided in the embodiments of the present invention. For example, the instructions may perform the steps of:

acquiring a second storage request for storing the first streaming data to a storage corresponding to a second service system, wherein the second storage request comprises a first storage condition; converting the first flow data into corresponding flow data in a second storage format; and storing the pipeline data in the second storage format into a memory corresponding to the second service system according to the second storage condition.

The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.

Wherein the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.

The data processing method and apparatus provided by the embodiments of the present invention are described in detail above, and the principles and embodiments of the present invention are explained herein by applying specific examples, and the description of the embodiments is only used to help understanding the method and the core idea of the present invention; meanwhile, for those skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in view of the above, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A method of data processing, the method comprising:

generating the flow query data according to the second query condition;

and outputting the first running water data at the second service system.

2. The data processing method of claim 1, wherein the method further comprises:

3. The data processing method of claim 1, wherein the method further comprises:

4. The data processing method according to claim 3, wherein the storing the pipeline data in the second storage format into a memory corresponding to the second service system according to the second storage condition includes:

counting the target data volume of the first flowing water data;

5. The data processing method according to claim 4, wherein the asynchronously storing the first streaming data according to the second storage condition comprises:

sending the first flow data to a preset message queue;

6. The data processing method according to claim 4, wherein after acquiring the second storage condition for storing the first streaming data in the memory corresponding to the second service system, the method further comprises:

7. The data processing method according to any one of claims 1 to 6, characterized in that the method further comprises:

while generating the pipeline data;

generating a new block according to the pipeline data;

adding the new tile to the chain of tiles.

8. A data processing apparatus, characterized in that the apparatus comprises:

9. The data processing apparatus of claim 8, wherein the apparatus further comprises:

10. The data processing apparatus of claim 8, wherein the apparatus further comprises: