CN115905315A - Multi-type data fusion processing method and system - Google Patents

Multi-type data fusion processing method and system Download PDF

Info

Publication number
CN115905315A
CN115905315A CN202211388587.9A CN202211388587A CN115905315A CN 115905315 A CN115905315 A CN 115905315A CN 202211388587 A CN202211388587 A CN 202211388587A CN 115905315 A CN115905315 A CN 115905315A
Authority
CN
China
Prior art keywords
data
user
query request
acquired
specified
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211388587.9A
Other languages
Chinese (zh)
Inventor
韩锡锐
程学涛
冯晋阳
蒋林雪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Deta Jingyao Information Technology Co ltd
Original Assignee
Beijing Deta Jingyao Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Deta Jingyao Information Technology Co ltd filed Critical Beijing Deta Jingyao Information Technology Co ltd
Priority to CN202211388587.9A priority Critical patent/CN115905315A/en
Publication of CN115905315A publication Critical patent/CN115905315A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application relates to a method and a system for fusion processing of multi-type data, which belong to the technical field of data processing and comprise the following steps: receiving a data query request sent by a user; the data query request comprises a data output type specified by a user; acquiring data according to the data query request, and fusing the acquired data into a data output type specified by a user according to a preset data fusion rule; desensitizing the fused data and sending the data to a user. According to the technical scheme, the process of storing the data by the intermediate hard disk is not needed, the data query result is returned quickly, and the real-time performance is high; and data redundancy can not be generated, the disk space is saved, and the data processing complexity is reduced. In addition, according to the technical scheme, the acquired data are fused into the data output type appointed by the user according to the preset data fusion rule, and the data returned to the user are diversified, so that flexible and changeable business requirements are met.

Description

Multi-type data fusion processing method and system
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and a system for processing multiple types of data in a fusion manner.
Background
The data used by the computer system has various types, such as structured data, document type data, graph data and the like, different types of data generally have specific storage modes, and multiple types of data are often required to be obtained simultaneously in an actual service scene to meet the service requirements of users.
In the prior art, sharing of different types of data is generally realized through data integration, that is, data of a plurality of storage tools is transferred to one storage tool, but the data needs to be read, stored and read again when being transferred, so that the processing chain is long, the processing process is complex, and the query time is long. This results in poor real-time data acquisition and delivery, data redundancy due to intermediate storage, and the problem of single format of data ultimately returned to the user.
Disclosure of Invention
In order to overcome the problems that the data acquisition and transmission real-time performance is poor, data redundancy is easily caused by intermediate storage and the finally returned data form is single in the related technology at least to a certain extent, the application provides a multi-type data fusion processing method and system.
The scheme of the application is as follows:
according to a first aspect of the embodiments of the present application, a method for processing fusion of multiple types of data is provided, including:
receiving a data query request sent by a user; the data query request comprises a data output type specified by a user;
acquiring data according to the data query request, and fusing the acquired data into a data output type specified by a user according to a preset data fusion rule;
desensitizing the fused data and sending the data to a user.
Preferably, the data query request further includes a data display requirement specified by the user;
the acquiring data according to the data query request includes:
according to the data display requirement appointed by the user, at least dividing the data acquisition into the following parts: data source definitions, return field definitions, desensitization field definitions, constant fields added in return values, paging queries, sort patterns, filter criteria, and nested queries.
Preferably, the acquiring data according to the data query request further includes:
and constructing a query statement according to a data output type specified by a user, and acquiring distributed concurrent data in a database according to the query statement based on a distributed architecture.
Preferably, the data output types include at least:
table data, tree structure data and graph relation type structure data;
the data fusion rule at least comprises:
when the main data of the acquired data comprise a first preset field, fusing the acquired data into form data;
when the main data of the acquired data contains a first preset field, fusing the acquired data into tree structure data;
and when the specified data source of the main data of the acquired data is a preset data source, fusing the acquired data into graph relation type structure data.
Preferably, the data query request further includes a user-defined data field specified by a user;
the method further comprises the following steps:
and customizing and modifying the fused data according to the custom data field specified by the user.
Preferably, after receiving a data query request sent by a user, the method further includes:
and authenticating the user based on a preset authentication mode.
Preferably, the authentication means at least includes:
inquiring whether the user is authorized;
after determining the user authorization, determining whether the user authorization is due.
Preferably, the method further comprises:
and carrying out log recording on the data query request and the fused data.
Preferably, the method further comprises:
and analyzing the using times, the calling party, the data volume and the response time of the data according to the recorded log, and generating an analysis result.
According to a second aspect of embodiments of the present application, there is provided a system for merging multiple types of data, including:
a processor and a memory;
the processor and the memory are connected through a communication bus:
the processor is used for calling and executing the program stored in the memory;
the memory is used for storing a program, and the program is at least used for executing the fusion processing method of the multi-type data.
The technical scheme provided by the application can comprise the following beneficial effects: the method for fusing and processing the multi-type data comprises the following steps: receiving a data query request sent by a user; the data query request comprises a data output type specified by a user; acquiring data according to the data query request, and fusing the acquired data into a data output type specified by a user according to a preset data fusion rule; and desensitizing the fused data and sending the data to a user. According to the technical scheme, the process of storing the data by the intermediate hard disk is not needed, the data query result is returned quickly, and the real-time performance is high; and data redundancy can not be generated, the disk space is saved, and the data processing complexity is reduced. In the technical scheme of the application, the acquired data are fused into the data output type appointed by the user according to the preset data fusion rule, and the data returned to the user are diversified, so that flexible and changeable service requirements are met.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
FIG. 1 is a flow chart illustrating a method for processing multi-type data fusion according to an embodiment of the present disclosure;
fig. 2 is a block diagram illustrating a multi-type data fusion processing system according to an embodiment of the present application.
Reference numerals: a processor-21; a memory-22.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the application, as detailed in the appended claims.
Example one
Fig. 1 is a schematic flowchart of a method for fusing multi-type data according to an embodiment of the present invention, and referring to fig. 1, the method for fusing multi-type data includes:
s11: receiving a data query request sent by a user; the data query request comprises a data output type specified by a user;
s12: acquiring data according to the data query request, and fusing the acquired data into a data output type specified by a user according to a preset data fusion rule;
s13: desensitizing the fused data and sending the data to a user.
It should be noted that the technical solution of this embodiment relates to the technical field of data processing, and is specifically applied to the process of acquiring and fusing multi-type data, such as relational data, document data, and graph relational data.
It can be understood that, in the technical scheme in this embodiment, fused data is desensitized before being sent to a user, data desensitization refers to performing data deformation on some sensitive information through a desensitization rule, so as to implement reliable protection of sensitive private data, and the technical scheme in this embodiment desensitizes the private data to meet the requirements of various service scenarios.
It can be understood that the method for fusing multiple types of data in this embodiment includes: receiving a data query request sent by a user; the data query request comprises a data output type specified by a user; acquiring data according to the data query request, and fusing the acquired data into a data output type specified by a user according to a preset data fusion rule; desensitizing the fused data and sending the data to a user. In the technical scheme of the embodiment, the process of storing the data by the intermediate hard disk is not needed, the data query result is returned quickly, and the real-time performance is high; and data redundancy can not be generated, the disk space is saved, and the data processing complexity is reduced. In addition, in the technical scheme of the embodiment, the acquired data is fused into the data output type specified by the user according to the preset data fusion rule, and the data returned to the user is diversified, so that flexible and changeable service requirements are met.
It should be noted that, in an implementation manner of this embodiment, the data query request further includes a data presentation requirement specified by the user;
acquiring data according to the data query request, comprising:
according to the data display requirement appointed by a user, at least dividing the data acquisition into the following parts: data source definitions, return field definitions, desensitization field definitions, constant fields added to return values, paging queries, sorting patterns, filtering conditions, and nested queries.
In particular practice, the data source definition includes a data source type and a data source name; the first half of the return field definition is the field name to be returned and the second half is the name of the data store.
In a specific practice, the data query request is generally in the form of a code, and in a code statement of the data query request, a name is used for representing a data fusion name; the dataSource is used for representing the data source type; indexName is used to represent the name of the data source; fields is used to represent the query result field mapping (field name in result to left of colon, data source to right of colon); dataMasking is used to indicate the field that needs desensitization; consts is used to represent fixed value fields (to supplement fields. Left is field name and right is value); the filters comprise three parts of pagination, sort and filter; wherein, pagination is used for representing paging; page is used to indicate page number, starting with 1; size is used to indicate how many bars per page; sort is used to denote the sort; the direction is used for indicating the sorting direction and includes two types: 1.ASC: descending order, 2.DESC: ascending; fields is used to indicate the sort field, which may be multiple; filter is used to denote filtering; and/or and field are field names; value is used to represent a value; linked is used to represent the associated query; rela is used for representing the association condition (the left side of the colon is the field of the association table, and the right side of the colon is the field of the main table, when the values of the two fields are equal, the two data are combined into one piece);
ops are used to represent operators, and include at least the following:
eq: is equal to
Neq: is not equal to
Gt: is greater than
Gte: is greater than or equal to
Lt: is less than
Lte: is less than or equal to
Contacts: comprises
It should be noted that, acquiring data according to the data query request further includes:
and constructing a query statement according to the data output type specified by the user, and acquiring distributed concurrent data in the database according to the query statement based on a distributed architecture.
It can be understood that, in the technical solution of this embodiment, a distributed architecture is adopted, and distributed concurrent data acquisition and multithread memory data fusion are performed based on the distributed architecture, so that the efficiency of data acquisition and data fusion can be greatly improved.
It should be noted that the data output types at least include:
table data, tree structure data and graph relational structure data;
the data fusion rule at least comprises:
when the main data of the acquired data contains a first preset field, fusing the acquired data into form data;
when the main data of the acquired data contains a first preset field, fusing the acquired data into tree structure data;
and when the specified data source of the main data of the acquired data is a preset data source, fusing the acquired data into graph relation type structure data.
The main data is the outermost data of the acquired data.
In specific practice, if the main data of the acquired data contains linked fields, which indicates that the data has nested association, and rela indicates associated fields, the associated fields in the acquired data are fused into table data. When the main data of the acquired data contains the agg field, it means that the main data is a tree structure, and the acquired data is merged into tree structure data. And when the appointed data source of the main data of the obtained data is a neo4j data source, indicating that the data source is a graph database, and fusing the obtained data into graph relation type structure data.
It should be noted that, in an implementation manner of this embodiment, the data query request further includes a custom data field specified by the user;
the method further comprises the following steps:
and customizing and modifying the fused data according to the custom data field specified by the user.
It can be understood that, the technical solution in this embodiment supports the user to define the data field and modify the personalized data field autonomously, which can meet the requirement of more diversified service scenarios.
It should be noted that, after receiving the data query request sent by the user, the method further includes:
and authenticating the user based on a preset authentication mode.
Specifically, the authentication mode at least comprises:
inquiring whether the user is authorized;
after determining the user authorization, determining whether the user authorization is expired.
It should be noted that different data have different rights, and the rights of the user need to be checked when the data is used.
It should be noted that the authentication mode can be customized, and different authentication modes can be defined according to different service scenarios in actual implementation.
It can be understood that, in the technical scheme in this embodiment, interface authentication is added to ensure data security.
It should be noted that the method further includes:
and carrying out log recording on the data query request and the fused data.
Further, the method further comprises:
and analyzing the using times, calling party, data quantity and response time of the data according to the recorded log, and generating an analysis result.
It can be understood that, in the technical solution in this embodiment, the data query request and the fused data are subjected to log recording, and the number of times of use, the caller, the data volume, and the response time of the data are analyzed according to the recorded log, and an analysis result is generated. The use efficiency of the data is improved by auditing and analyzing the use behavior of the data.
It should be noted that, in an extension of this embodiment, after receiving a data query request sent by a user, the method further includes:
current limiting is performed.
It can be understood that, in the technical scheme in this embodiment, after receiving a data query request sent by a user, current limiting is performed first, authentication is performed, and data desensitization is performed finally, so that the security of data is guaranteed to the greatest extent.
Example two
Fig. 2 is a block diagram of a system for fusing multiple types of data according to an embodiment of the present invention, and referring to fig. 2, the system for fusing multiple types of data includes:
a processor 21 and a memory 22;
the processor 21 is connected to the memory 22 by a communication bus:
the processor 21 is configured to call and execute a program stored in the memory 22;
the memory 22 is used for storing a program, and the program is at least used for executing the fusion processing method of the multi-type data in the above embodiment.
It is understood that the same or similar parts in the above embodiments may be mutually referred to, and the same or similar parts in other embodiments may be referred to for the content which is not described in detail in some embodiments.
It should be noted that, in the description of the present application, the terms "first", "second", etc. are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Further, in the description of the present application, the meaning of "a plurality" means at least two unless otherwise specified.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc.
In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.

Claims (10)

1.A method for processing fusion of multi-type data is characterized by comprising the following steps:
receiving a data query request sent by a user; the data query request comprises a data output type specified by a user;
acquiring data according to the data query request, and fusing the acquired data into a data output type specified by a user according to a preset data fusion rule;
and desensitizing the fused data and sending the data to a user.
2. The method according to claim 1, wherein the data query request further includes a data presentation requirement specified by a user;
the acquiring data according to the data query request includes:
according to the data display requirement appointed by the user, at least dividing the data acquisition into the following parts: data source definitions, return field definitions, desensitization field definitions, constant fields added in return values, paging queries, sort patterns, filter criteria, and nested queries.
3. The method of claim 1, wherein obtaining data according to the data query request further comprises:
and constructing a query statement according to a data output type specified by a user, and performing distributed concurrent data acquisition in a database according to the query statement based on a distributed architecture.
4. The method of claim 1, wherein the data output types include at least:
table data, tree structure data and graph relational structure data;
the data fusion rule at least comprises:
when the main data of the acquired data contains a first preset field, fusing the acquired data into form data;
when the main data of the acquired data comprise a first preset field, fusing the acquired data into tree structure data;
and when the specified data source of the main data of the acquired data is a preset data source, fusing the acquired data into graph relation type structure data.
5. The method of claim 1, wherein the data query request further includes a user-specified custom data field;
the method further comprises the following steps:
and customizing and modifying the fused data according to the custom data field specified by the user.
6. The method of claim 1, wherein after receiving a data query request sent by a user, the method further comprises:
and authenticating the user based on a preset authentication mode.
7. The method according to claim 6, wherein the authentication manner at least comprises:
inquiring whether the user is authorized;
after determining the user authorization, determining whether the user authorization is due.
8. The method of claim 1, further comprising:
and carrying out log recording on the data query request and the fused data.
9. The method of claim 8, further comprising:
and analyzing the using times, the calling party, the data volume and the response time of the data according to the recorded log, and generating an analysis result.
10. A system for processing multiple types of data in a fusion manner, comprising:
a processor and a memory;
the processor and the memory are connected through a communication bus:
the processor is used for calling and executing the program stored in the memory;
the memory is used for storing a program, and the program is at least used for executing the fusion processing method of multi-type data in any one of claims 1-9.
CN202211388587.9A 2022-11-08 2022-11-08 Multi-type data fusion processing method and system Pending CN115905315A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211388587.9A CN115905315A (en) 2022-11-08 2022-11-08 Multi-type data fusion processing method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211388587.9A CN115905315A (en) 2022-11-08 2022-11-08 Multi-type data fusion processing method and system

Publications (1)

Publication Number Publication Date
CN115905315A true CN115905315A (en) 2023-04-04

Family

ID=86477686

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211388587.9A Pending CN115905315A (en) 2022-11-08 2022-11-08 Multi-type data fusion processing method and system

Country Status (1)

Country Link
CN (1) CN115905315A (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106095862A (en) * 2016-06-02 2016-11-09 四川大学 The storage method of centralized expansible pattern of fusion multi-dimensional complicated structural relation data
CN106528810A (en) * 2016-11-18 2017-03-22 党玉龙 Method for integrating heterogeneous data to facilitate rapid big data analysis
CN108090154A (en) * 2017-12-08 2018-05-29 广州市申迪计算机系统有限公司 A kind of isomerous multi-source data fusion querying method and device
CN109446253A (en) * 2018-09-25 2019-03-08 平安科技(深圳)有限公司 Data query control method, device, computer equipment and storage medium
CN112115314A (en) * 2020-09-16 2020-12-22 江苏开拓信息与系统有限公司 General government affair big data aggregation retrieval system and construction method

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106095862A (en) * 2016-06-02 2016-11-09 四川大学 The storage method of centralized expansible pattern of fusion multi-dimensional complicated structural relation data
CN106528810A (en) * 2016-11-18 2017-03-22 党玉龙 Method for integrating heterogeneous data to facilitate rapid big data analysis
CN108090154A (en) * 2017-12-08 2018-05-29 广州市申迪计算机系统有限公司 A kind of isomerous multi-source data fusion querying method and device
CN109446253A (en) * 2018-09-25 2019-03-08 平安科技(深圳)有限公司 Data query control method, device, computer equipment and storage medium
CN112115314A (en) * 2020-09-16 2020-12-22 江苏开拓信息与系统有限公司 General government affair big data aggregation retrieval system and construction method

Similar Documents

Publication Publication Date Title
CN109862379A (en) A kind of log processing method, device, mobile terminal and storage medium
CN113704243A (en) Data analysis method, data analysis device, computer device, and storage medium
CN110109981B (en) Information display method and device for work queue, computer equipment and storage medium
CN102436503A (en) Data condition filtering and screening method and screener
CN112084167A (en) Authority filtering method and device and storage medium
CN111562953A (en) Interface calling method and device, computer device and readable storage medium
CN110597896A (en) Data display method, data display device and terminal equipment
CN111355802A (en) Information pushing method and device
CN110597861A (en) Real-time alarm method, device and equipment and computer readable storage medium
CN111338716A (en) Data processing method and device based on rule engine and terminal equipment
CN113448985A (en) API (application program interface) interface generation method, calling method and device and electronic equipment
CN117707478A (en) Demand management method, device, equipment and storage medium
CN115905315A (en) Multi-type data fusion processing method and system
CN109558403B (en) Data aggregation method and device, computer device and computer readable storage medium
WO2017117870A1 (en) Processing method and apparatus for trap instruction trap
CN110941658A (en) Data export method, device, server and storage medium
CN116010347A (en) Resource updating method, device, system, electronic equipment and storage medium
CN115033590A (en) Multi-domain data fusion method, device and storage medium
CN115599801A (en) Data query method, system, electronic equipment and storage medium
CN112333040B (en) Flow separation method and device, storage medium and computer equipment
CN112256731A (en) Data display method and device and electronic equipment
CN113849678A (en) Method and system for searching and rapidly positioning parking image by dichotomy
CN113641678A (en) Dynamic service configuration method and system based on multi-dimensional form
CN115827589A (en) Authority verification method and device, electronic equipment and storage medium
CN114371866A (en) Version reconfiguration test method, device and equipment of service system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20230404