WO2020083358A1 - 一种可视化业务数据操作系统、方法、设备及介质 - Google Patents
一种可视化业务数据操作系统、方法、设备及介质 Download PDFInfo
- Publication number
- WO2020083358A1 WO2020083358A1 PCT/CN2019/113197 CN2019113197W WO2020083358A1 WO 2020083358 A1 WO2020083358 A1 WO 2020083358A1 CN 2019113197 W CN2019113197 W CN 2019113197W WO 2020083358 A1 WO2020083358 A1 WO 2020083358A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data operation
- data
- operation instruction
- user
- server
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
Definitions
- This application relates to, but not limited to, the Internet and database fields, and in particular, to a visual service data operating system, method, device, and medium.
- the structure of the data set returned by the query is also different. It requires users to de-parse the data and reorganize it into a structured data set according to a specific format and store it in a file. Low, and more importantly, it requires careful operation and patience in operation. A slight negligence will cause errors, which will bring great inconvenience to data processing.
- a data analyst mainly filters and aggregates business data. Since the content of the business is not fixed, the data structure varies. The angle of the problem to be analyzed varies from time to time. This causes the analyst to query different content each time. Because the query syntax of each database is different, and the query syntax is difficult to learn, most of them are queryed through the command line interface, which is very inconvenient to use. For those who are not familiar with it, it is impossible to start, and the result is returned in a single form .
- this application provides a visual business data operating system and method.
- a visual business data operating system including:
- the client is configured to provide a GUI, receive user operation instructions executed by the user on the GUI, convert the user operation instructions into data operation instructions, send the data operation instructions to the server, and receive The data operation result, displaying the data operation result on the GUI, and / or storing the data operation result locally;
- the server is configured to receive the data operation instruction, generate a field structure corresponding to the data operation instruction and / or generate an available operation scheme of the field structure corresponding to the data operation instruction, and execute the data operation instruction, Obtain the data operation result, and send the data operation result to the client.
- the user operation instruction includes at least one of the following:
- the server is also set to:
- the valid data in the data operation instruction is spliced to generate a standard sentence for data operation.
- the data operation standard statement is Elasticsearch's Query and DSL format data operation statement
- the database and table are Elasticsearch index and type
- the server side provides Elasticsearch service
- the data operation instruction is called using Elasticsearch API, Send the data operation instruction to the server.
- the list selection supports regular expression matching.
- the client is also set to verify whether the syntax of the Query DSL format data operation statement is correct.
- a method for visualizing business data operation including:
- the user operation instruction includes at least one of the following:
- the method further includes:
- the data operation standard statement is Elasticsearch's Query and DSL format data operation statement
- the database and table are Elasticsearch index and type
- the server side provides Elasticsearch service
- the data operation instruction is called using Elasticsearch API, Send the data operation instruction to the server.
- the list selection supports regular expression matching.
- the method further includes:
- a computer device including a memory, a processor, and a computer program stored on the memory and executable on the processor, which is implemented when the processor executes the program The steps of the above method.
- a computer-readable storage medium is provided, and a computer program is stored on the storage medium, and when the program is executed by a processor, the steps of the above method are implemented.
- the visual graphical interface can intuitively display the query conditions, without the need to learn and master complex query statements.
- the results of the query can directly generate txt, xlsx and other format files.
- Fig. 1 is a block diagram of a visual business data operating system according to an exemplary embodiment
- Fig. 2 is a flowchart of a method for visualizing business data according to an exemplary embodiment
- Fig. 3 is a block diagram of an apparatus for visualizing business data operation method according to an exemplary embodiment.
- FIG. 1 exemplarily shows a schematic diagram of a visual service data operating system 100 according to an embodiment of the present invention.
- the visual service data operating system 100 includes:
- the client 101 is configured to provide a GUI, receive user operation instructions executed by the user 102 in the GUI, convert the user operation instructions into data operation instructions, send the data operation instructions to the server side, and receive the data operation results from the server side.
- the results of data operations are displayed on the website, and / or the results of data operations are stored locally;
- the server 103 is configured to receive a data operation instruction, generate a field structure corresponding to the data operation instruction and / or generate an available operation scheme of the field structure corresponding to the data operation instruction, execute the data operation instruction, obtain the data operation result, and convert the data operation result Send to client 101.
- the user operation instruction includes at least one of the following:
- connection account The data source, connection account and password input by the user 102 or selected through the list;
- the conditions for data operations include limiting conditions, query conditions, and aggregation conditions.
- the server 103 is also set to:
- the valid data in the data operation instruction is concatenated to generate a standard sentence for data operation.
- the standard data operation statement is Elasticsearch's Query format data operation statement.
- the database and table are the indexes and types of Elasticsearch.
- the server-side 103 provides Elasticsearch services.
- the Elasticsearch API is used to call the data operation instructions and send the data operation instructions. To the server 103.
- the list selection supports regular expression matching.
- the client 101 is also configured to verify whether the syntax of the data manipulation statement in the Query DSL format is correct.
- users can use the following methods to enter specific user operation instructions:
- the user can select a data storage framework (for example, Elasticsearch) on the interface, and configure the address, port, and identity authentication information required to connect to the data source.
- a data storage framework for example, Elasticsearch
- the operations applicable to each type of field can be predetermined. You can monitor the fields selected by the user in real time, showing the options that the user can choose. As long as the user selects the corresponding operation, the analysis setting of the selected data can be completed.
- the query statement required by the user-specified data storage framework (for example, the above-mentioned Query format data operation statement in DSL format) can be generated through user configuration. The detailed information of each data configuration can be saved.
- the detailed information includes the source of the data (for example, the above data source, database, and table), the field to be queried (that is, the above field information), and the specific operations performed on each field (for example, the types of data operations mentioned above—addition, deletion, checking, modification, and conditions of data operations).
- the client 101 receives the index and type to be queried input or selected by the user 102, and the index and type are equivalent to the URL part in the standard DSL query.
- Index and type can be one or multiple. You can match multiple indexes and types through wildcard matching (that is, the above supports regular expression matching).
- the client 101 receives the fields to be queried input or selected by the user 102, and these fields are equivalent to the _source part in the standard DSL query Body. That is, the user 102 can select the field information in the index based on the index information that has been selected in the previous step. For example, the user 102 can directly select multiple fields displayed in the list control by means of multiple selection. If the user does not make a selection, all fields are queried by default.
- the client 101 receives specific query conditions input or selected by the user 102, and these query conditions are equivalent to the query part in the standard DSL query Body.
- a bool compound query clause can be used, which can support must, must_not, should, filter, etc.
- the user 102 can add qualification conditions to any field through the interface, and the types of the qualification conditions include term, wildcard, perfix, fuzzy, range, missing, and so on.
- the client 101 receives specific aggregation conditions input or selected by the user 102 (for example, applicable to data operations of type query), and these aggregation conditions are equivalent to the part of aggs in the standard DSL query Body.
- the user 102 can directly select the type of aggregation conditions through the interface and set specific aggregation conditions, and the aggregation conditions can be repeatedly nested.
- the client 101 may include a sentence generating device for generating a standard DSL sentence, and the sentence generating device includes the following modules:
- Index type selection module set to generate the URL part of the DSL query. That is, it is set to set the Elasticsearch address after the user 102 uses the interface, and obtain the index list and alias information.
- the user 102 selects an index, separates multiple index types with a comma, or uses wildcards to query multiple indexes. For example: / index1, index2 / type1, type2 / _search.
- Field selection module set to the table (ie, type) determined by the index type selection module, determine the field to be queried, and splice it in the _source part of the body. That is, it is set to determine the fields included in the index selected from the graphical interface, and multiple fields may be separated by commas. For example: _source: [field 1, field 2].
- Query condition definition module It is set to determine the query condition definition, and the determined specific query condition can be spliced in the query part of the body. That is, it is set to determine the specific query condition based on the field selected by the user 102 from the graphical interface, the specific limit condition selected, and the boundary of the set limit condition.
- the format of the query condition is as follows:
- Aggregation query definition module set to determine the definition of the aggregation query (ie, the above aggregation condition), and the aggregation condition can be spliced in the aggs part of the body. That is, it is set to determine a specific aggregation condition based on the aggregation type, aggregation field, and other related settings selected by the user 102 through the graphical interface.
- the format of the aggregation condition is as follows:
- DSL splicing module It is set to splice the content determined by the above modules to form a complete DSL query statement. For example, an example of a DSL query statement is shown below:
- Grammar verification module set to verify whether the syntax of the DSL sentence generated by the DSL splicing module is correct.
- the server that is, the server-side 103 described above
- the response time of the request increase the timeout amount of the request
- the number of users who visited the website per unit time decreased.
- the original log data stored on the server 103 can be analyzed through the above technical solutions according to the embodiments of the present invention.
- the user 102 may determine the following specific setting parameters (that is, specific parameters involved in the operation instruction) through the GUI.
- remote_addr field set to store the IP address of the user client to be queried and analyzed.
- the user 102 can obtain the UIP number by setting a way to count different IP addresses per unit time obtained by the system such as Elasticsearch from the log.
- status field set to store the status code of the user request to be queried and analyzed.
- the user 102 can select the status field and select a way to divide statistics according to a specific time within a specified time period, and count the number and proportion of status codes of 5XX by time period.
- the request_time field is set to store the total elapsed time of customer requests to be queried and analyzed.
- the user 102 may set a time period for a customer request to be queried and analyzed, and select a method of dividing statistics according to specific moments within the time period, and calculate the average value of the total elapsed time of the customer request by time period.
- the client After receiving the specific settings of the user 102 for the above three fields, the client automatically splices to generate a DSL query statement.
- the user 102 can intuitively check the degree of the website being attacked and the service changes through the results returned by the query.
- the user 102 can also perform threshold setting, threat level setting, and time period setting on the query result through the GUI to obtain specific information about the different threat levels of the website in various periods, so as to further find the high threat level in a targeted manner Specific attack information for threat events that exist during the period.
- FIG. 2 exemplarily shows a schematic flowchart of a visual business data operation method according to an embodiment of the present invention.
- the visual business data operation method includes:
- Step S202c Provide a GUI
- Step S204c Receive a user operation instruction executed by the user in the GUI
- Step S206c Convert user operation instructions into data operation instructions
- Step S208c send the data operation instruction to the server
- Step S210c receiving the data operation result from the server
- Step S212c display the data operation result on the GUI, and / or store the data operation result locally,
- Step S202 receiving a data operation instruction, generating a field structure corresponding to the data operation instruction and / or generating an available operation scheme of the field structure corresponding to the data operation instruction, and executing the data operation instruction;
- Step S204s obtaining data operation results
- Step S206s send the data operation result to the client.
- the user operation instruction includes at least one of the following:
- the conditions for data operations include limiting conditions, query conditions, and aggregation conditions.
- the visual business data operation method further includes:
- the standard data operation statement is Elasticsearch's Query DSL format data operation statement.
- the database and table are the indexes and types of Elasticsearch.
- the server provides Elasticsearch services.
- the Elasticsearch API is used to call the data operation instructions and send the data operation instructions to Service-Terminal.
- the list selection supports regular expression matching.
- the method for operating visual business data according to an embodiment of the present invention further includes:
- Step S214c verify whether the syntax of the data operation statement in Query DSL format is correct.
- the unit modules in the visual business data operation method according to the embodiments of the present invention may be combined, divided, and deleted according to actual needs.
- An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, the steps of the foregoing method are implemented.
- An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor.
- a computer device including a memory, a processor, and a computer program stored on the memory and executable on the processor.
- the processor executes the program, the steps of the above method are implemented.
- Fig. 3 is a block diagram of a device 300 for a method for visualizing business data according to an exemplary embodiment.
- the device 300 may be provided as a server.
- the device 300 includes a processor 301, and the number of processors may be set to one or more as needed.
- the device 300 also includes a memory 302 configured to store instructions executable by the processor 301, such as application programs.
- the number of memories can be one or more according to needs, and there can be one or more application programs stored therein.
- the processor 301 is configured to execute instructions to perform the above method.
- the embodiments of the present application may be provided as methods, devices (equipment), or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer usable storage media containing computer usable program code.
- Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data , Including but not limited to RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or may be used for Any other medium or the like that stores the desired information and can be accessed by the computer.
- the communication medium generally contains computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium .
- These computer program instructions may also be stored in a computer readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including instruction means
- the device implements the functions specified in the flowchart one process or multiple processes and / or the block diagram one block or multiple blocks
- These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operating steps are performed on the computer or other programmable device to produce computer-implemented processing, which is executed on the computer or other programmable device
- the instructions provide steps for implementing the functions specified in one block or multiple blocks of the flowchart one flow or multiple flows and / or block diagrams.
- the visual business data operating system and method in this article can generate complex query statements accurately, flexibly, simply and conveniently in a graphical way. Due to the use of a visual graphical interface configuration, which supports the setting of specific parameters in the query statement, the query statement can be flexibly controlled, which can greatly reduce the work cost of data analysts and developers. Without the help of the developer, the data analyst can automatically generate the corresponding query statement and directly query the data through the configuration on the interface.
- the visual graphical interface in this article can intuitively display the query conditions, no need Learn and master complex query statements, and the results of the query can directly generate txt, xlsx and other format files, and can also add regular execution query conditions for queries that need to be executed regularly, generate files regularly, and can be automatically sent by mail, greatly
- the work efficiency of the data analyst is improved, and the syntax verification of the query statement can be supported to ensure the accuracy of the query statement.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
一种可视化业务数据操作系统及方法,所述系统包括:客户端(101),用于提供GUI,接收用户(102)在GUI执行的用户操作指令,将用户操作指令转换为数据操作指令,将数据操作指令发送至服务器端(103),接收来自服务器端(103)的数据操作结果,在GUI上展示数据操作结果,和/或在本地存储数据操作结果;服务器端(103),用于接收数据操作指令,生成数据操作指令对应的字段结构和/或生成数据操作指令对应的字段结构的可用操作方案,执行数据操作指令,获取数据操作结果,将数据操作结果发送至客户端(101)。该技术方案能够通过图形化的方式,准确、灵活、简单方便地完成业务数据操作。
Description
本申请要求在2018年10月25日提交中国专利局、申请号为201811252404.4,发明名称为“一种可视化业务数据操作系统及方法”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及但不限于互联网和数据库领域,尤其涉及一种可视化业务数据操作系统、方法、设备及介质。
随着移动互联网的普及和大数据时代的发展,大量的数据信息需要处理,由于业务数据都是存储在数据库中,且存储框架层出不穷,多种多样,比如传统的关系型数据库:Oracle、MySQL;新兴的NoSQL:HBase、Cassandra、Redis;全文检索框架:Elasticsearch、Solr等。而且,由于数据库查询需要使用标准的查询语句,非专业人员使用必须熟练掌握灵活度极高且语法复杂的查询语句,而且不同数据库服务的查询语言也不近相同。因此,对企业业务数据进行添加、修改、删除和查询分析操作时,势必导致获取数据效率很低,且极易出错。
另外,查询返回的数据集的结构也各不相同,需要用户对这些数据进行反解析并按照特定格式重新组建成结构化的数据集,存储到文件中,这些工作枯燥且繁重,不仅处理起来效率低下,更重要的是需要操作中及其细心和耐心,稍有疏忽就会出错,给数据处理带来极大不便。
例如,以数据查询操作为例,数据分析师主要对业务数据进行过滤、聚合操作。由于业务的内容不是固定的,数据结构各不相同。每次要分析的问题角度也不尽相同。这就导致分析师每次需要查询的内容也不同。由于各个数据库的查询语法各不相同,且查询语法学习难度大,大多数都是通过命令行界面进行查询,使用起来很不方便,对于不熟悉的人员来说更是无从下手,返回结果形式单一。现有技术在实现具体的数据查询和分析操作时,首先需要根据业务需求提出具体的数据需求。由于业务的复杂度不同,分析师往往要寻求开发人 员的帮助。跟开发人员说明需要的数据内容,开发人员理解后,编写对应的查询语句,再交由数据分析师进行查询、分析。这样加大了数据分析师和开发人员的工作成本,也加大了沟通和时间的成本。
为了解决上述技术问题,需要提出新的技术方案。
发明内容
为克服相关技术中存在的问题,本申请提供一种可视化业务数据操作系统及方法。
根据本申请的一方面,提供一种可视化业务数据操作系统,包括:
客户端,被设置为提供GUI,接收用户在所述GUI执行的用户操作指令,将所述用户操作指令转换为数据操作指令,将所述数据操作指令发送至服务器端,接收来自所述服务器端的数据操作结果,在所述GUI上展示所述数据操作结果,和/或在本地存储所述数据操作结果;
所述服务器端,被设置为接收所述数据操作指令,生成所述数据操作指令对应的字段结构和/或生成所述数据操作指令对应的字段结构的可用操作方案,执行所述数据操作指令,获取所述数据操作结果,将所述数据操作结果发送至所述客户端。
其中,所述用户操作指令包括下列中的至少一项:
由用户输入或通过列表选择的数据源、连接账户和密码;
由用户输入或通过列表选择的数据操作的类型、所涉及的数据库和表、字段信息、数据操作的条件,其中,所述数据操作的条件包括限定条件、查询条件、聚合条件。
其中,所述服务器端还被设置为:
在执行所述数据操作指令之前,拼接所述数据操作指令中的有效数据,生成数据操作标准语句。
其中,所述数据操作标准语句是Elasticsearch的Query DSL格式的数据操作语句,所述数据库和表是Elasticsearch的索引和类型,所述服务器端提供Elasticsearch服务,使用Elasticsearch的API调用所述数据操作指令,将所述数 据操作指令发送至服务器端。
其中,所述列表选择支持正则表达式匹配。
其中,所述客户端还被设置为,验证所述Query DSL格式的数据操作语句的语法是否正确。
根据本申请的另一方面,提供一种可视化业务数据操作方法,包括:
使用客户端来执行以下步骤:
提供GUI;
接收用户在所述GUI执行的用户操作指令;
将所述用户操作指令转换为数据操作指令;
将所述数据操作指令发送至服务器端;
接收来自所述服务器端的数据操作结果;
在所述GUI上展示所述数据操作结果,和/或在本地存储所述数据操作结果,
使用服务器端来执行以下步骤:
接收所述数据操作指令,生成所述数据操作指令对应的字段结构和/或生成所述数据操作指令对应的字段结构的可用操作方案,执行所述数据操作指令;
获取所述数据操作结果;
将所述数据操作结果发送至所述客户端。
其中,所述用户操作指令包括下列中的至少一项:
由用户输入或通过列表选择的数据源、连接账户和密码;
由用户输入或通过列表选择的数据操作的类型、所涉及的数据库和表、字段信息、数据操作的条件,其中,所述数据操作的条件包括限定条件、查询条件、聚合条件。
其中,所述方法还包括:
在执行所述数据操作指令之前,使用服务器端来执行以下步骤:
拼接所述数据操作指令中的有效数据,生成数据操作标准语句。
其中,所述数据操作标准语句是Elasticsearch的Query DSL格式的数据操作 语句,所述数据库和表是Elasticsearch的索引和类型,所述服务器端提供Elasticsearch服务,使用Elasticsearch的API调用所述数据操作指令,将所述数据操作指令发送至服务器端。
其中,所述列表选择支持正则表达式匹配。
其中,所述方法还包括:
使用客户端来执行以下步骤:
验证所述Query DSL格式的数据操作语句的语法是否正确。
根据本申请的另一方面,提供了一种计算机设备,包括存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机程序,所述处理器执行所述程序时实现上述方法的步骤。
根据本申请的另一方面,提供了一种计算机可读存储介质,所述存储介质上存储有计算机程序,所述程序被处理器执行时实现上述方法的步骤。
本申请的可视化业务数据操作系统及方法,可以实现以下有益效果:
1、通过图形化的方式,准确、灵活、简单方便地生成复杂的查询语句。由于采用了可视化图形界面配置,支持对查询语句中的具体参数进行设置,所以能够对查询语句进行灵活控制,能够极大的降低数据分析师和开发人员的工作成本。可以在不通过开发人员的帮助下,由数据分析师自己通过界面上的配置,自动产生对应的查询语句,直接进行数据查询。
2、可视化图形界面可以直观地展示出查询的条件,不需要学习和掌握复杂的查询语句。
3、查询的结果可以直接生成txt、xlsx等格式文件。
4、可以为需要定期执行的查询添加定期执行的查询条件,定时生成文件,并可以通过邮件自动发送,极大地提升了数据分析师的工作效率。
5、支持对查询语句的语法校验,保证查询语句的准确性。
构成本申请的一部分的附图用来提供对本申请的进一步理解,本申请的示 意性实施例及其说明用于解释本申请,并不构成对本申请的不当限定。在附图中:
图1是根据一示例性实施例示出的一种可视化业务数据操作系统的框图;
图2是根据一示例性实施例示出的一种可视化业务数据操作方法的流程图;
图3是根据一示例性实施例示出的一种用于可视化业务数据操作方法的装置的框图。
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。需要说明的是,在不冲突的情况下,本申请中的实施例及实施例中的特征可以相互任意组合。
图1示例性地示出了根据本发明实施例的可视化业务数据操作系统100的示意图。
如图1所示,根据本发明实施例的可视化业务数据操作系统100,包括:
客户端101,被设置为提供GUI,接收用户102在GUI执行的用户操作指令,将用户操作指令转换为数据操作指令,将数据操作指令发送至服务器端,接收来自服务器端的数据操作结果,在GUI上展示数据操作结果,和/或在本地存储数据操作结果;
服务器端103,被设置为接收数据操作指令,生成数据操作指令对应的字段结构和/或生成数据操作指令对应的字段结构的可用操作方案,执行数据操作指令,获取数据操作结果,将数据操作结果发送至客户端101。
可选地,用户操作指令包括下列中的至少一项:
由用户102输入或通过列表选择的数据源、连接账户和密码;
由用户102输入或通过列表选择的数据操作的类型、所涉及的数据库和表、字段信息、数据操作的条件,
其中,数据操作的条件包括限定条件、查询条件、聚合条件。
可选地,服务器端103还被设置为:
在执行数据操作指令之前,拼接数据操作指令中的有效数据,生成数据操作标准语句。
可选地,数据操作标准语句是Elasticsearch的Query DSL格式的数据操作语句,数据库和表是Elasticsearch的索引和类型,服务器端103提供Elasticsearch服务,使用Elasticsearch的API调用数据操作指令,将数据操作指令发送至服务器端103。
可选地,列表选择支持正则表达式匹配。
可选地,客户端101还被设置为,验证Query DSL格式的数据操作语句的语法是否正确。
例如,用户可以使用以下方式来输入具体的用户操作指令:
用户可以在界面上选择数据存储框架(例如,Elasticsearch),配置好连接数据源所需要的地址、端口、身份认证信息。
配置成功后,可通过以下方法再界面上显示数据源中所存储的全部数据,清晰的展示出数据的种类、结构等信息,以供用户执行诸如从列表中选择等后续的交互操作:
可以通过对数据源的扫描,在界面上展示出数据的字段结构。可以预先确定各个类型的字段所适用的操作。可以对用户选择的字段进行实时监听,展示出用户可以选择的方案。用户只要选择对应的操作,即可完成对所选数据的分析设置。可以通过用户的配置生成用户指定的数据存储框架所需要的查询语句(例如,上述Query DSL格式的数据操作语句)。可以保存每个数据配置的详细信息,该详细信息包括数据的来源(例如,上述数据源、数据库和表)、所要查询的字段(即,上述字段信息)、对每个字段执行的具体操作(例如,上述数据操作的类型——增、删、查、改,数据操作的条件)等。
为了使本领域技术人员更清楚地理解根据本发明实施例的上述技术方案,下面将以生成上文所述的Elasticsearch的Query DSL格式的数据操作语句为例进行说明。
1、客户端101接收用户102输入或选择的、待查询的索引和类型,索引和类型相当于标准DSL查询中的URL部分。索引和类型可以是一个,也可以是多个。可以通过通配符匹配对应多个索引和类型(即,上述支持正则表达式匹配)。
2、客户端101接收用户102输入或选择的、待查询的字段,这些字段相当于标准DSL查询Body中_source的部分。即,用户102可以基于上一步已经选择的索引信息,选择该索引中的字段信息。例如,用户102可以通过多选的方式直接选择列表控件中显示的多个字段。如果用户不进行选择,则默认查询全部字段。
3、客户端101接收用户102输入或选择的、具体查询条件,这些查询条件相当于标准DSL查询Body中query的部分。例如,可以采用bool复合查询子句,可以支持must、must_not、should、filter等。例如,用户102可以通过界面对任意字段添加限定条件,限定条件的类型包括term、wildcard、perfix、fuzzy、range、missing等。
4、客户端101接收用户102输入或选择的、具体(例如,适用于类型为查询的数据操作的)聚合条件,这些聚合条件相当于标准DSL查询Body中aggs的部分。例如,用户102可以通过界面直接选择聚合条件的类型,并设置具体的聚合条件,聚合条件可以重复嵌套。
此时,客户端101可以包括用于生成标准DSL语句的语句生成装置,该语句生成装置包括以下模块:
索引类型选择模块:被设置为生成DSL查询的URL部分。即,被设置为在用户102使用界面设置了Elasticsearch地址,获得索引列表和别名信息之后。用户102通过选择索引,将多个索引类型用逗号分开,或者采用通配符方式查询多个索引。例如:/索引1、索引2/类型1、类型2/_search。
字段选择模块:被设置为基于索引类型选择模块所确定的表(即,类型),确定要查询的字段,拼接在body中的_source部分。即,被设置为确定从图形界面选择的索引中包含的字段,多个字段可以用逗号隔开。例如:_source:[字段1,字段2]。
查询条件定义模块:被设置为确定查询的条件定义,可以将所确定的具体 查询条件拼接在body中的query部分。即,被设置为基于用户102从图形界面上所选择的字段、所选择的具体限定条件、所设置的限定条件的边界,来确定具体查询条件。例如,查询条件的格式如下文所示:
聚合查询定义模块:被设置为确定聚合查询的定义(即,上述聚合条件),聚合条件可以拼接在body中的aggs部分。即,被设置为基于用户102通过图形界面所选择的聚合类型、聚合字段和其他相关设置,来确定具体的聚合条件。例如,聚合条件的格式如下文所示:
DSL拼接模块:被设置为将上述模块所确定的内容拼接起来,形成一个完整的DSL查询语句。例如,DSL查询语句的一个示例如下文所示:
GET/索引1,索引2/类型1,类型2/_search
语法校验模块:被设置为验证DSL拼接模块所生成的DSL语句的语法是否正确。
例如,当一个网站被攻击时,会导致服务器(即,上述服务器端103)压力增大,请求响应时间增长,请求超时量增大,会产生大量的5XX状态。单位时间内访问网站的用户数量下降。在这种情况下,就能够通过根据本发明实施例的上述技术方案来对服务器端103上所存储的原始日志数据进行分析。例如,此时用户102可以通过GUI来确定以下具体设置参数(即,操作指令所涉及的具体参数)。
remote_addr字段:被设置为存储待查询和分析的用户客户端的IP地址。
例如,用户102可以通过设置对诸如Elasticsearch的系统从日志中获取到的单位时间内的不同IP地址进行统计的方式,来获得UIP数。
status字段:被设置为存储待查询和分析的用户请求的状态码。
例如,用户102可以通过选择该status字段、并且选择在指定时间段内按照具体时刻进行分割统计的方式,分时段统计状态码为5XX的数量和占比。
request_time字段:被设置为存储待查询和分析的客户请求的总用时。
例如,用户102可以设置待查询和分析的客户请求的时间段,在该时间段内选择按照具体时刻进行分割统计的方式,分时段统计客户请求的总用时的平均值。
客户端在接收到用户102针对上述三个字段的具体设置之后,会自动拼接生成DSL查询语句,用户102可以通过查询返回的结果直观地查看网站被攻击的程度,以及服务的变化。
可选地,用户102还可以通过GUI对查询结果进行阈值设置、威胁等级设置、时间段设置,以获取网站在各个时段的不同威胁等级的具体信息,从而可以进一步有针对性地查找高威胁等级的时段中存在的威胁事件的具体攻击信息。
图2示例性地示出了根据本发明实施例的可视化业务数据操作方法的示意流程图。
如图2(a)和图2(b)的实线框所示,根据本发明实施例的可视化业务数据操作方法,包括:
使用客户端来执行以下步骤:
步骤S202c:提供GUI;
步骤S204c:接收用户在GUI执行的用户操作指令;
步骤S206c:将用户操作指令转换为数据操作指令;
步骤S208c:将数据操作指令发送至服务器端;
步骤S210c:接收来自服务器端的数据操作结果;
步骤S212c:在GUI上展示数据操作结果,和/或在本地存储数据操作结果,
使用服务器端来执行以下步骤:
步骤S202s:接收数据操作指令,生成数据操作指令对应的字段结构和/或生成数据操作指令对应的字段结构的可用操作方案,执行数据操作指令;
步骤S204s:获取数据操作结果;
步骤S206s:将数据操作结果发送至客户端。
可选地,用户操作指令包括下列中的至少一项:
由用户输入或通过列表选择的数据源、连接账户和密码;
由用户输入或通过列表选择的数据操作的类型、所涉及的数据库和表、字段信息、数据操作的条件,
其中,数据操作的条件包括限定条件、查询条件、聚合条件。
可选地,根据本发明实施例的可视化业务数据操作方法,还包括:
在执行数据操作指令之前,使用服务器端来执行以下步骤:
拼接数据操作指令中的有效数据,生成数据操作标准语句。
可选地,数据操作标准语句是Elasticsearch的Query DSL格式的数据操作语句,数据库和表是Elasticsearch的索引和类型,服务器端提供Elasticsearch服务,使用Elasticsearch的API调用数据操作指令,将数据操作指令发送至服务器端。
可选地,列表选择支持正则表达式匹配。
可选地,如图2(a)的虚线框所示,根据本发明实施例的可视化业务数据操作方法,还包括:
使用客户端来执行以下步骤:
步骤S214c:验证Query DSL格式的数据操作语句的语法是否正确。
本发明实施例的方法中的步骤可以根据实际需要进行顺序调整、合并和删减。
本发明实施例的可视化业务数据操作方法中的单元模块可以根据实际需要进行合并、划分和删减。
本发明实施例还提供了一种计算机可读存储介质,此存储介质上存储有计算机程序,所述程序被处理器执行时实现上述方法的步骤。
本发明实施例还提供了一种计算机设备,包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序,处理器执行此程序时实现上述方法的步骤。
图3是根据一示例性实施例示出的一种用于可视化业务数据操作方法的装置300的框图。例如,装置300可以被提供为一服务器。参照图3,装置300包括处理器301,处理器的个数可以根据需要设置为一个或者多个。装置300还包括存储器302,被设置为存储可由处理器301执行的指令,例如应用程序。存储器的个数可以根据需要设置一个或者多个,其存储的应用程序可以为一个或者多个。处理器301被设置为执行指令,以执行上述方法。
本领域技术人员应明白,本申请的实施例可提供为方法、装置(设备)、或计算机程序产品。因此,本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且,本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质上实施的计算机程序产品的形式。计算机存储介质包括在用于存储信息(诸如计算机可读指令、数据结构、程序模块或其他数据)的任何方法或技术中实施的易失性和非易失性、可移除和不可移除介质,包括但不限于RAM、ROM、EEPROM、闪存或其他存储器技术、CD-ROM、数字多功能盘(DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访 问的任何其他的介质等。此外,本领域普通技术人员公知的是,通信介质通常包含计算机可读指令、数据结构、程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据,并且可包括任何信息递送介质。
本申请是参照根据本发明实施例的方法、装置(设备)和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器,使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中,使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品,该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上,使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理,从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
在本文中,术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含,从而使得包括一系列要素的物品或者设备不仅包括那些要素,而且还包括没有明确列出的其他要素,或者是还包括为这种物品或者设备所固有的要素。在没有更多限制的情况下,由语句“包括……”限定的要素,并不排除在包括所述要素的物品或者设备中还存在另外的相同要素。
尽管已描述了本申请的优选实施例,但本领域内的技术人员一旦得知了基本创造性概念,则可对这些实施例作出另外的变更和修改。所以,所附权利要求意欲解释为包括优选实施例以及落入本申请范围的所有变更和修改。
显然,本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的精神和范围。这样,倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内,则本申请的意图也包含这些改动和变型在内。
本文中的可视化业务数据操作系统及方法通过图形化的方式,准确、灵活、简单方便地生成复杂的查询语句。由于采用了可视化图形界面配置,支持对查询语句中的具体参数进行设置,所以能够对查询语句进行灵活控制,能够极大的降低数据分析师和开发人员的工作成本。可以在不通过开发人员的帮助下,由数据分析师自己通过界面上的配置,自动产生对应的查询语句,直接进行数据查询,本文中的可视化图形界面可以直观地展示出查询的条件,不需要学习和掌握复杂的查询语句,并且查询的结果可以直接生成txt、xlsx等格式文件,还可以为需要定期执行的查询添加定期执行的查询条件,定时生成文件,并可以通过邮件自动发送,极大地提升了数据分析师的工作效率,能够支持对查询语句的语法校验,保证查询语句的准确性。
Claims (14)
- 一种可视化业务数据操作系统,包括:客户端,被设置为提供GUI,接收用户在所述GUI执行的用户操作指令,将所述用户操作指令转换为数据操作指令,将所述数据操作指令发送至服务器端,接收来自所述服务器端的数据操作结果,在所述GUI上展示所述数据操作结果,和/或在本地存储所述数据操作结果;所述服务器端,被设置为接收所述数据操作指令,生成所述数据操作指令对应的字段结构和/或生成所述数据操作指令对应的字段结构的可用操作方案,执行所述数据操作指令,获取所述数据操作结果,将所述数据操作结果发送至所述客户端。
- 如权利要求1所述的可视化业务数据操作系统,其中,所述用户操作指令包括下列中的至少一项:由用户输入或通过列表选择的数据源、连接账户和密码;由用户输入或通过列表选择的数据操作的类型、所涉及的数据库和表、字段信息、数据操作的条件,其中,所述数据操作的条件包括限定条件、查询条件、聚合条件。
- 如权利要求1所述的可视化业务数据操作系统,其中,所述服务器端还被设置为:在执行所述数据操作指令之前,拼接所述数据操作指令中的有效数据,生成数据操作标准语句。
- 如权利要求3所述的可视化业务数据操作系统,其中,所述数据操作标准语句是Elasticsearch的Query DSL格式的数据操作语句,所述数据库和表是Elasticsearch的索引和类型,所述服务器端提供Elasticsearch服务,使用Elasticsearch的API调用所述数据操作指令,将所述数据操作指令发送至服务器端。
- 如权利要求2所述的可视化业务数据操作系统,其中,所述列表选择支持正则表达式匹配。
- 如权利要求4所述的可视化业务数据操作系统,其中,所述客户端还被设置为,验证所述Query DSL格式的数据操作语句的语法是否正确。
- 一种可视化业务数据操作方法,包括:使用客户端来执行以下步骤:提供GUI;接收用户在所述GUI执行的用户操作指令;将所述用户操作指令转换为数据操作指令;将所述数据操作指令发送至服务器端;接收来自所述服务器端的数据操作结果;在所述GUI上展示所述数据操作结果,和/或在本地存储所述数据操作结果,使用服务器端来执行以下步骤:接收所述数据操作指令,生成所述数据操作指令对应的字段结构和/或生成所述数据操作指令对应的字段结构的可用操作方案,执行所述数据操作指令;获取所述数据操作结果;将所述数据操作结果发送至所述客户端。
- 如权利要求7所述的可视化业务数据操作方法,其中,所述用户操作指令包括下列中的至少一项:由用户输入或通过列表选择的数据源、连接账户和密码;由用户输入或通过列表选择的数据操作的类型、所涉及的数据库和表、字段信息、数据操作的条件,其中,所述数据操作的条件包括限定条件、查询条件、聚合条件。
- 如权利要求7所述的可视化业务数据操作方法,还包括:在执行所述数据操作指令之前,使用服务器端来执行以下步骤:拼接所述数据操作指令中的有效数据,生成数据操作标准语句。
- 如权利要求9所述的可视化业务数据操作方法,其中,所述数据操作标准语句是Elasticsearch的Query DSL格式的数据操作语句,所述数据库和表是 Elasticsearch的索引和类型,所述服务器端提供Elasticsearch服务,使用Elasticsearch的API调用所述数据操作指令,将所述数据操作指令发送至服务器端。
- 如权利要求8所述的可视化业务数据操作方法,其中,所述列表选择支持正则表达式匹配。
- 如权利要求9所述的可视化业务数据操作方法,还包括:使用客户端来执行以下步骤:验证所述Query DSL格式的数据操作语句的语法是否正确。
- 一种计算机可读存储介质,其上存储有计算机程序,其特征在于,所述计算机程序被执行时实现如权利要求7-12中任意一项所述方法的步骤。
- 一种计算机设备,包括处理器、存储器和存储于所述存储器上的计算机程序,其特征在于,所述处理器执行所述计算机程序时实现如权利要求7-12中任意一项所述方法的步骤。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811252404.4 | 2018-10-25 | ||
CN201811252404.4A CN111190916A (zh) | 2018-10-25 | 2018-10-25 | 一种可视化业务数据操作系统及方法 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020083358A1 true WO2020083358A1 (zh) | 2020-04-30 |
Family
ID=70330275
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/113197 WO2020083358A1 (zh) | 2018-10-25 | 2019-10-25 | 一种可视化业务数据操作系统、方法、设备及介质 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN111190916A (zh) |
WO (1) | WO2020083358A1 (zh) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113127596A (zh) * | 2021-05-08 | 2021-07-16 | 北京明略昭辉科技有限公司 | 一种全文检索方法、系统、电子设备及存储介质 |
CN113448568A (zh) * | 2021-06-24 | 2021-09-28 | 未鲲(上海)科技服务有限公司 | 服务编排调用方法、装置、计算机设备及存储介质 |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7143107B1 (en) * | 2003-06-26 | 2006-11-28 | Microsoft Corporation | Reporting engine for data warehouse |
CN102163223A (zh) * | 2011-04-02 | 2011-08-24 | 锐词文化发展(北京)有限公司 | 一种后台数据自助式查询方法和装置 |
CN103279509A (zh) * | 2013-05-17 | 2013-09-04 | 北京首钢自动化信息技术有限公司 | 一种采用动态查询语言的快速查询方法 |
US20140019485A1 (en) * | 2002-07-02 | 2014-01-16 | American Express Travel Related Services Company, Inc. | System and method for data capture and reporting |
CN104516864A (zh) * | 2013-09-30 | 2015-04-15 | 北大方正集团有限公司 | 一种报表生成方法及装置 |
CN105843945A (zh) * | 2016-04-08 | 2016-08-10 | 联动优势科技有限公司 | 一种报表生成方法及系统 |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7558799B2 (en) * | 2004-06-01 | 2009-07-07 | Microsoft Corporation | Method, system, and apparatus for discovering and connecting to data sources |
CN102033951B (zh) * | 2010-12-15 | 2012-09-05 | 北京新媒传信科技有限公司 | Web数据库平台 |
CN106325856B (zh) * | 2016-08-10 | 2019-08-20 | 杭州玳数科技有限公司 | 一种实现Elasticsearch Dsl规则可视化编辑和数据展现的方法与系统 |
CN108153808A (zh) * | 2017-11-22 | 2018-06-12 | 链家网(北京)科技有限公司 | 大数据数据仓库数据访问方法及装置 |
-
2018
- 2018-10-25 CN CN201811252404.4A patent/CN111190916A/zh active Pending
-
2019
- 2019-10-25 WO PCT/CN2019/113197 patent/WO2020083358A1/zh active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140019485A1 (en) * | 2002-07-02 | 2014-01-16 | American Express Travel Related Services Company, Inc. | System and method for data capture and reporting |
US7143107B1 (en) * | 2003-06-26 | 2006-11-28 | Microsoft Corporation | Reporting engine for data warehouse |
CN102163223A (zh) * | 2011-04-02 | 2011-08-24 | 锐词文化发展(北京)有限公司 | 一种后台数据自助式查询方法和装置 |
CN103279509A (zh) * | 2013-05-17 | 2013-09-04 | 北京首钢自动化信息技术有限公司 | 一种采用动态查询语言的快速查询方法 |
CN104516864A (zh) * | 2013-09-30 | 2015-04-15 | 北大方正集团有限公司 | 一种报表生成方法及装置 |
CN105843945A (zh) * | 2016-04-08 | 2016-08-10 | 联动优势科技有限公司 | 一种报表生成方法及系统 |
Non-Patent Citations (1)
Title |
---|
WFS1994: "Elasticsearch Query DSL", 17 May 2018 (2018-05-17), XP055707998, Retrieved from the Internet <URL:https://blog.csdn.net/wfs1994/article/details/80351590> * |
Also Published As
Publication number | Publication date |
---|---|
CN111190916A (zh) | 2020-05-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11275730B2 (en) | Automated data analysis using combined queries | |
US11461545B2 (en) | Grid data management | |
US10409801B2 (en) | Validation of web-based database updates | |
CN106687955B (zh) | 简化将数据从数据源转移到数据目标的导入过程的调用 | |
US9507751B2 (en) | Managing seed data | |
US10261808B2 (en) | Access operation with dynamic linking and access of data within plural data sources | |
EP3887969A1 (en) | Systems and methods for storing object state on hash chains | |
US9330140B1 (en) | Transient virtual single tenant queries in a multi-tenant shared database system | |
CN115017182A (zh) | 一种可视化的数据分析方法及设备 | |
WO2020083358A1 (zh) | 一种可视化业务数据操作系统、方法、设备及介质 | |
CN113485927A (zh) | 一种测试数据生成方法、装置、设备及存储介质 | |
US10942732B1 (en) | Integration test framework | |
US11824837B2 (en) | End user creation of trusted integration pathways between different enterprise systems | |
US10534588B2 (en) | Data processing simulator with simulator module and data elements | |
US10698794B1 (en) | Application container and application service system | |
US10984119B2 (en) | Simplifying data protection in CDS based access | |
US9721041B2 (en) | Configurable data analysis using a configuration model | |
US20240126725A1 (en) | System and method for integrated function editor for use in data analytics environment | |
US20230081212A1 (en) | System and method for providing multi-hub datasets for use with data analytics environments | |
US12039069B2 (en) | Metadata-driven restricted measures | |
US20240126775A1 (en) | System and method for automatically enriching datasets with system knowledge data | |
US20240193122A1 (en) | Dynamic loading of files using asynchronous format | |
US20220147568A1 (en) | Mapping expression generator | |
US20160292210A1 (en) | System and method for automatically and efficiently validating database objects | |
US10372614B1 (en) | Smart data engine system and method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19876321 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 16/06/2021) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19876321 Country of ref document: EP Kind code of ref document: A1 |