WO2021232292A1

WO2021232292A1 - Log data processing method and related product

Info

Publication number: WO2021232292A1
Application number: PCT/CN2020/091319
Authority: WO
Inventors: 郭子亮
Original assignee: 深圳市欢太科技有限公司; Oppo广东移动通信有限公司
Priority date: 2020-05-20
Filing date: 2020-05-20
Publication date: 2021-11-25
Also published as: CN115380282A

Abstract

A log data processing method and a related product, which are applied to a log service system. The log service system comprises a log search engine and a log warehouse service. The method comprises: a log search engine acquiring target log data from a log warehouse service according to a preset time sequence (101); the log search engine performing keyword search and matching on the target log data by means of a preset log search algorithm, so as to obtain a target search result (102); and the log search engine outputting the target search result (103). By using the method, memory resources can be saved on.

Description

Log data processing method and related products

Technical field

This application relates to the computer field, and specifically relates to a log data processing method and related products.

Background technique

At present, mainstream Internet vendors generally adopt ELK solutions commonly used in the industry for the collection, storage, transmission, search query, and real-time analysis of massive log data. The implementation process of this solution is to use filebeat as the log collection terminal, collect the log data generated by the business, and transmit it to logstash through kafka. Logstash customizes and trims the log data according to the user's pre-customized query field requirements, and then writes it The search engine ES, ES is responsible for log storage and search queries, and graphically displays the log data required by users through the kibana component. However, for the search engine ES service, it not only needs to achieve log storage, but also needs to establish a diversified Indexes, therefore, need to consume a lot of memory resources.

Summary of the invention

The embodiments of the present application provide a log data processing method and related products, which can save memory resources.

In a first aspect, a log data processing method according to an embodiment of the present application is applied to a log service system, the log service system includes a log search engine and a log warehouse service, and the method includes:

The log search engine obtains target log data from the log warehouse service according to a preset time sequence;

The log search engine performs keyword search and matching on the target log data by using a preset log search algorithm to obtain target search results;

The log search engine outputs the target search result.

In a second aspect, an embodiment of the present application provides a log data processing device, which is applied to a log service system, the log service system includes a log search engine and a log warehouse service, and the device includes: an acquisition unit, a search unit, and an output unit ,in,

The obtaining unit is configured to obtain target log data from the log warehouse service according to a preset time sequence through the log search engine;

The search unit is configured to perform keyword search and matching on the target log data through the log search engine through a preset log search algorithm to obtain target search results;

The output unit is configured to output the target search result through the log search engine.

In a third aspect, an embodiment of the present application provides a server including a processor, a memory, a communication interface, and one or more programs. The server includes a log service system, and the log service system includes a log search engine and a log warehouse. The service is configured to be executed by the above-mentioned processor, and the above-mentioned program includes instructions for executing the steps in the first aspect of the embodiments of the present application.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the above-mentioned computer-readable storage medium stores a computer program for electronic data exchange, wherein the above-mentioned computer program enables a computer to execute Some or all of the steps described in one aspect.

In a fifth aspect, the embodiments of the present application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute as implemented in this application. Example part or all of the steps described in the first aspect. The computer program product may be a software installation package.

Description of the drawings

The following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art.

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1A is a schematic structural diagram of a server provided by an embodiment of the present application;

FIG. 1B is a schematic diagram of the architecture for implementing the log data processing method provided by the embodiment of the present application;

FIG. 1C is a schematic flowchart of a log data processing method disclosed in an embodiment of the present application;

FIG. 1D is a schematic diagram of a demonstration of obtaining log data disclosed in an embodiment of the present application;

2 is a schematic flowchart of another log data processing method disclosed in an embodiment of the present application;

Figure 3 is a schematic structural diagram of another server disclosed in an embodiment of the present application;

Fig. 4 is a schematic structural diagram of a log data processing device disclosed in an embodiment of the present application.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the application, the technical solutions in the embodiments of the application will be clearly and completely described below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only It is a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific sequence. In addition, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes unlisted steps or units, or optionally also includes Other steps or units inherent to these processes, methods, products or equipment.

The reference to "embodiments" herein means that a specific feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art clearly and implicitly understand that the embodiments described herein can be combined with other embodiments.

The log service system involved in the embodiment of the present application is applied to a cloud platform or a cloud server.

The following describes the embodiments of the application in detail.

Please refer to FIG. 1A. FIG. 1A is a schematic structural diagram of a server disclosed in an embodiment of the present application. The server 100 may include a control circuit, and the control circuit may include a storage and processing circuit 110. The storage and processing circuit 110 can be a memory, such as a hard disk drive memory, a non-volatile memory (such as flash memory or other electronic programmable read-only memory used to form a solid-state drive, etc.), and a volatile memory (such as static or dynamic random access memory). Access to memory, etc.), etc., are not limited in the embodiment of the present application. The processing circuit in the storage and processing circuit 110 can be used to control the operation of the server 100. The processing circuit can be implemented based on one or more microprocessors, microcontrollers, baseband processors, power management units, audio codec chips, application specific integrated circuits, display driver integrated circuits, and so on.

The storage and processing circuit 110 can be used to run software in the server 100, such as Internet browsing applications, voice over internet protocol (VOIP) phone call applications, email applications, media playback applications, operating system functions, etc. . These softwares can be used to perform some control operations, for example, camera-based image capture, ambient light measurement based on ambient light sensors, proximity sensor measurement based on proximity sensors, and information based on status indicators such as the status indicators of light-emitting diodes. Display functions, touch event detection based on touch sensors, functions associated with displaying information on multiple (for example, layered) displays, operations associated with performing wireless communication functions, operations associated with collecting and generating audio signals , The control operations associated with the collection and processing of button press event data, as well as other functions in the server 100, etc., are not limited in the embodiment of the present application.

The server 100 may also include an input-output circuit 150. The input-output circuit 150 can be used to enable the server 100 to implement data input and output, that is, to allow the server 100 to receive data from an external device and also allow the server 100 to output data from the server 100 to an external device. The input-output circuit 150 may further include a sensor 170. The sensor 170 may include an ambient light sensor, a proximity sensor based on light and capacitance, and a touch sensor (for example, a light-based touch sensor and/or a capacitive touch sensor. The touch sensor structure is used independently), acceleration sensor, gravity sensor, and other sensors.

The input-output circuit 150 may also include one or more displays, such as the display 130. The display 130 may include one or a combination of a liquid crystal display, an organic light emitting diode display, an electronic ink display, a plasma display, and a display using other display technologies. The display 130 may include a touch sensor array (ie, the display 130 may be a touch display screen). The touch sensor can be a capacitive touch sensor formed by an array of transparent touch sensor electrodes (such as indium tin oxide (ITO) electrodes), or it can be a touch sensor formed using other touch technologies, such as sonic touch, pressure-sensitive touch, and resistance. Touch, optical touch, etc., are not limited in the embodiment of the present application.

The audio component 140 may be used to provide audio input and output functions for the server 100. The audio component 140 in the server 100 may include a speaker, a microphone, a buzzer, a tone generator, and other components for generating and detecting sounds.

The communication circuit 120 may be used to provide the server 100 with the ability to communicate with external devices. The communication circuit 120 may include analog and digital input-output interface circuits, and wireless communication circuits based on radio frequency signals and/or optical signals. The wireless communication circuit in the communication circuit 120 may include a radio frequency transceiver circuit, a power amplifier circuit, a low noise amplifier, a switch, a filter, and an antenna. For example, the wireless communication circuit in the communication circuit 120 may include a circuit for supporting near field communication (NFC) by transmitting and receiving near-field coupled electromagnetic signals. For example, the communication circuit 120 may include a near field communication antenna and a near field communication transceiver. The communication circuit 120 may also include a cellular phone transceiver and antenna, a wireless local area network transceiver circuit and antenna, and so on.

The server 100 may further include a battery, a power management circuit, and other input-output units 160. The input-output unit 160 may include buttons, joysticks, click wheels, scroll wheels, touch pads, keypads, keyboards, cameras, light emitting diodes, and other status indicators.

The user can control the operation of the server 100 by inputting commands through the input-output circuit 150, and can use the output data of the input-output circuit 150 to realize receiving status information and other output from the server 100.

Further, please refer to FIG. 1B. FIG. 1B provides a system architecture that implements the method involved in the embodiment of this application, that is, the log service system. The method and the log service system described in the embodiment of this application can be applied to the server, and the log service The system may include a log warehouse service (LogHouse) and at least one log server engine (Logengine). The log service system may also include a receiving end. The receiving end may be at least one of the following: spark, flink, kfaka, client, etc., here Not limited.

In the embodiments of this application, Logengine, as the log query search engine module component of LogHouse, provides users with high-efficiency, low-energy and rich and diverse log data search, query and analysis functions and interfaces, such as supporting SQL-like queries and streaming docking The big data component analyzes log data in real time, supports keyword query matching, supports regular expression query filtering, supports real-time log tail, supports historical time period log query, supports log download, and so on.

In specific implementation, the Logengine architecture and design can be a high-performance stateless and linearly scalable service component, which is completely transparent to users, only exposes grpc and http interfaces for users to use, and realizes docking with multiple big data services such as spark, flink, kfaka Streaming log data for real-time analysis, statistics, search and filtering.

Based on the above log service system, the following operations can be performed:

The log search engine outputs the target search result.

It can be seen that the log data processing method described in the embodiment of this application is applied to a log service system. The log service system includes a log search engine and a log warehouse service. The log search engine obtains target log data from the log warehouse service according to a preset timing. The log search engine performs keyword search and matching on the target log data through the preset log search algorithm, and obtains the target search results. The log search engine outputs the target search results. In this way, the log search engine does not store any original log data, but from the log warehouse The service obtains log data, and performs real-time search and analysis of logs, which can save memory resources.

Please refer to FIG. 1C. FIG. 1C is a schematic flowchart of a log data processing method provided by an embodiment of the present application. The log data processing method described in this embodiment is applied to the server shown in FIG. 1A or the system shown in FIG. 1B Architecture, the server may include a log service system, the log service system includes a log search engine and a log warehouse service, and the log data processing method includes:

101. The log search engine obtains target log data from the log warehouse service according to a preset time sequence.

The preset time sequence can be set by the user or the system defaults. For example, the preset time sequence is a preset time interval, or the preset time sequence can be implemented by a pulse function.

In specific implementation, the log warehouse service can be used to save log data, while the log search engine may not need to save the log data. Furthermore, the log search engine can obtain target log data from the log warehouse service according to a preset timing.

In a possible example, the storage mode of the log warehouse service for the target log data is KV storage.

Among them, the storage method of the log warehouse service for the target log data is KV storage. KV storage can realize distributed storage and save memory resources to a certain extent.

In a possible example, the log search engine is only used to implement the log stream analysis function, and is not used to store data.

Among them, in the embodiment of the present application, the log search engine may only be used to implement the log analysis function, and is not used to store data. In this way, the data storage function and the analysis function can be separated, and the log analysis efficiency can be fully improved.

In a possible example, the key value stored by the KV is generated from the time dimension and log sequence, the V of the KV stores original log data, and the target log data is part of the log data of the original log data.

In specific implementation, the key value (key) stored by KV is generated by the time dimension and log sequence. In this way, the key value can be generated naturally and orderly, and the V of KV stores the original log data, which is convenient for subsequent time dimension cutting to obtain Data stream, the target log data is part of the log data of the original log data.

In a possible example, in step 101, the log search engine obtains target log data from the log warehouse according to a preset time sequence, which may include the following steps:

11. The log search engine cuts the original log data according to the time dimension to obtain cut log data;

12. Process the cutting log data into streaming log data to obtain the target log data.

Among them, in specific implementation, the log search engine can cut the original log data according to the time dimension to obtain the cut log data, and further, can process the cut log data into streaming log data to obtain the target log data. For example, as shown in Figure 1D, suppose the user needs to search for log data containing the hello keyword in the time range with the start time of T1 and the end time of T5, then T1-T5 can be divided into multiple time periods, each The time period corresponds to a log data stream, and each log data stream can correspond to a search thread.

102. The log search engine performs keyword search and matching on the target log data by using a preset log search algorithm to obtain target search results.

Among them, the preset log search algorithm can be set by the user or the system defaults. The preset log search algorithm can be at least one of the following: kmp algorithm, naive algorithm, neural network algorithm, etc., which are not limited here. The neural network algorithm can be It is at least one of the following: a fully connected neural network model, a convolutional neural network model, a spiking neural network model, a cyclic neural network model, etc., which are not limited here.

In specific implementation, the log search engine can perform keyword search and matching on target log data through a preset log search algorithm to obtain target search results, which can improve search efficiency to a certain extent.

In a possible example, in step 102, the log search engine performs keyword search and matching on the target log data through a preset log search algorithm to obtain target search results, which may include the following steps:

21. The log search engine performs keyword extraction on the target log data to obtain the first keyword;

22. Obtain pre-stored library data;

23. Perform keyword extraction on the database data to obtain multiple second keywords;

24. Use the preset log search algorithm to match the first keyword with each second keyword of the plurality of second keywords to obtain a plurality of matching values, and the plurality of second keywords Each second keyword in the word corresponds to a match value;

25. Select the maximum value from the multiple matching values, and use the content corresponding to the second keyword corresponding to the maximum value as the target search result.

Among them, in a specific implementation, the log search engine may extract keywords from the target log data to obtain the first keyword. The keywords may be at least one of the following: pictures, characters, and so on. The log warehouse service can store library data in advance, and then, the log search engine can obtain the pre-stored library data, extract keywords from the library data, and obtain multiple second keywords. The first key can be obtained through the preset log search algorithm The word is matched with each second keyword in the plurality of second keywords to obtain a plurality of matching values. Each second keyword in the plurality of second keywords corresponds to a matching value. Furthermore, it is possible to obtain multiple matching values from The maximum value is selected from the matching values, and the content corresponding to the second keyword corresponding to the maximum value is used as the target search result.

Further, in a possible example, in the above step 24, the first keyword is matched with each second keyword of the plurality of second keywords through the preset log search algorithm to obtain Multiple matching values can include the following steps:

241. Determine first attribute information of the first keyword;

242. Determine second attribute information of each keyword in the plurality of second keywords to obtain a plurality of second attribute information.

243. Match the first attribute information with the multiple second attribute information to obtain multiple attribute matching values, and each second attribute information corresponds to an attribute matching value.

244. Filter out the attribute matching values that meet the preset requirements from the multiple attribute matching values, obtain multiple target attribute matching values, obtain keywords corresponding to the multiple target attribute matching values, and obtain multiple target second Keywords;

245. Use the preset log search algorithm to match each of the first keyword and the multiple target second keywords to obtain the multiple matching values.

In specific implementation, both the first attribute information and the second attribute information may include at least one of the following: information type, keyword length, keyword location, etc., which are not limited herein. The information type can be at least one of the following: pictures, punctuation marks, log types, etc., which are not limited here.

In a specific implementation, the log search engine can determine the first attribute information of the first keyword, and at the same time, can also determine the second attribute information of each of the multiple second keywords to obtain multiple second attribute information. The above preset requirements can be set by the user or the system defaults. For example, the preset requirement is that the matching value is in a certain range. Further, the first attribute information can be matched with multiple second attribute information to obtain multiple attribute matches. Value, each second attribute information corresponds to an attribute matching value, which is equivalent to an initial matching process. Then, attribute matching values that meet the preset requirements can be filtered from multiple attribute matching values to obtain multiple target attribute matching values , Obtain keywords corresponding to multiple target attribute matching values, obtain multiple target second keywords, and match each of the first keyword with multiple target second keywords through a preset log search algorithm, Obtain multiple matching values. In this way, the log search and matching efficiency can be improved.

103. The log search engine outputs the target search result.

In specific implementation, the log search engine can output the target search results to the receiving end, and the receiving end can receive the target search results.

In a possible example, in the foregoing step 103, the log search engine outputs the target search result, which can be implemented in the following manner:

The log search engine outputs the target search results in a streaming manner.

In specific implementation, the log search engine can output the target search results in a streaming manner, so that an orderly and efficient output of the search results can be achieved.

Further, in a possible example, in the above steps, the log search engine outputs the target search results in a streaming manner, which may include the following steps:

31. The log search engine obtains target attribute information of the receiving end;

32. Determine the target processing parameter corresponding to the target attribute information according to the preset mapping relationship between the attribute information and the processing parameter;

33. Process the target search result according to the target processing parameter to obtain a processed search result;

34. Output the processing search results in a streaming manner.

Among them, in the embodiment of the present application, the attribute information may be at least one of the following: device identification, login information of the receiving end, etc., which are not limited here, and the device identification information may be at least one of the following: IP address, device model, MAC Address, ICCID, IMEI, etc. are not limited here. The login information of the receiving end can be at least one of the following: login account, login geographic location, login method, etc., which are not limited here. The processing parameters can be at least one of the following: display parameters (resolution, color, font size, language type, etc.), display effects (animation, text, voice, etc.), etc., which are not limited here.

In specific implementation, the log warehouse service can pre-store the mapping relationship between preset attribute information and processing parameters, and the log search engine can obtain the target attribute information of the receiving end, and further, it can be based on the preset attribute information and processing parameters. To determine the target processing parameters corresponding to the target attribute information, and process the target search results according to the target processing parameters to obtain the processed search results, and output the processed search results in a streaming manner. On the one hand, it can display the corresponding corresponding to different users. Search results, on the other hand, outputting search results in a data stream can improve search efficiency.

The embodiment of this application can solve the problem that the search engine ES in the ELK log service solution commonly adopted in related technologies requires user intervention to establish a log index, consumes a large amount of memory resources, and once the server goes down, data recovery is extremely slow and inefficient, and the original data It is a problem that IO performance needs to be placed multiple times and cannot be linearly expanded. In the above embodiment of the present application, Logengine is used as a search engine, which, while also having high performance, effectively solves the above shortcomings and greatly saves valuable server resources. In turn, it can provide a full range of log integration services for the massive log data generated.

In specific implementation, in the embodiment of this application, Logengine can be used as the log query search engine module component of the cloud platform log service loghouse, providing users with high-efficiency, low-energy and rich and diverse log data search query analysis functions and interfaces, such as support for SQL queries, support Streaming and docking big data components to analyze log data in real time, support keyword query matching, support regular expression query filtering, support real-time log tail, support log query of historical time period, support log download, etc.

1. Logengine architecture and design are high-performance, stateless and linearly scalable service components, which are completely transparent to users. Only grpc and http interfaces are exposed for users to use. At the same time, it can connect to a variety of big data services such as spark, flink, kfaka, etc. The log data is transmitted in a format for real-time analysis, statistics, search and filtering. The internal implementation is mainly divided into the following processes:

2. Logengine obtains log data from the loghouse system in a strict and orderly manner according to the time range. Loghouse is designed as kv storage. K is carefully designed to be generated in accordance with the time dimension and log sequence. V stores the original log data, which is dependent on the bottom layer. Rocksdb, the io performance of finding, locating and reading and writing log data is very good, and can reach the rate of TB/s. In the actual production environment, the limit rate is the upper limit of the network card traffic;

3. Cut according to the time dimension, concurrently obtain streaming log data, and use high-performance string matching algorithms (such as kmp algorithm, naive algorithm, etc., this application can use the Google open source re2 regular engine algorithm) for each stream as the log key Word search and matching, and the results are still streamed to the demand side. The log data of this application is transmitted and calculated in the way of log streams in the entire ecological chain, so it can greatly improve the performance of log search without the consumption of a large amount of additional memory resources;

4. Logengine only acts as the analysis and calculation module of the log stream, and does not store relevant data. Once the log data is analyzed and calculated, the results will be sent to the relevant peer immediately. Therefore, Logengine is a completely stateless service that can be expanded linearly and dynamically. , And can be migrated to any platform in a cloud native way;

5. Logengine's search performance is as high as GB/s, and it does not require huge memory and cpu resource support, which is impossible for other log search engines;

6. In terms of the function of Logengine, it provides a rich and diverse query interface for log demanders, allowing users to use the log system as simple and efficient as looking up log data locally. At the same time, logengine also implements the function of aggregated search, including cross- Features such as clusters, cross-machine rooms, and multiple activities in different places, and supports real-time intelligent monitoring of log data and alarms, and timely notification of abnormalities in the demand-side log to help users quickly locate problems;

7. The Logengine service provides related access methods for big data components, which can directly connect to spark, flink, kfaka, etc., and provide log stream pipeline transmission for applications that need to analyze and merge massive log data.

The log search engine (Logengine) in the above embodiment of the present application can have the following functions:

1. Logengine does not store any original log data, and searches and analyzes the log stream in real time, which saves a lot of memory resources;

2. Since there is no need to store any data, Logengine is completely cloud native, which can be dynamically transplanted and linearly expanded;

3. It is completely transparent and friendly to users. Users can write log data in any way they want. From writing to querying the entire ecological chain through Logengie, users do not perceive and do not need additional operations similar to indexing rules. ；

4. The number of log data placements is reduced, and the entire log stream only needs to be placed once (rocksdb is required to be placed once in the loghouse system), which greatly improves the IO and query performance of log data;

5. Logengine does not need to restore the state machine, and all runs in the container self-check mode. As long as not all nodes are down, it will not affect the user's query analysis experience, and once a node is down, the service container will pass the self-check Immediately re-launched and put into service in situ, so the availability and reliability of this application is as high as 6 9s.

In a possible example, before step 101, the following steps may also be included:

A1. Obtain the user's target physiological state parameters;

A2. Determine the target emotion type corresponding to the target physiological state parameter;

A3. When the target emotion type is a preset emotion type, execute the step of obtaining the target log data from the log warehouse service by the log search engine according to a preset time sequence.

Among them, in the embodiments of the present application, the physiological state parameters may be various parameters used to reflect the physiological functions of the user, and the physiological state parameters may be at least one of the following: heart rate, blood pressure, blood temperature, blood lipid content, blood glucose content, and thyroxine content , Adrenaline content, platelet content, blood oxygen content, etc., are not limited here. The preset emotion type can be set by the user or the system defaults. The preset emotion type can be at least one of the following: dull, crying, calm, irritable, excited, depressed, etc., which are not limited here.

In specific implementation, the electronic device can obtain the user's target physiological state parameters through a wearable device that can communicate with the electronic device. Different physiological state parameters reflect the user's emotional type. The electronic device can pre-store the physiological state parameters and emotions. The mapping relationship between the types, and further, the target emotion type corresponding to the target physiological state parameter can be determined according to the mapping relationship, and further, when the target emotion type is the preset emotion type, step 101 may be executed, otherwise, step 101 may not be executed 101.

In a possible example, when the target physiological state parameter is a heart rate change curve within a specified time period, the above step A1, determining the target emotion type corresponding to the target physiological state parameter, can be implemented in the following manner:

A11. Sampling the heart rate change curve to obtain multiple heart rate values;

A12. Perform an average calculation based on the multiple heart rate values to obtain an average heart rate value;

A13. Determine the target heart rate level corresponding to the average heart rate value;

A14. Determine the target first emotion value corresponding to the target heart rate level according to the preset mapping relationship between the heart rate level and the first emotion value;

A15. Perform the mean square error calculation according to the multiple heart rate values to obtain the target mean square error;

A16. Determine the target second emotion value corresponding to the target mean square error according to the preset mapping relationship between the mean square error and the second emotion value;

A17. Determine the target weight pair corresponding to the target heart rate level according to the preset mapping relationship between the heart rate level and the weight value pair, and the weight value pair includes a first weight value and a second weight value. A weight value is a weight value corresponding to the first emotion value, and the second weight value is a weight value corresponding to the second emotion value;

A18. Perform a weighted operation according to the target first emotion value, the target second emotion value, and the target weight pair to obtain a final emotion value;

A19. Determine the target emotion type corresponding to the target emotion value according to the preset mapping relationship between the emotion value and the emotion type.

The specified time period can be set by the user or the system defaults. The electronic device can pre-store the mapping relationship between the preset heart rate level and the first emotion value, and the preset mean square error and the second emotion value. The mapping relationship, and the mapping relationship between the preset heart rate level and the weight value pair, and the mapping relationship between the preset emotion value and the emotion type, the above weight value pair may include a first weight value and a second weight value, The first weight value is the weight value corresponding to the first sentiment value, and the second weight value is the weight value corresponding to the second sentiment value. The sum of the first weight value and the second weight value can be 1, and the first weight value , The value range of the second weight is 0～1. In the embodiment of the present application, the emotion can be evaluated by the heart rate change curve.

In specific implementation, the electronic device can sample the heart rate curve. The specific sampling method can be: uniform sampling or random sampling to obtain multiple heart rate values, and the average heart rate can be calculated based on the multiple heart rate values to obtain the average heart rate value. The mapping relationship between the heart rate value and the heart rate level can be pre-stored in the, and then the target heart rate level corresponding to the average heart rate value can be determined according to the mapping relationship, and further, can be based on the preset heart rate level and the first emotional value. The mapping relationship is used to determine the target first emotion value corresponding to the target heart rate level. Furthermore, the mean square error operation can be performed on multiple heart rate values to obtain the target mean square error, and the target mean square error can be calculated according to the preset mean square error and the second emotion value. The mapping relationship determines the target second sentiment value corresponding to the target mean square error.

Further, the electronic device may also determine a target weight pair corresponding to the target heart rate level according to the above-mentioned preset mapping relationship between the heart rate level and the weight value pair, and the target weight value pair may include the target first weight value and the target first weight value. A weight value, the target first weight value is the weight value corresponding to the target first emotion value, and the target second weight value is the weight value corresponding to the target second emotion value. Furthermore, the electronic device can be based on the target first emotion value and the target first emotion value. The second emotional value, the first weight of the target and the second weight of the target are weighted to obtain the final emotional value. The specific calculation formula is as follows:

Final sentiment value = target first sentiment value * target first weight + target second sentiment value * target second weight

Furthermore, the target emotion type corresponding to the target emotion value can be determined according to the foregoing preset mapping relationship between the emotion value and the emotion type. Among them, the above average heart rate reflects the user's heart rate value, the mean square error of the heart rate reflects the stability of the heart rate, and the user's emotion is reflected through the two dimensions of the average heart rate and the mean square error, and the user's emotion type can be accurately determined.

Consistent with the above, please refer to FIG. 2. FIG. 2 is a schematic flowchart of another log data processing method provided by an embodiment of the present application. The log data processing method described in this embodiment is applied to the server as shown in FIG. 1A or In the system architecture shown in FIG. 1B, the server may include a log service system, and the log service system includes a log search engine and a log warehouse service. The method may include the following steps:

201. The log search engine obtains target log data from the log warehouse service according to a preset time sequence, + the log warehouse service stores the target log data in KV storage, and the key value of the KV storage is determined by time The dimensions and log sequence are generated, the V of the KV stores original log data, and the target log data is part of the log data of the original log data.

202. The log search engine performs keyword search and matching on the target log data by using a preset log search algorithm to obtain target search results.

203. The log search engine outputs the processed search result in a streaming manner.

For the detailed description of the above steps 201 to 203, reference may be made to the log data processing method shown in FIG. 1C, which will not be repeated here.

It can be seen that the log data processing method described in the embodiment of this application is applied to a log service system. The log service system includes a log search engine and a log warehouse service. The log search engine does not store any original log data, but serves from the log warehouse. Obtain log data, and perform real-time search and analysis of logs, which can save memory resources. In addition, log data is transmitted and calculated in the form of log streams in the entire ecological chain, so that there is no need for additional large amounts of memory to the greatest extent. Under the consumption of resources, the performance of log search is greatly improved.

The following is a device for implementing the above log data processing method, which is specifically as follows:

Consistent with the above, please refer to FIG. 3, which is a server provided by an embodiment of the present application, including: a processor and a memory; and one or more programs, the server includes a log service system, and the log service system includes Log search engine and log warehouse service, the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the following steps:

The log search engine outputs the target search result.

It can be seen that the server described in the embodiment of the present application includes a log service system. The log service system includes a log search engine and a log warehouse service. The log search engine obtains target log data from the log warehouse service according to a preset timing, and the log search The engine performs keyword search and matching on the target log data through the preset log search algorithm to obtain the target search results, and the log search engine outputs the target search results. In this way, the log search engine does not store any original log data, but obtains it from the log warehouse service Log data, and real-time search and analysis of logs can save memory resources.

In a possible example, in terms of the log search engine obtaining target log data from the log warehouse according to a preset time sequence, the program includes instructions for executing the following steps:

The log search engine cuts the original log data according to the time dimension to obtain cut log data;

The cutting log data is processed into streaming log data to obtain the target log data.

In a possible example, in terms of the log search engine outputting the target search result, the program includes instructions for executing the following steps:

The log search engine outputs the target search results in a streaming manner.

In a possible example, in terms of the log search engine outputting the target search results in a streaming manner, the program includes instructions for executing the following steps:

The log search engine obtains the target attribute information of the receiving end;

Determine the target processing parameter corresponding to the target attribute information according to the preset mapping relationship between the attribute information and the processing parameter;

Process the target search result according to the target processing parameter to obtain a processed search result;

The processing search results are output in a streaming manner.

In a possible example, in terms of the log search engine performing keyword search and matching on the target log data through a preset log search algorithm to obtain target search results, the program includes instructions for executing the following steps:

The log search engine performs keyword extraction on the target log data to obtain the first keyword;

Obtain pre-stored library data;

Performing keyword extraction on the library data to obtain multiple second keywords;

Match the first keyword with each second keyword of the plurality of second keywords through the preset log search algorithm to obtain a plurality of matching values, and among the plurality of second keywords Each second keyword corresponds to a match value;

The maximum value is selected from the multiple matching values, and the content corresponding to the second keyword corresponding to the maximum value is used as the target search result.

In a possible example, the first keyword and each second keyword of the plurality of second keywords are matched by the preset log search algorithm to obtain a plurality of matching values In one aspect, the program includes instructions for performing the following steps:

Determine the first attribute information of the first keyword;

Determining second attribute information of each of the plurality of second keywords to obtain a plurality of second attribute information;

Matching the first attribute information with the plurality of second attribute information to obtain a plurality of attribute matching values, and each second attribute information corresponds to an attribute matching value;

Filter out the attribute matching values that meet the preset requirements from the multiple attribute matching values to obtain multiple target attribute matching values, obtain keywords corresponding to the multiple target attribute matching values, and obtain multiple target second keywords ；

The first keyword and each of the multiple target second keywords are matched by the preset log search algorithm to obtain the multiple matching values.

Please refer to FIG. 4, which is a schematic structural diagram of a log data processing apparatus provided by this embodiment. The log data processing device is applied to a log service system. The log service system includes a log search engine and a log warehouse service. The log service system is applied to the server shown in FIG. 1A or the system architecture shown in FIG. 1B. The data processing device includes: an acquisition unit 401, a search unit 402, and an output unit 403, where:

The obtaining unit 401 is configured to obtain target log data from the log warehouse service according to a preset time sequence through the log search engine;

The searching unit 402 is configured to perform keyword search and matching on the target log data through the log search engine through a preset log search algorithm to obtain target search results;

The output unit 403 is configured to output the target search result through the log search engine.

It can be seen that the log data processing device described in the embodiment of the present application is applied to a log service system. The log service system includes a log search engine and a log warehouse service. The log search engine obtains target log data from the log warehouse service according to a preset timing. The log search engine performs keyword search and matching on the target log data through the preset log search algorithm, and obtains the target search results. The log search engine outputs the target search results. In this way, the log search engine does not store any original log data, but from the log warehouse The service obtains log data, and performs real-time search and analysis of logs, which can save memory resources.

In a possible example, in terms of obtaining target log data from the log warehouse according to a preset time sequence through the log search engine, the obtaining unit 401 is specifically configured to:

Cutting the original log data according to the time dimension by the log search engine to obtain cutting log data;

In a possible example, in the output of the target search result through the log search engine, the output unit 403 is specifically configured to:

The target search result is output in a streaming manner through the log search engine.

In a possible example, in terms of outputting the target search result in a streaming manner through the log search engine, the output unit 401 is specifically configured to:

Obtaining target attribute information of the receiving end through the log search engine;

The processing search results are output in a streaming manner.

In a possible example, in terms of performing keyword search and matching on the target log data through the log search engine and the preset log search algorithm to obtain target search results, the search unit 402 is specifically configured to:

Performing keyword extraction on the target log data through the log search engine to obtain the first keyword;

Obtain pre-stored library data;

In a possible example, the first keyword and each second keyword of the plurality of second keywords are matched by the preset log search algorithm to obtain a plurality of matching values In aspect, the searching unit 402 is specifically configured to:

Determine the first attribute information of the first keyword;

It is understandable that the functions of the program modules of the log data processing apparatus of this embodiment can be implemented according to the method in the above method embodiment, and the specific implementation process can be referred to the relevant description of the above method embodiment, which will not be repeated here. .

An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute any log data processing method recorded in the above method embodiment. Part or all of the steps.

The embodiments of the present application also provide a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to cause a computer to execute the method described in the foregoing method embodiment. Part or all of the steps of any log data processing method.

It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited by the described sequence of actions. Because according to this application, some steps can be performed in other order or at the same time. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in an embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed device may be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or may be Integrate into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software program modules.

If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer readable memory. Based on this understanding, the technical solution of the present application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory. A number of instructions are included to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The foregoing memory includes: U disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable memory, and the memory can include: a flash disk , ROM, RAM, magnetic disk or CD, etc.

The embodiments of the application are described in detail above, and specific examples are used in this article to illustrate the principles and implementation of the application. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the application; at the same time, for Those of ordinary skill in the art, based on the idea of the application, will have changes in the specific implementation and the scope of application. In summary, the content of this specification should not be construed as a limitation to the application.

Claims

A log data processing method, characterized in that it is applied to a log service system, the log service system includes a log search engine and a log warehouse service, and the method includes:

The log search engine obtains target log data from the log warehouse service according to a preset time sequence;

The log search engine performs keyword search and matching on the target log data by using a preset log search algorithm to obtain target search results;

The log search engine outputs the target search result.
The method according to claim 1, wherein the log search engine acquiring target log data from the log warehouse according to a preset time sequence comprises:

The log search engine cuts the original log data according to the time dimension to obtain cut log data;

The cutting log data is processed into streaming log data to obtain the target log data.
The method according to claim 2, wherein the log search engine outputting the target search result comprises:

The log search engine outputs the target search results in a streaming manner.
The method according to claim 3, wherein the log search engine outputting the target search results in a streaming manner comprises:

The log search engine obtains the target attribute information of the receiving end;

Determine the target processing parameter corresponding to the target attribute information according to the preset mapping relationship between the attribute information and the processing parameter;

Process the target search result according to the target processing parameter to obtain a processed search result;

The processing search results are output in a streaming manner.
The method according to any one of claims 1 to 4, wherein the log search engine is only used to implement log stream analysis functions, and is not used to store data.
The method according to any one of claims 1-5, wherein the log search engine performs keyword search and matching on the target log data by using a preset log search algorithm to obtain target search results, comprising:

The log search engine performs keyword extraction on the target log data to obtain the first keyword;

Obtain pre-stored library data;

Performing keyword extraction on the library data to obtain multiple second keywords;

Match the first keyword with each second keyword of the plurality of second keywords through the preset log search algorithm to obtain a plurality of matching values, and among the plurality of second keywords Each second keyword corresponds to a match value;

The maximum value is selected from the multiple matching values, and the content corresponding to the second keyword corresponding to the maximum value is used as the target search result.
8. The method according to claim 6, wherein the matching of the first keyword and each second keyword of the plurality of second keywords is performed by the preset log search algorithm, Get multiple matching values, including:

Determine the first attribute information of the first keyword;

Determining second attribute information of each of the plurality of second keywords to obtain a plurality of second attribute information;

Matching the first attribute information with the plurality of second attribute information to obtain a plurality of attribute matching values, and each second attribute information corresponds to an attribute matching value;

Filter out the attribute matching values that meet the preset requirements from the multiple attribute matching values to obtain multiple target attribute matching values, obtain keywords corresponding to the multiple target attribute matching values, and obtain multiple target second keywords ；

The first keyword and each of the multiple target second keywords are matched by the preset log search algorithm to obtain the multiple matching values.
The method according to any one of claims 1-7, wherein the storage mode of the log warehouse service for the target log data is KV storage.
The method according to claim 8, wherein the key value stored in the KV is generated by the time dimension and log sequence, the V of the KV stores original log data, and the target log data is the original log Part of the log data of the data.
A log data processing device, characterized in that it is applied to a log service system, the log service system includes a log search engine and a log warehouse service, and the device includes: an acquisition unit, a search unit, and an output unit, wherein:

The obtaining unit is configured to obtain target log data from the log warehouse service according to a preset time sequence through the log search engine;

The search unit is configured to perform keyword search and matching on the target log data through the log search engine through a preset log search algorithm to obtain target search results;

The output unit is configured to output the target search result through the log search engine.
The device according to claim 10, wherein, in the aspect of obtaining target log data from the log warehouse according to a preset time sequence through the log search engine, the obtaining unit is specifically configured to:

Cutting the original log data according to the time dimension by the log search engine to obtain cutting log data;

The cutting log data is processed into streaming log data to obtain the target log data.
The device according to claim 11, wherein, in the output of the target search result through the log search engine, the output unit is specifically configured to:

The target search result is output in a streaming manner through the log search engine.
The device according to claim 12, wherein in the aspect of outputting the target search result in a streaming manner through the log search engine, the output unit is specifically configured to:

Obtaining target attribute information of the receiving end through the log search engine;

Determine the target processing parameter corresponding to the target attribute information according to the preset mapping relationship between the attribute information and the processing parameter;

Process the target search result according to the target processing parameter to obtain a processed search result;

The processing search results are output in a streaming manner.
The device according to any one of claims 10-13, wherein the log search engine is only used to implement log stream analysis functions, and is not used to store data.
The device according to any one of claims 10-14, wherein the target log data is searched and matched by keywords by the log search engine through a preset log search algorithm to obtain target search results On the one hand, the search unit is specifically used for:

Performing keyword extraction on the target log data through the log search engine to obtain the first keyword;

Obtain pre-stored library data;

Performing keyword extraction on the library data to obtain multiple second keywords;

Match the first keyword with each second keyword of the plurality of second keywords through the preset log search algorithm to obtain a plurality of matching values, and among the plurality of second keywords Each second keyword corresponds to a match value;

The maximum value is selected from the multiple matching values, and the content corresponding to the second keyword corresponding to the maximum value is used as the target search result.
15. The device of claim 15, wherein the first keyword is matched with each second keyword of the plurality of second keywords through the preset log search algorithm , In terms of obtaining multiple matching values, the search unit is specifically configured to:

Determine the first attribute information of the first keyword;

Determining second attribute information of each of the plurality of second keywords to obtain a plurality of second attribute information;

Matching the first attribute information with the plurality of second attribute information to obtain a plurality of attribute matching values, and each second attribute information corresponds to an attribute matching value;

Filter out the attribute matching values that meet the preset requirements from the multiple attribute matching values to obtain multiple target attribute matching values, obtain keywords corresponding to the multiple target attribute matching values, and obtain multiple target second keywords ；

The first keyword and each of the multiple target second keywords are matched by the preset log search algorithm to obtain the multiple matching values.
The device according to any one of claims 10-16, wherein the storage mode of the log warehouse service for the target log data is KV storage.
A server, characterized in that the server includes a processor, a memory, a communication interface, and one or more programs, the server includes a log service system, the log service system includes a log search engine and a log warehouse service, the one One or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the method according to any one of claims 1-9.
A computer-readable storage medium, characterized by storing a computer program for electronic data exchange, wherein the computer program causes a computer to execute the method according to any one of claims 1-9.
A computer program product, wherein the computer program product comprises a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute any one of claims 1-9 The method described.