CN114756850A - Data acquisition method, device, equipment and storage medium - Google Patents
Data acquisition method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114756850A CN114756850A CN202210288968.3A CN202210288968A CN114756850A CN 114756850 A CN114756850 A CN 114756850A CN 202210288968 A CN202210288968 A CN 202210288968A CN 114756850 A CN114756850 A CN 114756850A
- Authority
- CN
- China
- Prior art keywords
- platform
- data
- acquiring
- key information
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/31—User authentication
- G06F21/36—User authentication by graphic or iconic representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
- G06F16/9566—URL specific, e.g. using aliases, detecting broken or misspelled links
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Computer Security & Cryptography (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computer Hardware Design (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to the technical field of artificial intelligence, and discloses a data acquisition method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring addresses of all platforms and identity information corresponding to users; logging in each platform and acquiring key information in each platform based on the address of each platform and the identity information corresponding to the user; summarizing and editing based on the key information to obtain summarized data; processing the summarized data by using a judgment model, and judging whether to alarm or not; and when alarming is carried out, the summarized data is sent to the front end according to a preset alarming rule. The method and the device improve the efficiency and accuracy of data acquisition.
Description
Technical Field
The present application relates to the field of artificial intelligence, and in particular, to a method, an apparatus, a device, and a storage medium for acquiring data.
Background
At present, certain steps are often processed by depending on manual work in the data acquisition process, for example, certain platforms are logged in, so that the efficiency is low, and when the data is specifically acquired, the key information cannot be specifically mastered, so that when the data is acquired, the data is acquired only in a large-batch copying mode, and the effective data cannot be accurately acquired, so that the efficiency is low; in the prior art, the login state of a user on each platform is maintained by maintaining an account online manner to acquire data on the platform, but the efficiency of acquiring valid data is still low due to a direct copying manner, so how to improve the efficiency of acquiring valid data becomes a problem to be solved urgently.
Disclosure of Invention
The application provides a data acquisition method, a data acquisition device, data acquisition equipment and a storage medium, and aims to solve the problem that the efficiency of acquiring effective data is low in the prior art.
In order to solve the above problem, the present application provides a data acquisition method, including:
acquiring addresses of all platforms and identity information corresponding to users;
logging in each platform and acquiring key information in each platform based on the address of each platform and identity information corresponding to a user, wherein the key information comprises images and/or keywords;
summarizing and editing based on the key information to obtain summarized data;
processing the summarized data by using a judgment model, and judging whether to alarm or not;
and when alarming is carried out, the summarized data is sent to the front end according to a preset alarming rule.
Further, the logging in each of the platforms based on the address of each of the platforms and the identity information corresponding to the user includes:
based on the platform address, after entering a corresponding webpage, when detecting that the webpage has a verification code frame, acquiring a corresponding verification code according to an image recognition model;
and logging in the platform according to the identity information and the verification code corresponding to the user.
Further, when it is detected that the webpage has a verification code frame, acquiring a corresponding verification code according to the image recognition model includes:
positioning based on the verification code frame, and intercepting the image in the verification code frame;
and identifying the image according to the image identification model to obtain the verification code, wherein the image identification model is obtained based on CRNN model training.
Further, the obtaining key information in each of the platforms includes:
screenshot is carried out on an interface on the platform; and/or
And performing feature extraction on the data on the interface by using a feature extraction model to obtain key words in the key information.
Further, the performing summary editing based on the key information to obtain summary data includes:
and carrying out classification and summarization processing on the key information corresponding to each platform by using a classification model to respectively obtain corresponding summarized data, wherein the classification model is obtained based on Bayesian algorithm training.
Further, the processing the summarized data by using the judgment model, and judging whether to alarm or not includes:
respectively inputting the summarized data corresponding to each platform into the judgment model for judgment to obtain a corresponding judgment result, wherein the judgment model is obtained based on decision tree model training;
And processing by using a preset judgment rule based on each judgment result to judge whether to alarm or not.
Further, before the obtaining of the key information in each of the platforms, the method further includes:
setting a timer for preset time;
and executing the step of acquiring the key information in each platform when the timer reaches the preset time each time.
In order to solve the above problem, the present application further provides an apparatus for acquiring data, the apparatus including:
the acquisition module is used for acquiring the address of each platform and the identity information corresponding to the user;
the login module is used for logging in each platform and acquiring key information in each platform based on the address of each platform and the identity information corresponding to the user, wherein the key information comprises images and/or keywords;
the summarizing module is used for summarizing and editing based on the key information to obtain summarized data;
the judging module is used for processing the summarized data by utilizing a judging model and judging whether to alarm or not;
and the pushing module is used for sending the summarized data to the front end according to a preset alarm rule when an alarm is given.
In order to solve the above problem, the present application further provides a computer device, including:
at least one processor; and (c) a second step of,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method of data acquisition as described above.
In order to solve the above problem, the present application also provides a non-volatile computer readable storage medium, on which computer readable instructions are stored, and the computer readable instructions, when executed by a processor, implement the data acquisition method as described above.
Compared with the prior art, the data acquisition method, the data acquisition device, the data acquisition equipment and the data storage medium provided by the embodiment of the application have at least the following beneficial effects:
firstly, the address of each platform and the identity information corresponding to the user are obtained, based on the address of each platform and the identity information corresponding to the user, each platform is logged in, the online state of the user on each platform is kept, and then, key information in each platform is obtained, so as to obtain effective data, and the effective data is summarized and edited based on the key information to obtain summarized data corresponding to each platform, so that it is convenient for subsequent judgment of giving an alarm to the user, and the summarized data is processed by using a judgment model to judge whether the alarm is corresponding, namely, when the summarized data is judged to be abnormal, the alarm reminding is carried out on the user, when the alarm is carried out, and sending the summarized data to the front end according to a preset alarm rule, so that a user can conveniently check abnormal summarized data, the data can be efficiently acquired, and the efficiency and the accuracy of acquiring effective data are improved.
Drawings
In order to more clearly illustrate the solution of the present application, a brief description will be given below of the drawings required for describing the embodiments of the present application, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without inventive effort.
Fig. 1 is an overall flowchart of a data acquisition method according to an embodiment of the present application;
FIG. 2 is a flowchart illustrating one embodiment of step S2 of FIG. 1;
FIG. 3 is a flowchart illustrating one embodiment of step S21 of FIG. 2;
fig. 4 is a schematic block diagram of an apparatus for acquiring data according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present application.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof in the description and claims of this application and the description of the figures above, are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the above-described drawings are used for distinguishing between different objects and not for describing a particular order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. One skilled in the art will explicitly or implicitly appreciate that the embodiments described herein can be combined with other embodiments.
The application provides a data acquisition method. The method is mainly used for acquiring data on the platform, acquiring the data on the platform and judging the acquired data so as to determine whether to push the data to a user. Referring to fig. 1, fig. 1 is a schematic flowchart of a data acquisition method according to an embodiment of the present disclosure.
In this embodiment, the data acquiring method includes:
s1, acquiring the address of each platform and the identity information corresponding to the user;
specifically, the identity is maintained to be valid by acquiring the URL and the parameters of each platform, the user name, the password, the identity token and the like corresponding to the user in each platform, and circularly acquiring the identity token, and after once login, the identity authentication is not required to be maintained in the period, so that the user can be ensured to be maintained online in each platform. Each platform of the identity token is inconsistent and can be found in a configuration file in a corresponding website platform.
S2, logging in each platform and acquiring key information in each platform based on the address of each platform and the identity information corresponding to the user, wherein the key information comprises images and/or keywords;
specifically, logging in to each platform according to the URL and parameters of each platform, and the user name, the password, the identity token and the like corresponding to the user in each platform; thereby obtaining key information in each platform, wherein the key information comprises screenshots, images and/or keywords and the like; the screenshot is a certain area of a platform web page, and the keyword is a keyword of information under a corresponding account of the user. The acquired key information is information in a time period, and not all information displayed on the platform.
Further, as shown in fig. 2, the logging in each of the platforms based on the address of each of the platforms and the identity information corresponding to the user includes:
s21, based on the platform address, after entering a corresponding webpage, when detecting that the webpage has a verification code frame, acquiring a corresponding verification code according to an image recognition model;
and S22, logging in the platform according to the identity information and the verification code corresponding to the user.
Specifically, after entering a corresponding webpage based on the address of the network platform, namely a corresponding URL, a login interface is displayed first, wherein when a username and a password box under the webpage are detected and a verification code box is available, an image recognition model needs to be called to obtain an image in the verification code box to obtain a corresponding verification code;
and filling the obtained verification code into a corresponding area according to the identity information corresponding to the user, including the user name, the password, the identity token and the like, so as to log in the platform.
In other embodiments of the present application, the RPA hornet may be directly invoked for identifying the verification code; the RPA Anoman is an existing process automation robot and can perform the identifying code identification.
In other embodiments of the present application, in the login interface, only when a user name and a password box are detected, the platform can be logged in by directly using the identity information of the user.
The verification code is intelligently obtained, the platform is logged in according to the obtained verification code and the identity information, the login state is maintained, the key information can be conveniently obtained subsequently, and the processing efficiency is improved.
Still further, as shown in fig. 3, when it is detected that the webpage has a verification code frame, acquiring a corresponding verification code according to the image recognition model includes:
S211, positioning based on the verification code frame, and intercepting the image in the verification code frame;
s212, identifying the image according to the image identification model to obtain the verification code, wherein the image identification model is obtained based on CRNN model training.
Specifically, when a verification code frame is detected on a login interface, screenshot is carried out based on the positions of the upper left corner and the lower right corner of the verification code frame, and an image in the verification code frame is intercepted; and inputting the image into an image recognition model to recognize the image to obtain a corresponding verification code.
The CRNN (Convolutional Recurrent Neural Network) model is to extract the features of the picture through CNN (Convolutional Neural networks), predict the sequence through RNN (Convolutional Neural networks), and finally obtain the final result through a translation layer of CTC (Convolutional Temporal classification), that is, the Network structure of CRNN is the structure of CNN + RNN + CTC. The image recognition model is obtained based on a large number of marked verification code images.
The verification code is identified based on the image identification model, so that the processing efficiency is improved.
Further, the obtaining key information in each of the platforms includes:
screenshot is carried out on an interface on the platform; and/or
And performing feature extraction on the data on the interface by using a feature extraction model to obtain key words in the key information.
Specifically, the key information includes a screenshot and/or a keyword, the screenshot in the key information can be obtained by screenshot on a related interface in the platform, and further, the screenshot can be performed on information under an account corresponding to a user in the platform and/or screenshot on alarm information on the platform;
and corresponding to the text data in the platform, performing feature extraction on the text data in the interface by using a feature extraction model to obtain a keyword in the key information, wherein the feature extraction model is obtained by training based on the CNN (Convolutional Neural Networks) model.
Specifically, elements of the webpage in the designated area are grabbed according to the webpage element locator.
In other embodiments of the application, keywords to be extracted can be configured, and data in the interface is extracted according to the configured keywords to be extracted; because the information displayed by the platform has a fixed format, the key information on the platform can be well extracted by configuring the form of the key words to be extracted.
The key information in the platform is obtained through screenshot and/or keyword extraction, and the efficiency and accuracy of data acquisition are improved.
Further, before the obtaining of the key information in each of the platforms, the method further includes:
setting a timer for preset time;
and executing the step of acquiring the key information in each platform when the timer reaches the preset time each time.
Specifically, the configuration may be performed in units of seconds, minutes, hours, days, weeks, months, or years by setting a timer for a preset time, and when the timer reaches the preset time set each time, the step of acquiring the key information in each of the platforms and the subsequent steps are performed.
The key information is acquired at regular time by setting the timer, so that the data is acquired and processed at regular time, and the timeliness of subsequent alarm is ensured.
S3, summarizing and editing based on the key information to obtain summarized data;
specifically, the key information corresponding to each platform is classified, summarized and edited to obtain summarized data corresponding to each platform. When the summary editing is carried out, the key information data can be directly summarized and edited into a chart form, and the chart can be pushed to a user so as to be convenient for the user to check and carry out subsequent processing.
Further, the summarizing and editing based on the key information to obtain summarized data includes:
and classifying and summarizing the key information corresponding to each platform by using a classification model to respectively obtain corresponding summarized data, wherein the classification model is obtained by training based on a Bayesian algorithm.
Specifically, by using a classification model, classifying and summarizing the key information corresponding to each platform, that is, mainly classifying the keywords in the key information to obtain the keywords of each category, that is, summarized data;
the Bayesian algorithm is a Bayesian classification algorithm which is a statistical classification method and is classified by using probability statistical knowledge. In many cases, naive Bayes (the Na (a (es, N () classification algorithm can be compared with decision tree and neural network classification algorithms, which can be applied to large databases, and has the advantages of simple method, high classification accuracy and high speed.
The key information is classified and summarized to obtain summarized data, so that whether the alarm processing is continued or not can be conveniently judged subsequently, the processing efficiency is improved, and the user can conveniently check the alarm processing.
S4, processing the summarized data by using a judgment model, and judging whether to alarm or not;
specifically, the judgment model is respectively input for judgment based on the obtained summarized data corresponding to each platform to obtain a corresponding judgment result, and further judgment is performed based on the judgment result to judge whether to alarm or not. If the alarm is not carried out, the summarized data obtained in the next timing period is waited for, and then judgment is carried out.
Further, the processing the summarized data by using the judgment model, and judging whether to alarm includes:
respectively inputting the summarized data corresponding to each platform into the judgment model for judgment to obtain corresponding judgment results, wherein the judgment models are obtained based on decision tree model training;
and processing by using a preset judgment rule based on each judgment result to judge whether to alarm or not.
Specifically, the summarized data corresponding to each platform is respectively input into the judgment model for judgment to obtain a corresponding judgment result, for example, the bug unified management platform is tested by relevant testers in a software development and test stage, relevant bug problems are fed back, key information is obtained by performing feature extraction on text data of bug problems and is summarized to obtain summarized data, and the judgment model is used for judgment to obtain a corresponding judgment result according to the summarized data of the bug unified management platform;
When only one platform is provided, the judgment of whether to alarm or not can be directly obtained according to the judgment result, namely, when the judgment result is abnormal, the alarm is carried out, and when the judgment result is normal, the alarm is not carried out; when a plurality of platforms exist, according to a preset judgment rule, the preset judgment rule in the application is the priority of each platform and the like, for example, when platforms such as a bug unified management platform, a log management platform, a database platform, a project configuration platform and a project deployment environment platform exist, the respective priorities are set, and the priorities are sequentially reduced from left to right in the arrangement order, namely when the judgment result corresponding to the bug unified management platform is abnormal, the judgment result corresponding to the following platform has no effect, and the corresponding result is an alarm; when the judgment result corresponding to the bug unified management platform is normal, checking whether the judgment result corresponding to the log management platform is abnormal or not, and if the judgment result is abnormal, giving an alarm; if the database is normal, checking a judgment result corresponding to the database platform, and analogizing in turn; the priority order given above is merely exemplary, and the specific priority order can be set as required.
In other embodiments of the present application, the processing may also be performed according to other preset determination rules, for example, the number of platforms with abnormality is limited, and when two or more abnormal platforms occur, an alarm is performed.
In other embodiments of the application, the bug level dimension extracted from the bug unified management platform is also set, and when the bug level reaches a preset level, an alarm is directly given. For example, when the bug level in the obtained summary data of the time period is the level of L2 or above, an alarm is directly given.
Judging the summarized data by using a judgment model to obtain a judgment result corresponding to each platform; and based on the judgment result, further processing by using a preset judgment rule to finally determine whether to alarm or not, so that the processing accuracy is improved, and the efficiency of obtaining effective data by a user is improved.
And S5, when an alarm is given, the summarized data is sent to the front end according to a preset alarm rule.
Specifically, when an alarm is required, according to a preset alarm rule, such as an alarm mode and the like, the alarm mode is one or more of short message reminding, mail reminding, communication software reminding and the like; sending the summarized data to a front end, and reminding the user that the warning disappears after the user confirms the warning for the mode of eliminating the warning; the manner of the alarms and the time intervals between alarms may be self-defined by the user. The summary data sent is the data for that time period.
It is emphasized that all data of the summarized data may also be stored in a node of a block chain in order to further ensure the privacy and security of the data.
The block chain referred by the application is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. The blockchain (lockchain) is a decentralized database in nature, and is a string of data blocks generated by using a cryptographic method, each data block contains information of a batch of network transactions, and is used for verifying the validity (anti-counterfeiting) of the information and generating the next block.
Firstly, the address of each platform and the identity information corresponding to the user are obtained, based on the address of each platform and the identity information corresponding to the user, each platform is logged in, the online state of the user on each platform is kept, and then key information in each platform is obtained, to obtain effective data, and performing summary editing based on the key information to obtain summary data corresponding to each platform for subsequent judgment of whether to give an alarm to the user, processing the summary data by using a judgment model to judge whether to give an alarm, namely, when the summarized data is judged to be abnormal, the user is warned, when the warning is given, and sending the summarized data to the front end according to a preset alarm rule, so that a user can conveniently check abnormal summarized data, the data can be efficiently acquired, and the efficiency and accuracy of acquiring effective data are improved.
This embodiment further provides a data acquisition apparatus, which is a functional block diagram of the data acquisition apparatus of the present application, as shown in fig. 4.
The apparatus 100 for acquiring data described herein may be installed in an electronic device. According to the realized functions, the data acquisition device 100 may include an acquisition module 101, a login module 102, a summary module 103, a judgment module 104, and a push module 105. A module, also referred to as a unit in this application, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and are stored in a memory of the electronic device.
In the present embodiment, the functions of the respective modules/units are as follows:
an obtaining module 101, configured to obtain addresses of platforms and identity information corresponding to users;
a login module 102, configured to log in each platform and obtain key information in each platform based on an address of each platform and identity information corresponding to a user, where the key information includes an image and/or a keyword;
further, the login module 102 includes a verification code obtaining sub-module and a platform login sub-module;
the verification code obtaining sub-module is used for obtaining a corresponding verification code according to an image recognition model after entering a corresponding webpage based on the platform address and when detecting that the webpage has a verification code frame;
And the platform login submodule is used for logging in the platform according to the identity information and the verification code corresponding to the user.
The verification code is intelligently acquired through the matching of the verification code acquisition sub-module and the platform login sub-module, the platform is logged in according to the acquired verification code and the identity information, the login state is maintained, the subsequent key information acquisition is facilitated, and the processing efficiency is improved.
Still further, the verification code obtaining sub-module further comprises an image capturing unit and an image identifying unit;
the image intercepting unit is used for positioning based on the verification code frame and intercepting the image in the verification code frame
The image recognition unit is used for recognizing the image according to the image recognition model to obtain the verification code, wherein the image recognition model is obtained based on CRNN model training.
Through the cooperation of the image interception unit and the image identification unit, the verification code is identified based on the image identification model, so that the processing efficiency is improved.
Further, the login module 102 includes a screenshot sub-module and a keyword extraction sub-module;
the screenshot submodule is used for screenshot through an interface on the platform; and/or
And the keyword extraction sub-module is used for extracting the features of the data on the interface by using a feature extraction model to obtain the keywords in the key information.
By matching the screenshot sub-module and the keyword extraction sub-module, the key information in the platform is obtained by means of screenshot and/or keyword extraction, and the efficiency and accuracy of data acquisition are improved.
Further, the data acquisition apparatus 100 includes a timing module and an execution module;
the timing module is used for setting a timer with preset time;
the execution module is configured to execute the step of obtaining the key information in each platform each time the timer reaches the preset time.
Through the cooperation of the timing module and the execution module, the timer is set to acquire key information at regular time, so that data is acquired and processed at regular time, and the timeliness of subsequent alarm is guaranteed.
The summarizing module 103 is configured to perform summarizing and editing based on the key information to obtain summarized data;
further, the summarizing module comprises a classifying summarizing sub-module;
and the classification and summarization submodule is used for performing classification and summarization processing on the key information corresponding to each platform by using a classification model to respectively obtain corresponding summarized data, wherein the classification model is obtained based on Bayesian algorithm training.
Through the categorised submodule of gathering, obtain the summary data through categorised the gathering to key information, whether follow-up judgement of being convenient for continues to report an emergency and ask for help or increased vigilance and handle efficiency and be convenient for the user to look over.
A judging module 104, configured to process the summarized data by using a judging model, and judge whether to alarm;
further, the judging module 104 includes a data judging sub-module and a corresponding processing sub-module;
the data judgment submodule is used for respectively inputting the summarized data corresponding to each platform into the judgment model for judgment to obtain a corresponding judgment result, wherein the judgment model is obtained based on decision tree model training;
and the corresponding processing submodule is used for processing by using a preset judgment rule based on each judgment result and judging whether to alarm or not.
Judging the summarized data by utilizing a judgment model through the matching of a data judgment submodule and a corresponding processing submodule to obtain a judgment result corresponding to each platform; and based on the judgment result, further processing by using a preset judgment rule to finally determine whether to alarm, so that the processing accuracy is improved, and the efficiency of obtaining effective data by a user is improved.
And the pushing module 105 is configured to send the summarized data to a front end according to a preset alarm rule when an alarm is performed.
By adopting the device, the data acquisition device 100 firstly acquires the address of each platform and the identity information corresponding to the user through the matching use of the acquisition module 101, the login module 102, the summarization module 103, the judgment module 104 and the push module 105, logs in each platform based on the address of each platform and the identity information corresponding to the user, keeps the online state of the user on each platform, obtains effective data from the key information in each platform, performs summarization and editing based on the key information to obtain summarized data corresponding to each platform, facilitates subsequent judgment of whether to alarm the user, processes the summarized data by using a judgment model, judges whether to alarm correspondingly, namely reminds the user when the summarized data is judged to be abnormal, and reminds the user when the data is alarmed according to preset alarm rules, and the summarized data are sent to the front end, so that a user can conveniently check abnormal summarized data, the data are efficiently acquired, and the efficiency and accuracy of acquiring effective data are improved.
The embodiment of the application further provides computer equipment. Referring to fig. 5, fig. 5 is a block diagram of a basic structure of a computer device according to the embodiment.
The computer device 4 comprises a memory 41, a processor 42, and a network interface 43, which are communicatively connected to each other via a system bus. It is noted that only computer device 4 having components 41-43 is shown, but it is understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.
The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.
The memory 41 includes at least one type of readable storage medium including flash memory, hard disks, multimedia cards, card-type memory (e.g., SD or DX memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), magnetic memory, magnetic disks, optical disks, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the computer device 4. Of course, the memory 41 may also include both an internal storage unit of the computer device 4 and an external storage device thereof. In this embodiment, the memory 41 is generally used for storing an operating system installed in the computer device 4 and various types of application software, such as computer readable instructions of a data acquisition method. Further, the memory 41 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 42 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is configured to execute computer readable instructions stored in the memory 41 or to process data, for example, execute computer readable instructions of the data acquisition method.
The network interface 43 may comprise a wireless network interface or a wired network interface, and the network interface 43 is generally used for establishing communication connection between the computer device 4 and other electronic devices.
The present embodiment implements the steps of the data obtaining method as described in the above embodiments when executing a computer readable instruction stored in a memory through a processor, and first obtains an address of each platform and identity information corresponding to a user, logs in each platform based on the address of each platform and the identity information corresponding to the user, maintains an online state of the user on each platform, obtains valid data from key information in each platform, performs summary editing based on the key information to obtain summary data corresponding to each platform, facilitates subsequent determination of whether to alarm the user, processes the summary data using a determination model, determines whether to alarm, that is, when it is determined that the summary data is abnormal, performs alarm reminding on the user, and when it is determined that the summary data is abnormal, sends the summary data to a front end according to a preset alarm rule, the abnormal summarized data can be conveniently checked by the user, the data can be efficiently acquired, and the efficiency and the accuracy of acquiring the effective data are improved.
The embodiment of the present application further provides a computer-readable storage medium, where the computer-readable instruction is stored, and the computer-readable instruction can be executed by at least one processor, so that the at least one processor executes the steps of the data obtaining method, where first, an address of each platform and identity information corresponding to a user are obtained, based on the address of each platform and the identity information corresponding to the user, the platforms are logged in, an online state of the user on each platform is maintained, then, key information in each platform is obtained to obtain valid data, a summary editing is performed based on the key information, summarized data corresponding to each platform is obtained, so as to subsequently determine whether to alarm the user, the summarized data is processed by using a determination model, whether to alarm corresponds is determined, that is, when it is determined that the summarized data is abnormal, the method and the device have the advantages that the user is warned, when the warning is conducted, the summarized data are sent to the front end according to the preset warning rule, the user can check abnormal summarized data conveniently, efficient data obtaining is achieved, and the efficiency and the accuracy of obtaining effective data are improved.
Through the description of the foregoing embodiments, it is clear to those skilled in the art that the method of the foregoing embodiments may be implemented by software plus a necessary general hardware platform, and certainly may also be implemented by hardware, but in many cases, the former is a better implementation. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.
The data acquisition apparatus, the computer device, and the computer-readable storage medium according to the embodiments of the present application have the same technical effects as the data acquisition method according to the embodiments, and are not expanded herein.
It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and is provided for the purpose of enabling a thorough understanding of the disclosure of the application. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields and are within the protection scope of the present application.
Claims (10)
1. A method for acquiring data, the method comprising:
acquiring addresses of all platforms and identity information corresponding to users;
logging in each platform and acquiring key information in each platform based on the address of each platform and identity information corresponding to a user, wherein the key information comprises images and/or keywords;
summarizing and editing based on the key information to obtain summarized data;
processing the summarized data by using a judgment model to judge whether to alarm or not;
and when alarming is carried out, the summarized data is sent to the front end according to a preset alarming rule.
2. The method according to claim 1, wherein the logging in each of the platforms based on the address of each of the platforms and the identity information corresponding to the user comprises:
based on the platform address, after entering a corresponding webpage, when detecting that the webpage has a verification code frame, acquiring a corresponding verification code according to an image recognition model;
and logging in the platform according to the identity information and the verification code corresponding to the user.
3. The method for acquiring data according to claim 2, wherein when detecting that the webpage has a verification code frame, acquiring a corresponding verification code according to an image recognition model comprises:
Positioning based on the verification code frame, and intercepting the image in the verification code frame;
and identifying the image according to the image identification model to obtain the verification code, wherein the image identification model is obtained based on CRNN model training.
4. The method for acquiring data according to claim 1, wherein the acquiring key information in each of the platforms includes:
screenshot is carried out on an interface on the platform; and/or
And performing feature extraction on the data on the interface by using a feature extraction model to obtain key words in the key information.
5. The method for acquiring data according to claim 1, wherein the performing summary editing based on the key information to obtain summary data includes:
and classifying and summarizing the key information corresponding to each platform by using a classification model to respectively obtain corresponding summarized data, wherein the classification model is obtained by training based on a Bayesian algorithm.
6. The method according to claim 1, wherein the processing the summarized data by using the judgment model to judge whether to alarm comprises:
Respectively inputting the summarized data corresponding to each platform into the judgment model for judgment to obtain a corresponding judgment result, wherein the judgment model is obtained based on decision tree model training;
and processing by using a preset judgment rule based on each judgment result to judge whether to alarm or not.
7. The method for acquiring data according to any one of claims 1 to 6, further comprising, before the acquiring key information in each of the platforms:
a timer for setting a preset time;
and executing the step of acquiring the key information in each platform when the timer reaches the preset time each time.
8. An apparatus for acquiring data, the apparatus comprising:
the acquisition module is used for acquiring the address of each platform and the identity information corresponding to the user;
the login module is used for logging in each platform and acquiring key information in each platform based on the address of each platform and the identity information corresponding to the user, wherein the key information comprises images and/or keywords;
the summarizing module is used for summarizing and editing based on the key information to obtain summarized data;
The judging module is used for processing the summarized data by utilizing a judging model and judging whether to alarm or not;
and the pushing module is used for sending the summarized data to the front end according to a preset alarm rule when an alarm is given.
9. A computer device, characterized in that the computer device comprises:
at least one processor; and (c) a second step of,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores computer readable instructions which, when executed by the processor, implement a method of acquiring data as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium, having computer-readable instructions stored thereon, which, when executed by a processor, implement a method of acquiring data as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210288968.3A CN114756850A (en) | 2022-03-22 | 2022-03-22 | Data acquisition method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210288968.3A CN114756850A (en) | 2022-03-22 | 2022-03-22 | Data acquisition method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114756850A true CN114756850A (en) | 2022-07-15 |
Family
ID=82327099
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210288968.3A Pending CN114756850A (en) | 2022-03-22 | 2022-03-22 | Data acquisition method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114756850A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115174866A (en) * | 2022-07-18 | 2022-10-11 | 珠海金智维信息科技有限公司 | RPA-based water supply pump room video monitoring system, method and storage medium |
-
2022
- 2022-03-22 CN CN202210288968.3A patent/CN114756850A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115174866A (en) * | 2022-07-18 | 2022-10-11 | 珠海金智维信息科技有限公司 | RPA-based water supply pump room video monitoring system, method and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113489713B (en) | Network attack detection method, device, equipment and storage medium | |
| CN113726784B (en) | Network data security monitoring method, device, equipment and storage medium | |
| CN112565250B (en) | Website identification method, device, equipment and storage medium | |
| CN111835737B (en) | WEB attack protection method based on automatic learning and related equipment thereof | |
| US11897527B2 (en) | Automated positive train control event data extraction and analysis engine and method therefor | |
| CN111666298A (en) | Method and device for detecting user service class based on flink, and computer equipment | |
| CN111639360A (en) | Intelligent data desensitization method and device, computer equipment and storage medium | |
| CN114756850A (en) | Data acquisition method, device, equipment and storage medium | |
| CN113282920B (en) | Log abnormality detection method, device, computer equipment and storage medium | |
| CN114386013A (en) | Automatic student status authentication method and device, computer equipment and storage medium | |
| CN114070642A (en) | Network security detection method, system, device and storage medium | |
| CN112632249A (en) | Method and device for displaying different versions of information of product, computer equipment and medium | |
| CN115757075A (en) | Task abnormity detection method and device, computer equipment and storage medium | |
| CN112966988A (en) | XGboost model-based data evaluation method, device, equipment and storage medium | |
| CN115913688B (en) | Network data security monitoring method, device, equipment and storage medium | |
| CN110321195B (en) | Data caching method for operation page, electronic device and readable storage medium | |
| CN111598159B (en) | Training method, device, equipment and storage medium of machine learning model | |
| CN110719260B (en) | Intelligent network security analysis method and device and computer readable storage medium | |
| CN117112415A (en) | Business process monitoring method based on EDA model and related equipment thereof | |
| CN117278298A (en) | Domain name detection method, device, equipment and storage medium based on artificial intelligence | |
| CN116896455A (en) | Network attack detection method and device, electronic equipment and storage medium | |
| CN115913688A (en) | Network data security monitoring method, device, equipment and storage medium | |
| CN117272256A (en) | Sensitive data detection method and device, computer equipment and storage medium | |
| CN116010971A (en) | Storage type XSS vulnerability detection system based on deep learning | |
| CN115202966A (en) | Picture processing method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |