CN116719702B - Method and device for collecting open source information, electronic equipment and storage medium - Google Patents
Method and device for collecting open source information, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116719702B CN116719702B CN202311004574.1A CN202311004574A CN116719702B CN 116719702 B CN116719702 B CN 116719702B CN 202311004574 A CN202311004574 A CN 202311004574A CN 116719702 B CN116719702 B CN 116719702B
- Authority
- CN
- China
- Prior art keywords
- file
- open source
- source information
- information acquisition
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 150
- 238000012544 monitoring process Methods 0.000 claims abstract description 33
- 230000006870 function Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 13
- 230000002159 abnormal effect Effects 0.000 claims description 10
- 238000009434 installation Methods 0.000 claims description 9
- 238000012986 modification Methods 0.000 claims description 7
- 230000004048 modification Effects 0.000 claims description 7
- 238000010276 construction Methods 0.000 claims description 5
- 238000004891 communication Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000033001 locomotion Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to the technical field of information acquisition, and discloses a method, a device, electronic equipment and a storage medium for acquiring open source information, wherein the method comprises the following steps: constructing an open source information acquisition framework, wherein the open source information acquisition framework comprises a process manager and a monitor; configuring service parameters and file catalogues on a process manager, and creating configuration files according to the file catalogues and pre-acquired open source information acquisition requirements; writing a management subprogram by using a monitor, and monitoring the task running state of the configuration file according to the management subprogram; and updating the open source information acquisition architecture according to the task running state and the service parameters, and acquiring the open source information by using the updated open source information acquisition architecture. According to the method, the open source information acquisition framework is constructed, and the open source information is acquired according to the open source information acquisition framework, so that the efficiency of program operation starting and program operation state monitoring in the open source information acquisition process can be improved.
Description
Technical Field
The present invention relates to the field of information acquisition technologies, and in particular, to a method and apparatus for acquiring open source information, an electronic device, and a storage medium.
Background
The open source information acquisition is an important link in the development process of a software component analysis (SCA, software Composition Analysis) tool, and the acquired open source information has the characteristics of various types and large data volume, so that the open source information acquisition technology needs to ensure that different types of open source information can be acquired orderly through a frame structure and algorithm logic with reasonable design. The current similar technology can generate the following problems in the process of collecting open source information: firstly, the collection procedures are various in variety and quantity, and overall management cannot be performed in a centralized manner; secondly, the acquisition program is started and restarted frequently, and the restarting speed is low; thirdly, the acquisition procedure process is complicated to check, and the appointed procedure process cannot be quickly found and stopped; fourthly, the updating iteration of the acquisition program code is more, so that the updating iteration efficiency is continuously reduced; fifthly, the acquisition program log path is complex and cannot be checked quickly; and sixthly, the running state of the acquisition program cannot be monitored rapidly and intuitively. In summary, the problem of low efficiency of program operation starting and program operation state monitoring in the open source information acquisition process exists in the prior art.
Disclosure of Invention
The invention provides a method, a device, electronic equipment and a storage medium for acquiring open source information, and mainly aims to solve the problem that the efficiency of program operation starting and program operation state monitoring is low in the open source information acquisition process.
In order to achieve the above object, the present invention provides a method for collecting open source information, including: constructing an open source information acquisition framework, wherein the open source information acquisition framework comprises a process manager and a monitor; configuring service parameters and file catalogues on a process manager, and creating configuration files according to the file catalogues and pre-acquired open source information acquisition requirements; writing a management subprogram by using a monitor, and monitoring the task running state of the configuration file according to the management subprogram; and updating the open source information acquisition architecture according to the task running state and the service parameters, and acquiring the open source information by using the updated open source information acquisition architecture.
The invention also provides a device for collecting open source information, which comprises: the acquisition architecture construction module is used for constructing an open source information acquisition architecture, wherein the open source information acquisition architecture comprises a process manager and a monitor; the configuration file generation module is used for configuring service parameters and file catalogues on the process manager and creating configuration files according to the file catalogues and the pre-acquired open source information acquisition requirements; the running state monitoring module is used for compiling a management subprogram by using a monitor and monitoring the task running state of the configuration file according to the management subprogram; and the open source information acquisition module is used for updating the open source information acquisition architecture according to the task running state and the service parameters, and acquiring the open source information by using the updated open source information acquisition architecture.
The present invention also provides an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor, so that the at least one processor can execute the method for collecting open source information.
The invention also provides a computer readable storage medium, wherein at least one computer program is stored in the computer readable storage medium, and the at least one computer program is executed by a processor in an electronic device to realize the method for collecting open source information.
According to the embodiment of the invention, the open source information acquisition architecture is constructed, so that the efficiency of acquisition and development from the multidimensional optimization program can be acquired, and the open source information can be acquired rapidly and comprehensively; by configuring service parameters and file catalogues on the process manager, the integrity of the process manager can be improved; the configuration file is created through the file catalogue and the open source information acquisition requirement, so that the accuracy of the configuration file can be ensured; the management subprogram is written by the monitor, so that the efficiency of starting the program operation and the efficiency of monitoring the program operation in the open source information acquisition process can be improved; according to the monitoring configuration file of the management subprogram, the task running state can be accurately obtained; the open source information acquisition architecture is updated through the task running state and the service parameters to obtain the open source information acquisition architecture, and the open source information is acquired by utilizing the open source information acquisition architecture, so that the rapid iteration, updating and expansion of the open source information acquisition task can be realized, and meanwhile, the running stability and flexibility of an acquisition system are ensured. Therefore, the method, the device, the electronic equipment and the storage medium for acquiring the open source information can solve the problem that the efficiency of program operation starting and program operation state monitoring is low in the open source information acquisition process.
Drawings
FIG. 1 is a flowchart illustrating a method for acquiring open source information according to an embodiment of the present invention;
FIG. 2 is a flow chart of configuring service parameters and file directories on a process manager according to an embodiment of the present invention;
FIG. 3 is a flowchart of a monitor writing management subroutine according to an embodiment of the present invention;
FIG. 4 is a functional block diagram of an apparatus for collecting open source information according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device for implementing a method for collecting open source information according to an embodiment of the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The method aims to solve the problem that in the method for acquiring open source information provided in the prior art, the efficiency of program operation starting and program operation state monitoring is low in the process of acquiring the open source information. The invention provides a method for acquiring open source information, which is characterized in that an open source information acquisition framework comprising a process manager and a monitor is constructed, and the open source information is acquired by utilizing the open source information acquisition framework, so that the efficiency of program operation starting and program operation state monitoring in the open source information acquisition process can be improved.
Referring to fig. 1, a flowchart of a method for collecting open source information according to an embodiment of the present invention is shown. In this embodiment, the method for collecting open source information includes:
s1, constructing an open source information acquisition framework, wherein the open source information acquisition framework comprises a process manager and a monitor.
In one embodiment, the open source information acquisition architecture comprises at least one process manager and at least one monitor, wherein the process manager refers to a Supervisor framework, and all acquisition processes are managed by using the Supervisor framework; the monitor refers to a Watchdog, from which to monitor file system events; and the management of all the acquisition programs on the server is realized according to the process manager and the monitor, so that the stability and the flexibility of the acquisition programs are ensured.
Further, the process manager, i.e., the Supervisor framework, is a high performance process management tool of python, which can start, stop, restart, and monitor multiple processes on a Linux system. The main function of the process manager is to ensure that the application program can be restarted automatically under the condition of breakdown or abnormality so as to maintain the stability and reliability of the system; the process to be managed can be defined by a simple configuration file, and once the configuration is completed, the Supervisor framework will automatically monitor the process.
Further, the monitor, i.e., watchdog, is a library of python for monitoring file system events, which may trigger custom operations when a file or directory changes, where the operations include creation, deletion, modification, movement, etc. of the file or directory.
In one embodiment, an open source information acquisition architecture is constructed comprising: generating an initial process manager according to a pre-acquired process management program code; performing file configuration and process configuration on an initial process manager to obtain a process configuration file and a management process, and setting a management command of the initial process manager; filling the initial process manager according to the process configuration file, the management process and the management command to obtain a process manager; setting an installation environment, installing a database according to the installation environment, and carrying out parameter modification on the connection configuration in the database to obtain an updated database; adding a search engine in the updated database to obtain a monitor; and generating an open source information acquisition architecture according to the process manager and the monitor.
In one embodiment, the process manager code refers to the installation code of the initial process manager; program installation can be performed according to the installation code, specifically, the process management program code is installed by using a preset operation command, wherein the operation command can be sudo yum install supervisor, and an initial execution manager is obtained.
In one embodiment, file configuration refers to installing/etc/supervisord. Conf process configuration files on an initial process manager; the process configuration refers to configuring a service name, a service port, a starting directory, a log file and the like of an initial process manager to obtain a management process of the initial process manager; the management commands include a view all tasks status command (supervisorctl status), a close all tasks command (supervisorctl shutdown), a start tasks command (supervisorctl start), etc.; and combining the process configuration file, the management process and the management command with the initial process manager to obtain the process manager.
In one embodiment, the installation environment refers to a preset condition for installing a database, such as an operating system, a hardware size, a memory, and the like; installing a database by using a preset installation command based on an installation environment, wherein the database comprises a web database and a client database; the install command refers to sudo apt install postgresql postgresql-conctrieb-y; the connection configuration refers to configuration of TCP/IP parameters in a database, including IP configuration, subnet mask configuration, default gateway configuration and the like; the search engine refers to a shadan api search engine, and an updated database added by the search engine is used as a monitor.
In one embodiment, the process manager is integrated with the monitor to obtain the open source information acquisition architecture, so that the stability and flexibility of the open source information acquisition architecture can be improved.
S2, configuring service parameters and file catalogues on a process manager, and creating configuration files according to the file catalogues and pre-acquired open source information acquisition requirements.
Referring to FIG. 2, in one embodiment, configuring service parameters and file directories on a process manager includes: s21, acquiring a process configuration file on a process manager, and modifying information of the process configuration file to obtain a modified file; s22, setting service parameters of a process manager according to the modification file; s23, setting a file storage path, a file management command and a file directory structure; s24, generating a file directory according to the file storage path, the file management command and the file directory structure.
In one embodiment, modifying the process configuration file refers to modifying information such as a service port, a log path, a configuration file path and the like in the process configuration file to obtain a modified file; further, an instruction corresponding to the information in the modification file is taken as a service parameter.
In one embodiment, the file storage path refers to a storage address corresponding to a file, and includes an absolute path and a relative path, wherein the absolute path refers to a path from the beginning, and the relative path refers to a path from the current directory; the file management commands include a tree command (view file command), a touch command (create file command), a mkdir command (create directory command), a stat command (view file time command), and the like.
Further, the file directory structure may be a tree structure, that is, the file directory is built in the form of a tree structure, that is, the root node is connected with the child nodes to obtain the file directory structure, where the root node includes a root directory (a directory related to a boot system),/usr directory (a directory related to software installation and execution), and/or var directory (a directory related to a system operation process); and summarizing according to the file storage path, the file management command and the file directory structure to obtain the file directory.
In one embodiment, creating a configuration file according to a file directory and a pre-acquired open source information acquisition requirement includes: carrying out demand classification on the open source information acquisition demands to obtain demand types; acquiring a file task, extracting acquisition function information of the file task, and carrying out secondary division on the requirement type based on the acquisition function information to obtain a task type; defining task parameters according to task types, and selecting a task catalog from the file catalog according to the task parameters; and generating a configuration file according to the demand type, the task parameters and the task catalog.
In one embodiment, a preset association algorithm can be utilized to classify the requirements of open source information acquisition to obtain a requirement type, wherein the association algorithm can be a Prefixspan algorithm or a Freespan algorithm; the requirement types include Maven component collection, python component collection and the like.
Further, the file task refers to a preset activity completed by the open source information acquisition architecture, and can be a process or a thread; collecting function information refers to the corresponding function of the file task; and further classifying the requirement types according to the acquisition function information to obtain task types, wherein the task types comprise a component name acquisition task, a component detail acquisition task, an acquisition keyword issuing task and the like.
In one embodiment, task parameters such as a task name, a task execution command, a task output log path and the like corresponding to the configuration file are set according to the task type; matching the task parameters with the file catalogs to obtain task catalogs corresponding to the file tasks; and integrating the requirement type, the task parameters and the task catalog to obtain a configuration file.
In one embodiment, service parameters and file directories are configured on the process manager, so that the integrity of the process manager can be improved; and creating a configuration file according to the file directory and the open source information acquisition requirement, so that the accuracy of the configuration file can be ensured.
And S3, compiling a management subprogram by using a monitor, and monitoring the task running state of the configuration file according to the management subprogram.
Referring to FIG. 3, in one embodiment, a management subroutine is written using a monitor, comprising: s31, extracting a file task and a file path corresponding to the configuration file in the process manager by using a monitor; s32, setting a snapshot plan corresponding to the file path; s33, integrating the file task, the file path and the snapshot plan to obtain a management subroutine.
In one embodiment, the file tasks, the file paths and the snapshot plans in the management subroutine are in a one-to-one correspondence; acquiring a data interface in a process manager by using a monitor, and extracting a file task and a file path corresponding to a configuration file according to the data interface; setting plan parameters such as the period, the frequency, the speed and the like of snapshot execution at the position of a file path, and integrating according to the plan parameters to obtain a snapshot plan; integrating the file task, the file path and the snapshot plan refers to inserting a snapshot technology at the position of the file path, generating a plurality of monitoring applets corresponding to the file task according to the file task corresponding to the file path and the snapshot plan, and taking the plurality of monitoring applets as management subroutines.
In one embodiment, monitoring the task running state of the configuration file according to the management subroutine includes: sequentially intercepting path snapshots of file paths corresponding to two configuration files by using a management subprogram according to a preset time interval, and judging whether the path snapshots are consistent; when the two path snapshots are consistent, judging that the configuration file is in an un-updated state, extracting a file state corresponding to the configuration file, and taking the file state as a task running state; when the two path snapshots are inconsistent, judging that the configuration file is in an updated state, extracting a file state corresponding to the updated configuration file, and taking the file state as a task running state.
In one embodiment, a configuration file is read according to a management subroutine, wherein the configuration file comprises a file path corresponding to the configuration file; and intercepting the file paths in sequence according to the time interval of the set snapshots by utilizing the corresponding snapshot plans in the management subprogram, and monitoring the difference between the front snapshot and the rear snapshot.
Further, judging whether the file content corresponding to the configuration file is changed according to the difference, and when the file content is unchanged, indicating that the path snapshot is unchanged, namely kept consistent, wherein the configuration file is not updated; when the file content is changed, the path snapshot is changed, namely inconsistency occurs, and the configuration file is updated; at this time, according to the file path corresponding to the configuration file, the information interface provided by the process manager is used to obtain the file state of the configuration file, where the file state includes states such as RUNNING, stop, failure, etc.
In one embodiment, the monitor is used to accurately write the management sub-program, thereby ensuring the accuracy of the management sub-program; the configuration file is monitored through the management subprogram, so that the task running state can be accurately obtained, and the abnormal condition of the configuration file can be timely found.
And S4, updating the open source information acquisition architecture according to the task running state and the service parameters, and acquiring the open source information by using the updated open source information acquisition architecture.
In one embodiment, updating the open source information acquisition architecture according to the task running state and the service parameters includes: extracting abnormal task running states in the task running states according to the information interfaces and the service parameters corresponding to the process manager; correcting the abnormal task running state by using the service parameters to obtain a corrected task running state; and replacing the abnormal task running state in the open source information acquisition architecture by using the corrected task running state to obtain an updated open source information acquisition architecture.
In one embodiment, the information interface may be a Micro USB interface, a USB Type-c interface, a lighting interface, etc.; the abnormal task running state can be extracted by adopting a mode corresponding to the information interface or a command corresponding to the service parameter in the Web interface of the Web page, wherein the abnormal task running state comprises a restarting update state, a starting failure state and the like.
In one embodiment, the abnormal task running state is corrected by using a correction command in the service parameters to obtain the corrected task running state, wherein the correction command comprises a start command, a restart command, a stop command and the like; further, the open source information acquisition architecture can be restarted by updating the code files mapped on the server through tools such as FTP, SFTP and the like, so that the open source information acquisition architecture is obtained.
In one embodiment, acquiring the open source information by using the open source information acquisition architecture refers to managing a task process corresponding to the open source information according to a process manager and monitoring a task state corresponding to the open source information according to a monitor, wherein managing the task process comprises adding, deleting, changing, searching and other management operations; monitoring the task state refers to timely finding out abnormality of a configuration file corresponding to open source information and guaranteeing stable operation of an open source information acquisition framework.
In one embodiment, the open source information acquisition architecture is updated according to the task running state and the service parameters, so that the open source information acquisition architecture can be accurately obtained, and the stability and the flexibility of the open source information acquisition architecture are ensured; the open source information is acquired by using the open source information acquisition framework, so that the writing and executing efficiency of an acquisition program can be improved, and the time cost is reduced.
According to the embodiment of the invention, the open source information acquisition architecture is constructed, so that the efficiency of acquisition and development from the multidimensional optimization program can be acquired, and the open source information can be acquired rapidly and comprehensively; by configuring service parameters and file catalogues on the process manager, the integrity of the process manager can be improved; the configuration file is created through the file catalogue and the open source information acquisition requirement, so that the accuracy of the configuration file can be ensured; the management subprogram is written by the monitor, so that the efficiency of starting the program operation and the efficiency of monitoring the program operation in the open source information acquisition process can be improved; according to the monitoring configuration file of the management subprogram, the task running state can be accurately obtained; the open source information acquisition architecture is updated through the task running state and the service parameters to obtain the open source information acquisition architecture, and the open source information is acquired by utilizing the open source information acquisition architecture, so that the rapid iteration, updating and expansion of the open source information acquisition task can be realized, and meanwhile, the running stability and flexibility of an acquisition system are ensured. Therefore, the method for acquiring the open source information can solve the problem that the efficiency of program operation starting and program operation state monitoring is low in the open source information acquisition process.
Fig. 4 is a functional block diagram of an apparatus for collecting open source information according to an embodiment of the present invention.
The apparatus 400 for collecting open source information according to the present invention may be installed in an electronic device. Depending on the functions implemented, the apparatus 400 for collecting open source information may include a collection architecture construction module 401, a configuration file generation module 402, an operation state monitoring module 403, and an open source information collection module 404. The module of the invention, which may also be referred to as a unit, refers to a series of computer program segments, which are stored in the memory of the electronic device, capable of being executed by the processor of the electronic device and of performing a fixed function.
In the present embodiment, the functions concerning the respective modules/units are as follows: the acquisition architecture construction module 401 is configured to construct an open source information acquisition architecture, where the open source information acquisition architecture includes a process manager and a monitor; the configuration file generation module 402 is configured to configure service parameters and file directories on the process manager, and create a configuration file according to the file directories and the pre-acquired open source information acquisition requirements; an operation state monitoring module 403, configured to write a management subroutine by using the monitor, and monitor a task operation state of the configuration file according to the management subroutine; the open source information acquisition module 404 is configured to update an open source information acquisition architecture according to the task running state and the service parameter, and acquire open source information by using the updated open source information acquisition architecture.
In detail, each module in the apparatus 400 for collecting open source information in one embodiment adopts the same technical means as the method for collecting open source information in the drawings when in use, and can produce the same technical effects, which are not described herein.
Fig. 5 is a schematic structural diagram of an electronic device for implementing a method for collecting open source information according to an embodiment of the present invention.
Electronic device 500 may include a processor 501, memory 502, a communication bus 503, and a communication interface 504, and may also include computer programs stored in memory 502 and executable on processor 501, such as programs that gather open source information.
The processor 501 may be formed by an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be formed by a plurality of integrated circuits packaged with the same function or different functions, including one or more central processing units (Central Processing Unit, CPU), a microprocessor, a digital processing chip, a combination of a graphics processor and various control chips, etc. The processor 501 is a Control Unit (Control Unit) of the electronic device, connects various components of the entire electronic device using various interfaces and lines, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory 502 (e.g., executing programs or the like for collecting open source information), and calling data stored in the memory 502.
Memory 502 includes at least one type of readable storage medium including flash memory, a removable hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 502 may in some embodiments be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. The memory 502 may also be an external storage device of the electronic device in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the electronic device. Further, the memory 502 may also include both internal storage units and external storage devices of the electronic device. The memory 502 may be used not only for storing application software installed in an electronic device and various types of data, such as code of a program for collecting open source information, but also for temporarily storing data that has been output or is to be output.
The communication bus 503 may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, or the like. The bus may be classified as an address bus, a data bus, a control bus, etc. The bus is arranged to enable connected communication between the memory 502 and the at least one processor 501 etc.
The communication interface 504 is used for communication between the electronic device and other devices described above, including network interfaces and user interfaces. Optionally, the network interface may include a wired interface and/or a wireless interface (e.g., WI-FI interface, bluetooth interface, etc.), typically used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a Display (Display), an input unit such as a Keyboard (Keyboard), or alternatively a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, or the like. The display may also be referred to as a display screen or display unit, as appropriate, for displaying information processed in the electronic device and for displaying a visual user interface.
Fig. 5 illustrates only an electronic device having components, and it will be appreciated by those skilled in the art that the configuration illustrated in fig. 5 is not limiting of electronic device 500 and may include fewer or more components than illustrated, or may combine certain components, or a different arrangement of components.
For example, although not shown, the electronic device may further include a power source (such as a battery) for powering the respective components, and the power source may be logically connected to the at least one processor 501 through a power management device, so as to perform functions of charge management, discharge management, and power consumption management through the power management device. The power supply may also include one or more of any of a direct current or alternating current power supply, recharging device, power failure detection circuit, power converter or inverter, power status indicator, etc. The electronic device may also include various sensors, bluetooth modules, wi-Fi modules, etc., which are not described in detail herein.
It should be understood that the examples are for illustrative purposes only and are not limited to this configuration in the scope of the patent application.
In particular, the specific implementation method of the above instruction by the processor 501 may refer to the description of the relevant steps in the corresponding embodiment of the drawings, which is not repeated herein.
Further, the integrated modules/units of the electronic device 500 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as a stand alone product. The computer readable storage medium may be volatile or nonvolatile. For example, the computer readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM).
The invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program can realize the method for collecting open source information in any embodiment when being executed by a processor. The computer-readable storage medium may be volatile or nonvolatile. For example, the computer readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM).
In the several embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of modules is merely a logical function division, and other manners of division may be implemented in practice.
The modules illustrated as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional module in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.
The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The embodiment of the application can acquire and process the related data based on the artificial intelligence technology. Among these, artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use knowledge to obtain optimal results.
Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. A plurality of units or means recited in the system claims can also be implemented by means of software or hardware by means of one unit or means. The terms first, second, etc. are used to denote a name, but not any particular order.
Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.
Claims (10)
1. A method of collecting open source information, the method comprising:
constructing an open source information acquisition architecture, wherein the open source information acquisition architecture comprises a process manager and a monitor, the process manager is a Supervisor framework, and the monitor is a library for monitoring file system events;
configuring service parameters and file catalogues on the process manager, and creating configuration files according to the file catalogues and pre-acquired open source information acquisition requirements;
writing a management subprogram by using the monitor, and monitoring the configuration file according to the management subprogram to obtain a task running state;
and updating the open source information acquisition architecture according to the task running state and the service parameters, and acquiring open source information by using the updated open source information acquisition architecture.
2. The method for collecting open source information according to claim 1, wherein said constructing an open source information collecting architecture comprises:
generating an initial process manager according to a pre-acquired process management program code;
performing file configuration and process configuration on the initial process manager to obtain a process configuration file and a management process, and setting a management command of the initial process manager;
filling the initial process manager according to the process configuration file, the management process and the management command to obtain a process manager;
setting an installation environment, installing a database according to the installation environment, and carrying out parameter modification on connection configuration in the database to obtain an updated database;
adding a search engine into the updated database to obtain a monitor;
and generating an open source information acquisition framework according to the process manager and the monitor.
3. The method for collecting open source information according to claim 1, wherein said configuring service parameters and file directories on said process manager comprises:
acquiring a process configuration file on the process manager, and modifying information of the process configuration file to obtain a modified file;
setting service parameters of the process manager according to the modification file;
setting a file storage path, a file management command and a file directory structure;
and generating a file directory according to the file storage path, the file management command and the file directory structure.
4. The method for collecting open source information according to claim 1, wherein creating a configuration file according to the file directory and a pre-acquired open source information collection requirement comprises:
carrying out demand classification on the open source information acquisition demands to obtain demand types;
acquiring a file task, extracting acquisition function information of the file task, and carrying out secondary division on the requirement type based on the acquisition function information to obtain a task type;
defining task parameters according to the task types, and selecting a task catalog from the file catalog according to the task parameters;
and generating a configuration file according to the requirement type, the task parameters and the task catalog.
5. The method for collecting open source information according to claim 1, wherein said writing a management subroutine with said monitor comprises:
extracting a file task and a file path corresponding to the configuration file in the process manager by using the monitor;
setting a snapshot plan corresponding to the file path;
and integrating the file task, the file path and the snapshot plan to obtain a management subprogram.
6. The method for collecting open source information according to claim 1, wherein said monitoring said configuration file according to said management subroutine to obtain a task running state comprises:
sequentially intercepting path snapshots of file paths corresponding to two configuration files by using the management subprogram according to a preset time interval, and judging whether the path snapshots are consistent;
when the two path snapshots are consistent, judging that the configuration file is in an un-updated state, extracting a file state corresponding to the configuration file, and taking the file state as a task running state;
when the two path snapshots are inconsistent, judging that the configuration file is in an updated state, extracting a file state corresponding to the updated configuration file, and taking the file state as a task running state.
7. The method of any of claims 1 to 6, wherein updating the open source information collection architecture according to the task running state and the service parameters comprises:
extracting abnormal task running states in the task running states according to the information interfaces corresponding to the process manager and the service parameters;
correcting the abnormal task running state by utilizing the service parameters to obtain a corrected task running state;
and replacing the abnormal task running state in the open source information acquisition architecture by using the corrected task running state to obtain an updated open source information acquisition architecture.
8. An apparatus for collecting open source information, the apparatus comprising:
the system comprises an acquisition architecture construction module, a file system event monitoring module and a file system event monitoring module, wherein the acquisition architecture construction module is used for constructing an open source information acquisition architecture, the open source information acquisition architecture comprises a process manager and a monitor, the process manager is a Supervisor frame, and the monitor is a library for monitoring file system events;
the configuration file generation module is used for configuring service parameters and file catalogues on the process manager and creating configuration files according to the file catalogues and the pre-acquired open source information acquisition requirements;
the running state monitoring module is used for compiling a management subprogram by using the monitor, and monitoring the configuration file according to the management subprogram to obtain a task running state;
and the open source information acquisition module is used for updating the open source information acquisition framework according to the task running state and the service parameters, and acquiring the open source information by utilizing the updated open source information acquisition framework.
9. An electronic device, the electronic device comprising:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the method of collecting open source information as claimed in any one of claims 1 to 7.
10. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the method of collecting open source information according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311004574.1A CN116719702B (en) | 2023-08-10 | 2023-08-10 | Method and device for collecting open source information, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311004574.1A CN116719702B (en) | 2023-08-10 | 2023-08-10 | Method and device for collecting open source information, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116719702A CN116719702A (en) | 2023-09-08 |
| CN116719702B true CN116719702B (en) | 2024-03-26 |
Family
ID=87871995
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311004574.1A Active CN116719702B (en) | 2023-08-10 | 2023-08-10 | Method and device for collecting open source information, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116719702B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6457142B1 (en) * | 1999-10-29 | 2002-09-24 | Lucent Technologies Inc. | Method and apparatus for target application program supervision |
| US8739161B1 (en) * | 2004-12-13 | 2014-05-27 | Infoblox Inc. | Process management |
| CN104486445A (en) * | 2014-12-30 | 2015-04-01 | 北京天云融创软件技术有限公司 | Distributed extendable resource monitoring system and method based on cloud platform |
| CN106844137A (en) * | 2016-12-08 | 2017-06-13 | 腾讯科技（深圳）有限公司 | The monitoring method and device of server |
| CN112732528A (en) * | 2021-01-08 | 2021-04-30 | 卓望数码技术（深圳）有限公司 | Index acquisition method, system, equipment and storage medium based on IT operation and maintenance monitoring |
| CN113608765A (en) * | 2021-08-05 | 2021-11-05 | 北京华云安信息技术有限公司 | Data processing method, device, equipment and storage medium |
| CN114584464A (en) * | 2022-03-07 | 2022-06-03 | 浪潮云信息技术股份公司 | Cloud platform full-automatic management log collection method and terminal |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11150950B2 (en) * | 2016-12-01 | 2021-10-19 | Vmware, Inc. | Methods and apparatus to manage workload domains in virtual server racks |
-
2023
- 2023-08-10 CN CN202311004574.1A patent/CN116719702B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6457142B1 (en) * | 1999-10-29 | 2002-09-24 | Lucent Technologies Inc. | Method and apparatus for target application program supervision |
| US8739161B1 (en) * | 2004-12-13 | 2014-05-27 | Infoblox Inc. | Process management |
| CN104486445A (en) * | 2014-12-30 | 2015-04-01 | 北京天云融创软件技术有限公司 | Distributed extendable resource monitoring system and method based on cloud platform |
| CN106844137A (en) * | 2016-12-08 | 2017-06-13 | 腾讯科技（深圳）有限公司 | The monitoring method and device of server |
| CN112732528A (en) * | 2021-01-08 | 2021-04-30 | 卓望数码技术（深圳）有限公司 | Index acquisition method, system, equipment and storage medium based on IT operation and maintenance monitoring |
| CN113608765A (en) * | 2021-08-05 | 2021-11-05 | 北京华云安信息技术有限公司 | Data processing method, device, equipment and storage medium |
| CN114584464A (en) * | 2022-03-07 | 2022-06-03 | 浪潮云信息技术股份公司 | Cloud platform full-automatic management log collection method and terminal |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116719702A (en) | 2023-09-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102323945B (en) | SQL (Structured Query Language)-based database management method and device | |
| US8255899B2 (en) | Techniques for upgrade dependency management | |
| US8601460B2 (en) | Systems and methods for firmware cloning | |
| US10671061B2 (en) | Devices, methods, and systems for a distributed rule based automated fault detection | |
| CN104679500B (en) | Method and device for realizing automatic generation of entity class | |
| EP2808807B1 (en) | Data structures migration and upgrade | |
| CN113238924B (en) | Chaotic engineering realization method and system in distributed graph database system | |
| CN109857649B (en) | Resource testing method and system | |
| CN110737594A (en) | Database standard conformance testing method and device for automatically generating test cases | |
| CN111694612A (en) | Configuration checking method, device, computer system and storage medium | |
| CN110825409B (en) | Parameter management method, electronic device and computer storage medium | |
| Nagavaram et al. | A cloud-based dynamic workflow for mass spectrometry data analysis | |
| CN111400332A (en) | Project progress monitoring method and device | |
| CN113986237A (en) | Method and device for creating Jenkins compiling task | |
| CN116719702B (en) | Method and device for collecting open source information, electronic equipment and storage medium | |
| CN110580253B (en) | Time sequence data set loading method and device, storage medium and electronic equipment | |
| CN112256277A (en) | Software compiling method, system, electronic device and storage medium | |
| CN114816470A (en) | Metadata database management method and device, electronic equipment and medium | |
| CN115827179A (en) | Computing power scheduling method, device, equipment and storage medium for physical machine equipment | |
| CN114936111A (en) | Front-end variable anomaly detection and restoration method, device, equipment and storage medium | |
| CN114153457A (en) | Application program installation file generation method and device and storage medium | |
| CN114048070A (en) | Data batch backup method, device, equipment and storage medium | |
| Fördős et al. | CRDTs for the configuration of distributed Erlang systems | |
| CN115292194B (en) | Method for debugging flow, electronic equipment and computer readable storage medium | |
| CN113934573B (en) | Data recovery method and device for memory database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |