CN115510112A - Data reporting method and device, storage medium and electronic equipment - Google Patents

Data reporting method and device, storage medium and electronic equipment Download PDF

Info

Publication number
CN115510112A
CN115510112A CN202211358055.0A CN202211358055A CN115510112A CN 115510112 A CN115510112 A CN 115510112A CN 202211358055 A CN202211358055 A CN 202211358055A CN 115510112 A CN115510112 A CN 115510112A
Authority
CN
China
Prior art keywords
data
reporting
task
platform
reported
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211358055.0A
Other languages
Chinese (zh)
Inventor
郭继琼
海洋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Topsec Technology Co Ltd
Beijing Topsec Network Security Technology Co Ltd
Beijing Topsec Software Co Ltd
Original Assignee
Beijing Topsec Technology Co Ltd
Beijing Topsec Network Security Technology Co Ltd
Beijing Topsec Software Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Topsec Technology Co Ltd, Beijing Topsec Network Security Technology Co Ltd, Beijing Topsec Software Co Ltd filed Critical Beijing Topsec Technology Co Ltd
Priority to CN202211358055.0A priority Critical patent/CN115510112A/en
Publication of CN115510112A publication Critical patent/CN115510112A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24568Data stream processing; Continuous queries
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25Integrating or interfacing systems involving database management systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/284Relational databases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

Some embodiments of the present application provide a method, an apparatus, a storage medium, and an electronic device for reporting data, where the method includes: acquiring a task configuration file and a reporting time period corresponding to a data reporting task, wherein the task configuration file comprises: reporting the task type, the data query interface, the data content and the platform address; when the reporting time period is confirmed to be reached, starting a task thread corresponding to the reporting task type through the data query interface; and acquiring an execution result of the task thread, acquiring target data of a platform to be reported corresponding to the reported data content, and reporting the target data to a target platform corresponding to the reported platform address. Some embodiments of the application can realize automatic management of data reporting tasks, and have high flexibility.

Description

Data reporting method and device, storage medium and electronic equipment
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a method and an apparatus for reporting data, a storage medium, and an electronic device.
Background
The flow in the network needs to be monitored and analyzed in order to maintain the network security, which is beneficial for enterprises or platforms to monitor the network security state and improve the risk response capability.
Because the network platform terminals and the areas distributed by each platform are different from the cooperative operators, the information reported by the data required by each area is different. At present, different task flows are developed for different network platforms in different regions to realize data reporting. However, the number of network platforms is large, and the workload of research and development personnel is large, so that the working efficiency is low, and the automatic control of data reporting cannot be realized.
Therefore, how to provide a technical scheme of an efficient data reporting method becomes a technical problem which needs to be solved urgently.
Disclosure of Invention
Some embodiments of the present application are directed to a method, an apparatus, a storage medium, and an electronic device for reporting data, according to the technical scheme of the embodiment of the application, automatic control over data reporting can be achieved through the method of the task configuration file, the workload of research and development personnel is reduced, and the working efficiency is improved.
In a first aspect, some embodiments of the present application provide a method for reporting data, including: acquiring a task configuration file and a reporting time period corresponding to a data reporting task, wherein the task configuration file comprises: reporting the task type, the data query interface, the data content and the platform address; when the report time period is confirmed to be reached, starting a task thread corresponding to the report task type through the data query interface; and acquiring an execution result of the task thread, acquiring target data of a platform to be reported corresponding to the reported data content, and reporting the target data to a target platform corresponding to the reported platform address.
According to some embodiments of the application, corresponding information is configured in the task configuration file, a data reporting task can be completed according to the information in the task configuration file, and target data are sent to the corresponding target platform, so that automatic control of data reporting is realized, workload of research and development personnel is reduced, working efficiency is improved, the task configuration file is easy to manage and maintain, and adaptability is good.
In some embodiments, before the obtaining the task configuration file and the reporting time period corresponding to the data reporting task, the method further includes: under the condition that the reporting state is the opening state, adding the task corresponding to the opening state into a data reporting task list; traversing the data reporting task list to obtain the data reporting task, wherein the data reporting task is one of the data reporting task list.
Some embodiments of the application may obtain the task in the open state to form a data reporting task list by monitoring the reporting state, so as to obtain the data reporting task, and may implement real-time monitoring of the data reporting task.
In some embodiments, before adding the task corresponding to the on state to a data reporting task list, the method further includes: within a preset time, the following operations are executed in a circulating mode: confirming and opening province codes of services to be reported; and judging the reporting state corresponding to the task under the provincial code, wherein the reporting state comprises the following steps: the open state and the closed state.
Some embodiments of the application perform polling monitoring on the province code and the task under the province code within a preset time, so that the data reporting task can be obtained in time, and real-time monitoring on the network is realized.
In some embodiments, the obtaining of the task configuration file corresponding to the data reporting task includes: acquiring a task configuration label corresponding to the data reporting task; and searching the task configuration file corresponding to the task configuration label.
According to some embodiments of the application, the task configuration file corresponding to the data reporting task is obtained by determining the task configuration tag, so that the automatic control of the data reporting task flow can be realized.
In some embodiments, the starting, by the data query interface, a task thread corresponding to the reporting task type includes: reading the mapping relation of the data query interface in the task configuration file, and creating a sub-thread; and sending the reported data content to the sub-thread so as to facilitate the sub-thread to start the corresponding task thread.
According to some embodiments of the application, the sub-threads can be automatically created through the interface mapping relation in the task configuration file, so that the task threads corresponding to the data reporting task can be started, the increase of resource internal consumption caused by the starting of unnecessary sub-threads is effectively avoided, and the execution efficiency of the data reporting task is ensured.
In some embodiments, the obtaining the execution result of the task thread to obtain the target data of the platform to be reported corresponding to the reported data content includes: acquiring original data of the platform to be reported through the task thread; and processing the original data to obtain the target data.
According to some embodiments of the application, the target data is obtained by processing the obtained original data, so that unified and standardized management of the data is realized.
In some embodiments, reporting the data content includes: at least one of a data source, a data type, a data field name, a data field type, and a data field value.
According to some embodiments of the application, unified and standardized management of data can be realized by configuring the reported data content in the task configuration file.
In some embodiments, the processing the raw data to obtain the target data includes: writing the original data into a file according to a data format in the task configuration file to obtain the target data; or, the processing data is arranged into a preset data stream form to obtain the target data.
According to some embodiments of the application, target data is obtained by writing or sorting original data into a preset data stream form, so that unified standardized management of the data is realized.
In some embodiments, the reporting the target data to the target platform corresponding to the reporting platform address includes: reading the reporting platform address, wherein the reporting platform address includes: at least one of the area of the platform, the destination address of the platform, the port information and the reported directory; and sending the target data to the target platform according to the reported platform address.
According to some embodiments of the application, the target data is directly sent to the target platform through the information in the reporting platform address in the task configuration file, so that the target data is accurately reported, and the method is simple and efficient.
In some embodiments, the method further comprises: and updating the task configuration file in response to an input operation instruction of a user.
According to some embodiments of the application, when the information in the task configuration file is sent and changed, the content in the task configuration file can be updated and expanded through the operation of a user (or a research and development staff), and the method and the system are high in expandability, maintainability and flexibility.
In a second aspect, some embodiments of the present application provide an apparatus for data reporting, including: an obtaining module, configured to obtain a task configuration file and a reporting time period corresponding to a data reporting task, where the task configuration file includes: reporting the task type, the data query interface, the data content and the platform address; the starting module is configured to start a task thread corresponding to the reporting task type through the data query interface when the reporting time period is determined to be reached; and the reporting module is configured to acquire an execution result of the task thread, obtain target data of a platform to be reported corresponding to the reported data content, and report the target data to a target platform corresponding to the reported platform address.
In a third aspect, some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, which when executed by a processor, may implement the method according to any of the embodiments of the first aspect.
In a fourth aspect, some embodiments of the present application provide an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program may implement the method according to any of the embodiments of the first aspect.
In a fifth aspect, some embodiments of the present application provide a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to any of the embodiments of the first aspect.
Drawings
In order to more clearly illustrate the technical solutions of some embodiments of the present application, the drawings that are required to be used in some embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that for a person skilled in the art, other relevant drawings can be obtained from these drawings without inventive effort.
Fig. 1 is a system diagram of data reporting according to some embodiments of the present application;
fig. 2 is a flowchart of a method for reporting data according to some embodiments of the present application;
fig. 3 is a second flowchart of a data reporting method according to some embodiments of the present application;
fig. 4 is a block diagram illustrating an apparatus for reporting data according to some embodiments of the present disclosure;
fig. 5 is a schematic diagram of an electronic device according to some embodiments of the present application.
Detailed Description
The technical solutions in some embodiments of the present application will be described below with reference to the accompanying drawings in some embodiments of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
In the related art, since there are many network platforms (also referred to as "platforms"), and the provinces distributed by each network platform are different from the cooperating operators, the data types, data formats, reporting modes, etc. required by each province platform are different. If a set of reporting process is realized for different platforms of different provinces, the workload is increased, and the updating and expanding difficulty is increased. At present, in the prior art, in the process of reporting data, binary log data in a service database is first obtained, a service data object is generated by analyzing field attributes of the binary log data and the service data, and finally the service data object is written into a data bus after being processed. As can be seen from the above related technologies, the emphasis in the prior art is to analyze the binary logs in real time, which does not support the diversification of data sources, and the output data format is single, which is not suitable for the diversification and automation control of the whole reporting process. And is not suitable for scenes with more business objects and larger demand difference. When the number of the docking platforms is continuously expanded and the demand is continuously increased, the functions of the reporting system need to be continuously updated, modification and expansion of the functions inevitably affect other functions, and the expandability is poor.
In view of this, some embodiments of the present application provide a data reporting method, which can implement control over the entire process of a data reporting task by configuring different attribute information in a task configuration file. The task configuration file can be connected with a plurality of platforms to be reported or target platforms, and a plurality of different data reporting tasks, so that the unified management of the data reporting tasks of different provinces of different platforms is realized, and the automatic control of data reporting is realized. And the reporting task can be modified, expanded or deleted through a task configuration mode, and the whole reporting system is more flexible and reliable.
As shown in fig. 1, some embodiments of the present application provide a data reporting system, which includes an enterprise platform 100 (as a specific example of a platform to be reported), a network security device 200, and a target platform 300. The enterprise platform 100 and the network security appliance 200 may be connected through a wired or wireless network to enable communication, and the network security appliance 200 and the target platform 300 may communicate through a server. The network security device 200 queries the data reporting task, and after the reporting time period is reached, the network security device 200 may obtain the target data corresponding to the data reporting task in the enterprise platform 100, and finally report the target data to the target platform 300.
It should be noted that, in some embodiments of the present application, since a plurality of task configuration files corresponding to a plurality of data reporting tasks are deployed in the network security device 200, automatic control of the data reporting tasks may be implemented. One data reporting task corresponds to one task configuration label, and one task configuration label corresponds to one task configuration file, so that unified management of the data reporting tasks is realized. The method and the device have the advantages that different data reporting tasks can be completely managed and governed by corresponding task configuration files, the method and the device are simple and efficient, codes are greatly reduced, and reusability of the codes is improved.
The following describes, by way of example, implementation procedures of data reporting performed by the network security device 200 according to some embodiments of the present application with reference to fig. 2.
Referring to fig. 2, fig. 2 is a flowchart of a data reporting method according to some embodiments of the present application, where the data reporting method includes: s210, acquiring a task configuration file and a reporting time period corresponding to a data reporting task, wherein the task configuration file comprises: reporting the task type, the data query interface, the data content and the platform address; s220, when the report time period is confirmed to be reached, starting a task thread corresponding to the report task type through the data query interface; and S230, acquiring the execution result of the task thread, acquiring target data of a platform to be reported corresponding to the reported data content, and reporting the target data to a target platform corresponding to the reported platform address.
For example, in some embodiments of the present application, the network security appliance 200 may be pre-deployed with a task profile. The task configuration file includes key information required by the data reporting task, such as a reporting task type, a data query interface, a reporting data content, a reporting data format (which may also be referred to as a "data format") reporting information description, a reporting platform address, and the like. It should be appreciated that the content in the task configuration file may be flexibly extended, and embodiments of the present application are not limited thereto. It should be noted that the reporting time period may be specifically set according to the actual situation, and may be 30 seconds, 10 minutes, 30 minutes, and the like, and the application is not specifically limited herein. If the time length from the initial time of starting the data reporting task to the current time is not less than the reporting time period, it is determined that the reporting time period is reached, and then a corresponding event thread (as a specific example of a task thread) may be started, and the network security device 200 executes the event thread, acquires the related target data of the enterprise platform 100, and reports the target data to the target platform 300.
In some embodiments of the present application, the method further comprises: within a preset time, the following operations are executed in a circulating mode: confirming and opening province codes of services to be reported; and judging the reporting state corresponding to the task under the provincial code, wherein the reporting state comprises the following steps: the open state and the closed state.
For example, in some embodiments of the present application, the network security device 200 may confirm the province code for starting the service to be reported every second (as a specific example of a preset time), sleep for 1 second and repeat the query again if the province code for starting the service to be reported is not started, and acquire the province ID (that is, the province code) if the province code for starting the service to be reported is queried, and query the configuration tag corresponding to the province ID. The configuration tag is used for representing the data which needs to be reported by each province, such as stiff wood worm event data, vulnerability data and the like. And then, confirming the state of the inquiry button (namely judging the reporting state corresponding to the configuration task under the province ID) based on the configuration label. If the button is in an open state, the reporting state of the representation configuration task is in an open state, namely the task to be reported exists, otherwise, the button is in a closed state. In the case of the off state, the sleep is performed for 1 second to confirm the state of the inquiry button again. It should be noted that the preset time may be specifically set according to actual situations, and the application is not specifically limited herein.
In some embodiments of the present application, before performing S210, the method for reporting data further includes: under the condition that the reporting state is the opening state, adding the task corresponding to the opening state into a data reporting task list; traversing the data reporting task list to obtain the data reporting task, wherein the data reporting task is one of the data reporting task list.
For example, in some embodiments of the present application, a configuration corresponding to a task whose button is in an on state (i.e., a task to be reported) is added to a reporting list (as a specific example of a data reporting task list). And traversing the tasks in the reporting list to acquire the data reporting tasks.
In some embodiments of the present application, S210 may include: acquiring a task configuration label corresponding to the data reporting task; and searching the task configuration file corresponding to the task configuration label.
For example, in some embodiments of the present application, the network security device 200 may find the corresponding task configuration file through the task configuration tag of the data reporting task. The task configuration tag may be a file number or a file name. The task configuration tag corresponding to each task (namely, the data reporting task) in the report list can be obtained by traversing the report list, and further the task configuration file corresponding to each task is obtained.
In some embodiments of the present application, S220 may include: reading the mapping relation of the data query interface in the task configuration file, and creating a sub-thread; and sending the reported data content to the sub-thread so as to facilitate the sub-thread to open the corresponding task thread.
For example, in some embodiments of the present application, the network security device 200 may create a sub-thread corresponding to the data reporting task through an interface mapping relationship corresponding to the data query interface defined in the task configuration file, and transmit configuration data (as a specific example of reported data content) to the sub-thread. The child thread may directionally open an event thread corresponding to the configuration data. For example, the configuration data is a reported stiff wood worm event, a sub-thread is created through an interface mapping relation and is sent to the sub-thread, and then the sub-thread starts an event thread corresponding to the reported stiff wood worm event. Some embodiments of the application may avoid turning on unnecessary child threads increasing resource consumption.
In some embodiments of the present application, S230 may include: acquiring original data of the platform to be reported through the task thread; and processing the original data to obtain the target data. Wherein, the reporting data content comprises: at least one of a data source, a data type, a data field name, a data field type, and a data field value.
For example, in some embodiments of the present application, the task configuration file configures a data source, a data type, a field name, a processing manner of a field value, and the like to report data content. By acquiring the original data corresponding to the reported data content, the standardized target data can be obtained. The task configuration file is also configured with the type of the field, the processing mode required by the field, the data inquired by the field is the data content reported by default when the field is empty, and the configuration has the function of customizing the field and the field type according to the requirement of a customizing party, such as integer, character string, boolean type and the like. And defining the method for converting the field to obtain the final data reporting result, wherein the conversion aims at that the inquired original data does not necessarily meet the customized requirement, so that the corresponding method is needed to process the data, and different methods are provided for processing the field to obtain the target data.
In some embodiments of the present application, S230 may include: writing the original data into a file according to a data format in the task configuration file to obtain the target data; or, the processing data is arranged into a preset data stream form to obtain the target data.
For example, in some embodiments of the present application, a data format, such as a write format or a data stream format, is configured in the task configuration file. For example, the target data is written into a file, and may also be processed into a data stream (as a specific example of a preset data stream), and the like. Data formats such as a data writing mode, a file format, a file naming mode, a compression mode and the like can be configured in the task configuration file.
In some embodiments of the present application, S230 may include: reading the reporting platform address, wherein the reporting platform address includes: at least one of the area of the platform, the destination address of the platform, the port information and the reported directory; and sending the target data to the target platform according to the reported platform address.
For example, in some embodiments of the present application, an administrator may customize, on the interface of the network security device 200, a target platform to be reported. Because the operators of the provinces are different and the platforms to be reported are different, the provinces can turn on the switches of the platforms to be reported through the interface of the network security device 200. When data is reported, the address of the reporting platform is read to obtain a reporting destination address, a port and a directory (namely a reporting directory) to which the data needs to be reported, so as to complete the automatic reporting of the target data. The task configuration file can also configure different reporting modes according to different provincial operators. The reporting mode may be: FTP (File Transfer Protocol), SFTP (SSH File Transfer Protocol), UDP (User Datagram Protocol), and the like.
In some embodiments of the present application, the method further comprises: and updating the task configuration file in response to an input operation instruction of a user.
For example, in some embodiments of the present application, a user may make editing updates to a task profile. For example, when the reported information is changed, the change of the information can be directly realized by modifying the task configuration file. When the number of the reported fields needs to be increased, the fields and the field attributes can be directly increased in the task configuration file. And if the field needs to be added with a new data processing mode, adding a corresponding processing method in the public part. That is to say, the task configuration file is used as the core function of the whole data reporting service, and the flexibility of the reporting system is improved. Moreover, the unified configuration template is provided, similar functions can be reused, different functions can be customized, and the expansion and updating of the functions are simpler and more efficient. In addition, descriptive information which is easy to understand can be added to the configuration, and the readability of the system is improved.
The following describes, by way of example, a specific implementation process of data reporting provided by some embodiments of the present application with reference to fig. 3.
Referring to fig. 3, fig. 3 is a flowchart of a data reporting method according to some embodiments of the present application. The following describes the above process by taking the beijing bureau of custody as a target platform 300, taking the centralized management system as an enterprise platform 100 or a platform to be reported, and reporting the stiff wood creep event in real time as an example.
S310, the province code of the service to be reported is confirmed to be started.
For example, as a specific example of the present application, the network security device 200 determines that the province starting the service to be reported is beijing, and queries the beijing ID corresponding to the beijing.
And S320, judging the reporting state corresponding to the configuration task under the province code, if the reporting state is the starting state, executing S330, and if not, returning to S310.
For example, as a specific example of the present application, a configuration label with the beijing ID in the on state is acquired, and a button corresponding to a stiff wood worm event is in the on state.
And S330, adding the task corresponding to the starting state into the data reporting task list, and traversing the data reporting task list to obtain the data reporting task.
For example, as one specific example of the present application, a stiff worm event is added to a reporting list. It should be noted that, in an actual application scenario, multiple data reporting tasks may exist in the reporting list at the same time, and the multiple data reporting tasks may be sequentially executed in a traversal manner. For convenience of explanation, it is assumed in this embodiment that only the current stiff wood worm event is reported in the reporting list, that is, the stiff wood worm event is reported as a data reporting task.
S340, acquiring a task configuration label and a reporting time period corresponding to the data reporting task, and searching the task configuration file corresponding to the task configuration label.
For example, as a specific example of the present application, a task configuration tag for reporting a stiff wood worm event and a reporting time period of 10min are obtained. And searching a task configuration file corresponding to the reported dead wood creep event through the task configuration label. The task configuration file is provided with: the reporting task type is a stiff wood creeping event, a data query interface, data content, a reporting mode and a Beijing bureau platform address.
And S350, judging whether the reporting time period is reached, if so, executing S360, and otherwise, executing S330.
For example, as a specific example of the present application, the initial time for reporting the stiff wood creep event is 8.
And S360, reading the mapping relation of the data query interface in the task configuration file, and creating a sub thread.
For example, as a specific example of the present application, a corresponding interface is found through a mapping relationship of a data query interface and a child thread is created.
And S370, sending the reported data content in the task configuration file to the sub-thread, and starting the corresponding task thread by the sub-thread.
For example, as a specific example of the present application, a child thread starts an event thread corresponding to a reported stiff wood worm event.
And S380, inquiring original data of the platform to be reported corresponding to the reported data content, processing the original data and acquiring target data.
For example, as a specific example of the present application, according to the content of the reported data, the original data corresponding to the content of the reported data in the event log corresponding to the stiff wood worm event is searched from the database in the centralized management system. The raw data is then processed to obtain stiff worm event data (as a specific example of target data). For example, the time format is converted from a character string to a number, such as modifying a field name. And writing the data into a txt file according to an ' I ' separator, wherein the file name is defined as ' BJ + time + serial number.
And S390, reporting the target data to a target platform corresponding to the reported platform address.
For example, as a specific example of the present application, the reporting mode configured in the task configuration file is FTP, and the stiff wood wriggling event data is reported to the beijing bureau platform through FTP according to the address of the beijing bureau platform.
According to some embodiments of the present application, when data is reported, the address, the protocol, the port, and the directory to be reported of the target platform to be reported can be obtained according to the task configuration file, and it is not necessary to obtain address information of an opposite terminal at each reporting time. Compared with the traditional streaming data processing mode in which the analytic flow needs to be written repeatedly according to the streaming architecture according to different types of platforms, the similar function needs a plurality of interfaces to be realized, all processing logics are distributed into the service, provinces are distinguished in the service, the platform calls different interfaces to process data, the configuration-based streaming reporting mode adopts a mode of starting different threads to process data through page switch configuration, the directional processing of different events is completely governed by a task configuration file, the method is simple and efficient, code redundancy is greatly reduced, code reusability is improved, a data reporting system in the using process has the advantages of higher maintainability, system readability, stronger expansibility, rapid and accurate positioning problems and the like, a set of unified reporting flow can be realized, and reporting automation is realized.
Referring to fig. 4, fig. 4 is a block diagram illustrating an apparatus for reporting data according to some embodiments of the present disclosure. It should be understood that the device for reporting data corresponds to the above method embodiment, and can perform the steps related to the above method embodiment, and the specific functions of the device for reporting data can be referred to the above description, and a detailed description is appropriately omitted here to avoid redundancy.
The data reporting apparatus of fig. 4 includes at least one software functional module that can be stored in a memory in the form of software or firmware or solidified in the data reporting apparatus, and the data reporting apparatus includes: an obtaining module 410, configured to obtain a task configuration file and a reporting time period corresponding to a data reporting task, where the task configuration file includes: reporting a task type, a data query interface, data content and a platform address; a starting module 420 configured to start a task thread corresponding to the reporting task type through the data query interface when it is determined that the reporting time period is reached; the reporting module 430 is configured to obtain an execution result of the task thread, obtain target data of a platform to be reported corresponding to the reported data content, and report the target data to a target platform corresponding to the reported platform address.
In some embodiments of the present application, the obtaining module 410 is configured to, when the reporting state is the on state, add the task corresponding to the on state to a data reporting task list; traversing the data reporting task list to obtain the data reporting task, wherein the data reporting task is one of the data reporting task list.
In some embodiments of the present application, the obtaining module 410 is configured to, within a preset time, cyclically perform the following operations: confirming and opening province codes of services to be reported; and judging the reporting state corresponding to the task under the provincial code, wherein the reporting state comprises the following steps: the open state and the closed state.
In some embodiments of the present application, the obtaining module 410 is configured to obtain a task configuration tag corresponding to the data reporting task; and searching the task configuration file corresponding to the task configuration label.
In some embodiments of the present application, the starting module 420 is configured to read the mapping relationship of the data query interface in the task configuration file, and create a child thread; and sending the reported data content to the sub-thread so as to facilitate the sub-thread to start the corresponding task thread.
In some embodiments of the present application, the reporting module 430 is configured to obtain, through the task thread, original data of the platform to be reported; and processing the original data to obtain the target data.
In some embodiments of the present application, the reporting data content includes: at least one of a data source, a data type, a data field name, a data field type, and a data field value.
In some embodiments of the present application, the reporting module 430 is configured to write the original data into a file according to a data format in the task configuration file to obtain the target data; or, the processing data is arranged into a preset data stream form to obtain the target data.
In some embodiments of the present application, the reporting module 430 is configured to read the reporting platform address, where the reporting platform address includes: at least one of the area of the platform, the destination address of the platform, the port information and the reporting catalog; and sending the target data to the target platform according to the reported platform address.
In some embodiments of the present application, the data reporting apparatus further includes an updating module (not shown in the figure), and the updating module is configured to update the task configuration file in response to an input operation instruction of a user.
Some embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, can implement the operations of the method corresponding to any of the above-mentioned methods provided by the above-mentioned embodiments.
Some embodiments of the present application further provide a computer program product, which includes a computer program, wherein the computer program, when executed by a processor, can implement the operations of the method corresponding to any of the above-mentioned methods provided by the above-mentioned embodiments.
As shown in fig. 5, some embodiments of the present application provide an electronic device 500, the electronic device 500 including: a memory 510, a processor 520, and a computer program stored on the memory 510 and executable on the processor 520, wherein the method of any of the embodiments described above can be implemented when the processor 520 reads the program from the memory 510 via the bus 530 and executes the program.
Processor 520 may process digital signals and may include various computing structures. Such as a complex instruction set computer architecture, a structurally reduced instruction set computer architecture, or an architecture that implements a combination of instruction sets. In some examples, processor 520 may be a microprocessor.
Memory 510 may be used to store instructions that are executed by processor 520 or data related to the execution of the instructions. The instructions and/or data may include code for performing some or all of the functions of one or more of the modules described in embodiments of the application. The processor 520 of the disclosed embodiments may be used to execute instructions in the memory 510 to implement the methods illustrated above. Memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory known to those skilled in the art.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.

Claims (14)

1. A method for reporting data, comprising:
acquiring a task configuration file and a reporting time period corresponding to a data reporting task, wherein the task configuration file comprises: reporting the task type, the data query interface, the data content and the platform address;
when the reporting time period is confirmed to be reached, starting a task thread corresponding to the reporting task type through the data query interface;
and acquiring an execution result of the task thread, acquiring target data of a platform to be reported corresponding to the reported data content, and reporting the target data to a target platform corresponding to the reported platform address.
2. The method of claim 1, wherein prior to the obtaining the task profile and the reporting time period corresponding to the data reporting task, the method further comprises:
under the condition that the reporting state is the opening state, adding the task corresponding to the opening state into a data reporting task list;
traversing the data reporting task list to obtain the data reporting task, wherein the data reporting task is one of the data reporting task list.
3. The method of claim 2, wherein before the adding the task corresponding to the on state to a data reporting task list, the method further comprises:
within a preset time, the following operations are executed in a circulating mode:
confirming and opening province codes of services to be reported;
and judging the reporting state corresponding to the task under the provincial code, wherein the reporting state comprises the following steps: the open state and the closed state.
4. The method according to any one of claims 1 to 3, wherein the obtaining of the task configuration file corresponding to the data reporting task includes:
acquiring a task configuration label corresponding to the data reporting task;
and searching the task configuration file corresponding to the task configuration label.
5. The method according to any one of claims 1-3, wherein the starting, by the data query interface, a task thread corresponding to the reporting task type includes:
reading the mapping relation of the data query interface in the task configuration file, and creating a sub-thread;
and sending the reported data content to the sub-thread so as to facilitate the sub-thread to open the corresponding task thread.
6. The method of claim 5, wherein the obtaining the execution result of the task thread to obtain the target data of the platform to be reported corresponding to the reported data content comprises:
acquiring original data of the platform to be reported through the task thread;
and processing the original data to obtain the target data.
7. The method of claim 6, wherein reporting the data content comprises: at least one of a data source, a data type, a data field name, a data field type, and a data field value.
8. The method of claim 7, wherein said processing said raw data to obtain said target data comprises:
writing the original data into a file according to a data format in the task configuration file to obtain the target data; or,
and arranging the processing data into a preset data stream form to obtain the target data.
9. The method according to any one of claims 1 to 3, wherein the reporting the target data to the target platform corresponding to the reporting platform address includes:
reading the reporting platform address, wherein the reporting platform address includes: at least one of the area of the platform, the destination address of the platform, the port information and the reported directory;
and sending the target data to the target platform according to the reported platform address.
10. The method of any one of claims 1-3, further comprising:
and updating the task configuration file in response to an input operation instruction of a user.
11. An apparatus for reporting data, comprising:
an obtaining module, configured to obtain a task configuration file and a reporting time period corresponding to a data reporting task, where the task configuration file includes: reporting the task type, the data query interface, the data content and the platform address;
the starting module is configured to start a task thread corresponding to the reporting task type through the data query interface when the reporting time period is determined to be reached;
and the reporting module is configured to acquire an execution result of the task thread, obtain target data of a platform to be reported corresponding to the reported data content, and report the target data to a target platform corresponding to the reported platform address.
12. A computer-readable storage medium, having stored thereon a computer program, wherein the computer program, when executed by a processor, performs the method of any one of claims 1-10.
13. A computer program product, characterized in that the computer program product comprises a computer program, wherein the computer program, when being executed by a processor, performs the method according to any one of claims 1-10.
14. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the computer program, when executed by the processor, performs the method of any one of claims 1-10.
CN202211358055.0A 2022-11-01 2022-11-01 Data reporting method and device, storage medium and electronic equipment Pending CN115510112A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211358055.0A CN115510112A (en) 2022-11-01 2022-11-01 Data reporting method and device, storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211358055.0A CN115510112A (en) 2022-11-01 2022-11-01 Data reporting method and device, storage medium and electronic equipment

Publications (1)

Publication Number Publication Date
CN115510112A true CN115510112A (en) 2022-12-23

Family

ID=84512630

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211358055.0A Pending CN115510112A (en) 2022-11-01 2022-11-01 Data reporting method and device, storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN115510112A (en)

Similar Documents

Publication Publication Date Title
EP3709227B1 (en) System and method for interoperable communication of an automation system component with multiple information sources
US8146100B2 (en) System and method for event-based information flow in software development processes
CN109299183A (en) A kind of data processing method, device, terminal device and storage medium
CN111061678B (en) Service data processing method, device, computer equipment and storage medium
CN107329765B (en) Program file processing method and device
CN113377626B (en) Visual unified alarm method, device, equipment and medium based on service tree
KR20200046769A (en) Method for providing business process management system based on automatic report generation
CN108345691B (en) Data source general processing framework construction method, data source processing method and device
CN105843871B (en) Control and management system of distributed application files
CN111651196A (en) Document publishing method, device and server
CN113792008A (en) Method and device for acquiring network topology structure, electronic equipment and storage medium
CN108345658A (en) Algorithm calculates decomposing process, server and the storage medium of track
CN117271460B (en) Scientific research digital networking service method and system based on scientific research digital object language relation
CN101170436A (en) A method for managing template in network management system
CN116743790B (en) Device data acquisition, device data analysis method and device and computer device
CN115510112A (en) Data reporting method and device, storage medium and electronic equipment
CN116483707A (en) Test method, test device, test apparatus, test program, and test program
CN115062084A (en) Method and device for constructing API (application program interface) based on database metadata
JP7381290B2 (en) Computer system and data management method
CN111190607B (en) Task plugin processing method and device, task scheduling server and storage medium
KR20210008545A (en) Method for providing business process management system based on automatic report generation
CN111061471B (en) Method and system for controlling display of interface control element
CN112131239A (en) Data processing method, computer equipment and readable storage medium
CN113127549B (en) Incremental data synchronization method, device, computer equipment and storage medium
CN115952188A (en) Component management method, system, device and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination