CN116796015A - Visual multi-mode data acquisition, transmission and processing method - Google Patents
Visual multi-mode data acquisition, transmission and processing method Download PDFInfo
- Publication number
- CN116796015A CN116796015A CN202310748531.8A CN202310748531A CN116796015A CN 116796015 A CN116796015 A CN 116796015A CN 202310748531 A CN202310748531 A CN 202310748531A CN 116796015 A CN116796015 A CN 116796015A
- Authority
- CN
- China
- Prior art keywords
- data
- component
- acquisition
- reading
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000000007 visual effect Effects 0.000 title claims abstract description 18
- 230000005540 biological transmission Effects 0.000 title claims description 37
- 238000003672 processing method Methods 0.000 title description 3
- 238000012545 processing Methods 0.000 claims abstract description 93
- 238000000034 method Methods 0.000 claims abstract description 41
- 238000013461 design Methods 0.000 claims description 17
- 238000012795 verification Methods 0.000 claims description 16
- 238000004891 communication Methods 0.000 claims description 15
- 238000006243 chemical reaction Methods 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 9
- 238000004140 cleaning Methods 0.000 claims description 8
- 238000004458 analytical method Methods 0.000 claims description 6
- 238000009826 distribution Methods 0.000 claims description 6
- 239000012634 fragment Substances 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 5
- 238000013075 data extraction Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 4
- 238000013467 fragmentation Methods 0.000 claims description 4
- 238000006062 fragmentation reaction Methods 0.000 claims description 4
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 239000000523 sample Substances 0.000 claims 2
- 239000000758 substrate Substances 0.000 claims 2
- 238000013523 data management Methods 0.000 abstract description 4
- 230000000875 corresponding effect Effects 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000002596 correlated effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000013480 data collection Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000005111 flow chemistry technique Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
- -1 where conditions Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/44—Browsing; Visualisation therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/904—Browsing; Visualisation therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F3/0486—Drag-and-drop
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/163—In-band adaptation of TCP data exchange; In-band control procedures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method for acquiring, transmitting and processing visual multi-mode data, which belongs to the technical field of data management. The visualized assembly arrangement is realized, the data processing task is quickly constructed, and the efficient data processing is realized.
Description
Technical Field
The invention relates to the technical field of data management, in particular to a method for acquiring, transmitting and processing visual multi-mode data.
Background
At present, the data management complexity is further complicated due to the multi-modal of scattered data sources and data types, and the difficulty of data management is increased. Aiming at the characteristics of multi-modal data such as structuring, unstructured, internet of things sensing, audio and video or space data, large data volume, high-speed circulation, uneven quality, different value density and the like, how to realize the visual access, acquisition, transmission and processing of the multi-modal data, acquisition components, data processing components and component arrangement supporting the visualization, which meet the requirements of different modal data, are required to be provided, and meanwhile, the quick and efficient data processing is realized based on the flow batch integrated data transmission of the Flink.
Disclosure of Invention
In order to solve the technical problems, the invention provides a method for acquiring, transmitting and processing visual multi-mode data. Through constructing the multi-mode data acquisition component and combining abundant and various data processing components, the component nodes are supported to be dragged in the canvas, the visualized component arrangement is realized, the data processing task is quickly constructed, meanwhile, the flow batch integrated data transmission based on the Flink is realized, the distributed processing engine is adopted for data acquisition, and the efficient data processing is realized.
The technical scheme of the invention is as follows:
a visual multi-mode data acquisition, transmission and processing method is used for realizing component model description and expansion, constructing a multi-mode acquisition component, integrally transmitting data transmission based on a Flink stream, encrypting and decrypting data in a transmission process to ensure data safety, providing a processing component for data cleaning, data extraction and data identification, arranging the component to realize visual design of a data processing task model, and constructing a model language converter to support completion of data processing.
Further, the method comprises the steps of,
implementing component model descriptions and extensions
Firstly, defining a public component model, which comprises a public basic attribute and a public method, wherein the basic attribute comprises data source information, an output field and a converted sql column, and the public method comprises the steps of obtaining a corresponding column information list of a current node data source, a verification component, verifying whether parameter values are legal or not and obtaining an output column of a previous node;
three types of components of a Reader, a Rule and a Writer are expanded by a public component model, and the expanded components are added with personality attributes to realize an abstract method of the public component; and then the Reader, rule and Writer component models are respectively expanded to realize a multi-mode acquisition component, a processing component and a distribution component, and the attribute of the component model is displayed as a component parameter in the use process of the visual drag increasing component and needs to be filled by a user.
Constructing a multi-modality acquisition assembly comprising
a) Structured data
The data table reading component supports collection of structured data, and component parameters comprise a data source, a database, a mode, a data table, a fragmentation number, a sphere condition, a fragmentation field and an update field;
b) Unstructured data
Providing acquisition components such as hbase reading, hdfs reading, redis reading, mongoDB reading and the like for unstructured data, wherein parameters of the hdfs reading components comprise a data source, a path, separators, reading fields, regular matching, file types and fragment numbers;
c) Internet of things sensing
The method comprises the steps that an Internet of things data reading component is required to provide a communication protocol type, allow clients to carry out IP, check bills and data formats, a monitoring network program is started in the data acquisition process, the client is connected, the client is used for receiving a connection request of an acquired client, waiting for the connection of the clients, then checking the legality of the clients, and the client can establish connection with the acquisition component to send data after the verification is passed;
if the communication protocol type is TCP protocol, the acquisition component generates TCP acquisition drive to start acquisition, if the communication protocol type is RTU protocol, RTU acquisition drive is generated to start acquisition, if the communication protocol type is RAW protocol, RAW acquisition drive is generated to start acquisition, and the acquisition drive performs data conversion according to the data format and then writes the data into a database, an elastic search or sends the data into the Topic of MQTT and Kafka.
d) Audio and video
The audio and video reading component supports acquisition of audio in an AAC format and video in an H264 format; the component selects a corresponding audio/video decoder according to the file name suffix, sends a frame of undecoded compressed data to the decoder, decodes the video image frame by the decoder to obtain a YUV video image, analyzes the image based on a content recognition algorithm, and screens according to a preset event strategy;
e) Spatial data
The vector data reading component supports the collection of SHP, DWG, DXF type space data and the raster data reading component supports the collection of ERDAS image, geoTIFF type space data.
Data transmission
The data transmission is based on Apache link while supporting batch and stream processing.
Data processing assembly
The method comprises the steps of realizing a processing component based on Rule component model expansion, wherein the attribute of the processing component is a parameter to be configured in a page, and the data processing component comprises a base component, a data extraction component, a data cleaning component, a data identification component and a data quality component;
1) The basic component supports selection fields, data merging, aggregation operation, marking and sorting topN;
2) The extraction component comprises Chinese character extraction, mobile phone number extraction, xml analysis, json analysis and data field splitting;
3) The cleaning component supports replacing character strings, converting 15 bits of an identity card into 18 bits, adding a fixed value, emptying the character strings, replacing a dictionary, encrypting fields, expanding the fields, calculating values, filtering data and removing duplication;
4) The identification component comprises a third party identification interface, a personnel time tag and a vehicle type tag;
5) The quality component comprises an identity card verification component, a dictionary value verification component and a mobile phone number verification and time relevance verification component.
Visual design data processing task model
The method supports the dragging type adding components in the canvas, connecting lines are added among the components, the multi-mode acquisition component is used as a source, only the output is carried out, the middle part is a processing component, both the input and the output are provided, the distribution component is used as an ending only input, and the visualized component arrangement is realized to quickly construct a data processing task model.
Construction model language converter
The conversion of the data processing language of Flink, flinkSQL, spark, kettle is implemented.
The method provides a scene of data acquisition, transmission and processing when fire-fighting fire alarms occur, and the operation is as follows:
the data table is read to gather equipment information, the thing allies oneself with data reading subassembly and gathers transmission equipment terminal data, and the audio and video reads the subassembly and gathers on-the-spot camera video data, and the vector data reads the subassembly and gathers the architectural design. The method comprises the following specific steps:
1) And (3) adding an acquisition component:
1.1 The reading data table component collects the equipment information table of the Mysql type database, drags and adds the reading data table component in the canvas and configures the parameter data source, the database, the mode and the data table;
1.2 The data reading component of the internet of things acquires terminal data of the TCP communication protocol transmission equipment, the acquisition component generates a TCP acquisition drive to start acquisition and data format conversion, acquires an alarm condition occurrence position, and then sends the alarm condition occurrence position to the Topic of Kafka;
1.3 The audio and video reading component collects video data of the field camera, decodes video images, analyzes the images based on a content recognition algorithm, and screens out images related to fire occurrence reasons;
1.4 The vector data reading component acquires a building design diagram, extracts marked fire-fighting equipment entity objects, and associates alarm condition position data to acquire nearby fire-fighting equipment;
3) Orchestration design data processing task model
2.1 Removing blank space in the equipment serial number field, dragging and adding a character string emptying component, and selecting equipment serial numbers by selecting filling field and backfilling field parameters;
2.2 If there are several pieces of data with repeated equipment serial numbers, reserving one piece of data with latest warehousing time; dragging to increase a de-duplication component, wherein a de-duplication field selects a device serial number, a condition field selects a warehouse-in time, and a de-duplication condition selects de-duplication according to a condition maximum value;
2.3 The internet of things sensing data is associated with the processed equipment information data and the image decoded by the video data through the data converging component, so that associated data of fire occurrence time, occurrence position and occurrence reason are obtained;
2.4 After confirming the fire, sending a notification by adding a sending component;
2.5 The data converging component continues to correlate the related data obtained in the step 2.3) with the fire-fighting equipment near the police condition extracted by the vector data reading component;
2.6 A data table writing component is added in a dragging mode, and finally the processed data is written into the database.
3) Data transmission and data processing
After the design of the data processing task is finished, the data processing task model is executed, the model language converter converts the generated data processing task model into json files required by the Flink data processing language, the json files are submitted to the Flink engine for data transmission and processing, the time consumption and the number of data processing tasks can be checked in a log, and the processed data are finally written into a database.
The invention has the beneficial effects that
According to the invention, the multi-mode data acquisition component and the data processing component are constructed, the visual arrangement component is supported, the rapid construction of the data processing task model is realized, the flow batch integrated data transmission and the data encryption based on the Flink are realized, and the distributed processing engine is adopted for data acquisition, so that the aims of efficiently acquiring multi-mode data and processing data such as structured, unstructured, internet of things data, audio and video, space data and the like are fulfilled.
Drawings
FIG. 1 is a schematic diagram of a data processing flow of the present invention;
FIG. 2 is a diagram of an example of multi-modal data collection, transmission, processing.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments, and all other embodiments obtained by those skilled in the art without making any inventive effort based on the embodiments of the present invention are within the scope of protection of the present invention.
The invention provides a method for visual multi-mode data acquisition, transmission and processing, which has the flow shown in figure 1 and is specifically implemented as follows:
1. implementing component model descriptions and extensions
Firstly, a public component model is defined, which comprises public basic attributes and a plurality of public methods, wherein the basic attributes comprise data source information, output fields, converted sql columns and the like, and the public methods comprise the steps of obtaining a corresponding column information list of a current node data source, checking a component, checking whether parameter values are legal or not, obtaining an output column of a previous node and the like.
And expanding three types of components of a Reader, a Rule and a Writer by a public component model, adding personality attribute to the expanded components, and realizing an abstract method of the public components. And then the Reader, rule and Writer component models are respectively expanded to realize a multi-mode acquisition component, a processing component and a distribution component, and the attribute of the component model is displayed as a component parameter in the use process of the visual drag increasing component and needs to be filled by a user.
2. Constructing a multi-modality acquisition assembly
a) Structured data
The read data table component supports collection of structured data, and component parameters include data sources, databases, modes, data tables, number of fragments, where conditions, fragment fields, update fields, and the like.
b) Unstructured data
Acquisition components such as hbase read, hdfs read, redis read, mongoDB read and the like are provided for unstructured data, wherein parameters of the hdfs read components comprise data sources, paths, separators, read fields, regular matches, file types and fragment numbers.
c) Internet of things sensing
The internet of things data reading component needs to provide communication protocol types, allow clients to carry out IP, check bills, data formats and the like, a monitoring network program is started firstly in the data acquisition process, the clients are connected and used for receiving connection requests of the acquired clients, waiting for the connection of the clients, then checking the legality of the clients, including bill checking, prohibiting connection of unknown and illegal clients, and the clients can establish connection with the acquisition component to send data after checking.
If the communication protocol type is TCP protocol, the acquisition component generates TCP acquisition drive to start acquisition, if the communication protocol type is RTU protocol, RTU acquisition drive is generated to start acquisition, if the communication protocol type is RAW protocol, RAW acquisition drive is generated to start acquisition, and the acquisition drive performs data conversion according to the data format and then writes the data into a database, an elastic search or sends the data into the Topic of MQTT and Kafka.
d) Audio and video
The audio-video reading component supports acquisition of audio in AAC format, video in H264 format, and so on. For example, the original video in H264 format consists of several groups of pictures, each of which in turn consists of one key frame (commonly referred to as an I-frame) and multiple bi-directional reference frames (commonly referred to as B-frames) and multiple forward reference frames (commonly referred to as P-frames). The component selects a corresponding audio/video decoder according to the file name suffix, sends a frame of undecoded compressed data to the decoder, decodes the video image frame by the decoder to obtain a YUV video image, analyzes the image based on a content recognition algorithm, and screens according to a preset event strategy.
e) Spatial data
The vector data reading component supports the collection of SHP, DWG, DXF and other types of spatial data, and the raster data reading component supports the collection of ERDAS image, geoTIFF and other types of spatial data. For example, the acquisition component converts the DWG file into a DXF file format adapted to the ARM CPU architecture, obtains a file to be analyzed in the DXF file format, obtains entity types contained in each DXF layer of the file to be analyzed, and extracts entity objects corresponding to each entity type. The entity objects comprise some or all of point entities, labeling entities, line entities, plane entities, arc entities, circle entities, elliptical arc entities and elliptical entities, and each entity object is converted into a target application Json format to obtain the entity object in the target application format.
3. Data transmission
Data transmission is based on Apache link, which supports both batch and stream processing. The Flink is a pure stream computing engine whose basic data model is data stream. The Flink program consists of Source, transformation and Sink, wherein Source is mainly responsible for reading data and supports HDFS, kafka, text and the like; the Transformation is mainly responsible for converting data; sink is responsible for the output of the final data, supporting HDFS, kafka, text output, etc. The data that flows between the sections is called a stream (stream). The flow may be borderless, unrestricted-flow, i.e. flow processing in general. There may also be bounded limited flow, i.e., batch processing. Thus, the Flink supports both stream processing and batch processing with a suite of architectures.
Batch processing supports data source types: HDFS, text, CSV, JDBC, HBase, and Collections, etc. Stream processing supports data source types: files, socket streams, kafka, rabbitMQ, flume, etc.
On the basis of stream batch integrated data transmission, the data encryption component ensures the data security in the data transmission process, and encryption algorithms supported by the encryption component are MD5, BASE64, DES, AES-128, AES-192, AES-256, RSA, SM2, SM3 and SM4. The algorithms supported by the data decryption component are BASE64, DES, AES-128, AES-192, AES-256, RSA, SM2, SM4.
4. Data processing assembly
The processing component is realized based on Rule component model extension, and the attribute of the processing component is the parameter to be configured in the page, such as a selection field, a backfill field, a deduplication condition, the number of fragments and the like. The data processing component comprises a basic component, a data extraction component, a data cleaning component, a data identification component, a data quality component and the like,
1) The base component supports selection fields, data merging, aggregation operations, marking, ordering topN, and the like.
2) The extraction component comprises Chinese character extraction, mobile phone number extraction, xml analysis, json analysis, data field splitting and the like.
3) The cleaning component supports more than 30 components such as character string replacement, 15-bit conversion of an identity card to 18-bit conversion, fixed value addition, character string exhaustion, dictionary replacement, field encryption, field expansion, calculated value, data filtering, duplication removal and the like.
4) The identification component includes a third party identification interface, a person year tag, a vehicle type tag, and the like.
5) The quality component comprises an identity card verification, dictionary value verification, mobile phone number verification and other format validity components and a time relevance verification and other logic rationality components.
5. Visual design data processing task model
The method supports the dragging type adding components in the canvas, connecting lines are added among the components, the multi-mode acquisition component is used as a source, only the output is carried out, the middle part is a processing component, both the input and the output are provided, the distribution component is used as an ending only input, and the visualized component arrangement is realized to quickly construct a data processing task model.
6. Construction model language converter
The conversion of the data processing language such as Flink, flinkSQL, spark, kettle is realized, for example, the model language converter converts the data processing task model generated in the last step into json files required by the link data processing language.
The scene of data acquisition, transmission and processing when the fire alarm occurs is provided, and an example is shown in the attached figure 2, and the specific operation is as follows:
the data table is read to gather equipment information, thing allies oneself with data reading subassembly and gathers transmission equipment terminal data, and the audio and video reads the subassembly and gathers on-the-spot camera video data, and vector data reads the subassembly and gathers the architectural design drawing, i. add collection subassembly:
1) The data reading table component collects the equipment information table of the Mysql type database, drags and adds the data reading table component in the canvas, configures parameter data sources, databases, modes and data tables, and comprises equipment serial numbers, equipment types, units, buildings, coordinates X, coordinates Y, warehouse-in time and the like.
2) The data reading component of the internet of things acquires terminal data of the TCP communication protocol transmission equipment, the acquisition component generates a TCP acquisition drive to start acquisition and data format conversion, acquires data such as alarm condition occurrence positions and the like, and then sends the data to the Topic of Kafka.
3) The audio and video reading component collects video data of the field camera, decodes video images, analyzes the images based on a content recognition algorithm, and screens out images related to fire occurrence reasons.
4) The vector data reading component collects the building design diagram, extracts the marked fire-fighting equipment entity object, and associates the alarm situation position data to obtain the nearby fire-fighting equipment.
Orchestration design data processing task model
1) And removing the blank in the equipment serial number field, dragging the character string emptying component, and selecting the equipment serial number by selecting the filling field and backfilling field parameters.
2) If a plurality of pieces of data with repeated equipment serial numbers exist, one piece of data with latest warehousing time is reserved.
And dragging to increase a deduplication component, wherein a deduplication field selects a device serial number, a condition field selects a warehouse-in time, and a deduplication condition selects deduplication according to a condition maximum value.
3) And correlating the internet of things sensing data with the processed equipment information data and the processed image decoded by the video data through a data converging component to obtain correlated data such as fire occurrence time, occurrence position, occurrence reason and the like.
4) And after the fire disaster is confirmed, a short message sending component is added to send a short message notification.
5) And the data converging component continuously correlates the correlated data obtained in the step 3) with the fire-fighting equipment near the police condition extracted by the vector data reading component, thereby being beneficial to rapidly positioning the fire-fighting equipment for rescue.
6) And dragging the added data writing table component to realize the final writing of the processed data into the database.
ii, i data transmission and data processing
After the design of the data processing task is finished, the data processing task model is executed, the model language converter converts the generated data processing task model into json files required by the Flink data processing language, the json files are submitted to the Flink engine for data transmission and processing, the time consumption and the number of data processing tasks can be checked in a log, and the processed data are finally written into a database.
The multi-mode data acquisition component is constructed, structured, unstructured, internet of things data, audio and video and space data acquisition is supported, and multi-mode data of different scenes is quickly accessed.
And visually designing a data processing task model, dragly adding component nodes or task nodes, supporting arranging components or tasks, and realizing rapid construction of the data processing task model.
The model language converter realizes conversion of data processing languages such as Flink, flinkSQL, spark, kettle.
The foregoing description is only illustrative of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (9)
1. A method for visual multi-mode data acquisition, transmission and processing is characterized in that,
comprising the following steps:
the method comprises the steps of realizing component model description and expansion, constructing a multi-mode acquisition component, integrally transmitting a stream batch based on the Flink in data transmission, encrypting and decrypting data in the transmission process to ensure data safety, providing a processing component for data cleaning, data extraction and data identification, arranging the component to realize a visual design data processing task model, and constructing a model language converter to support completion of data processing.
2. The method of claim 1, wherein the step of determining the position of the substrate comprises,
implementing component model descriptions and extensions
Firstly, defining a public component model, which comprises a public basic attribute and a public method, wherein the basic attribute comprises data source information, an output field and a converted sql column, and the public method comprises the steps of obtaining a corresponding column information list of a current node data source, a verification component, verifying whether parameter values are legal or not and obtaining an output column of a previous node;
three types of components of a Reader, a Rule and a Writer are expanded by a public component model, and the expanded components are added with personality attributes to realize an abstract method of the public component; and then the Reader, rule and Writer component models are respectively expanded to realize a multi-mode acquisition component, a processing component and a distribution component, and the attribute of the component model is displayed as a component parameter in the use process of the visual drag increasing component and needs to be filled by a user.
3. The method of claim 2, wherein the step of determining the position of the substrate comprises,
constructing a multi-modality acquisition assembly comprising
a) Structured data
The data table reading component supports collection of structured data, and component parameters comprise a data source, a database, a mode, a data table, a fragmentation number, a sphere condition, a fragmentation field and an update field;
b) Unstructured data
Providing acquisition components such as hbase reading, hdfs reading, redis reading, mongoDB reading and the like for unstructured data, wherein parameters of the hdfs reading components comprise a data source, a path, separators, reading fields, regular matching, file types and fragment numbers;
c) Internet of things sensing
The method comprises the steps that an Internet of things data reading component is required to provide a communication protocol type, allow clients to carry out IP, check bills and data formats, a monitoring network program is started in the data acquisition process, the client is connected, the client is used for receiving a connection request of an acquired client, waiting for the connection of the clients, then checking the legality of the clients, and the client can establish connection with the acquisition component to send data after the verification is passed;
if the communication protocol type is TCP protocol, the acquisition component generates TCP acquisition drive to start acquisition, if the communication protocol type is RTU protocol, RTU acquisition drive is generated to start acquisition, if the communication protocol type is RAW protocol, RAW acquisition drive is generated to start acquisition, and the acquisition drive performs data conversion according to the data format and then writes the data into a database, an elastic search or sends the data into the Topic of MQTT and Kafka.
d) Audio and video
The audio and video reading component supports acquisition of audio in an AAC format and video in an H264 format; the component selects a corresponding audio/video decoder according to the file name suffix, sends a frame of undecoded compressed data to the decoder, decodes the video image frame by the decoder to obtain a YUV video image, analyzes the image based on a content recognition algorithm, and screens according to a preset event strategy;
e) Spatial data
The vector data reading component supports the collection of SHP, DWG, DXF type space data and the raster data reading component supports the collection of ERDAS image, geoTIFF type space data.
4. The method of claim 3, wherein the step of,
data transmission
The data transmission is based on Apache link while supporting batch and stream processing.
5. The method of claim 4, wherein the step of determining the position of the first electrode is performed,
data processing assembly
The method comprises the steps of realizing a processing component based on Rule component model expansion, wherein the attribute of the processing component is a parameter to be configured in a page, and the data processing component comprises a base component, a data extraction component, a data cleaning component, a data identification component and a data quality component;
1) The basic component supports selection fields, data merging, aggregation operation, marking and sorting topN;
2) The extraction component comprises Chinese character extraction, mobile phone number extraction, xml analysis, json analysis and data field splitting;
3) The cleaning component supports replacing character strings, converting 15 bits of an identity card into 18 bits, adding a fixed value, emptying the character strings, replacing a dictionary, encrypting fields, expanding the fields, calculating values, filtering data and removing duplication;
4) The identification component comprises a third party identification interface, a personnel time tag and a vehicle type tag;
5) The quality component comprises an identity card verification component, a dictionary value verification component and a mobile phone number verification and time relevance verification component.
6. The method of claim 5, wherein the step of determining the position of the probe is performed,
visual design data processing task model
The method supports the dragging type adding components in the canvas, connecting lines are added among the components, the multi-mode acquisition component is used as a source, only the output is carried out, the middle part is a processing component, both the input and the output are provided, the distribution component is used as an ending only input, and the visualized component arrangement is realized to quickly construct a data processing task model.
7. The method of claim 6, wherein the step of providing the first layer comprises,
construction model language converter
The conversion of the data processing language of Flink, flinkSQL, spark, kettle is implemented.
8. The method of claim 7, wherein the step of determining the position of the probe is performed,
the method provides a scene of data acquisition, transmission and processing when fire-fighting fire alarms occur, and the operation is as follows:
the data table is read to gather equipment information, the thing allies oneself with data reading subassembly and gathers transmission equipment terminal data, and the audio and video reads the subassembly and gathers on-the-spot camera video data, and the vector data reads the subassembly and gathers the architectural design.
9. The method of claim 8, wherein the step of determining the position of the first electrode is performed,
the method comprises the following specific steps:
1) And (3) adding an acquisition component:
1.1 The reading data table component collects the equipment information table of the Mysql type database, drags and adds the reading data table component in the canvas and configures the parameter data source, the database, the mode and the data table;
1.2 The data reading component of the internet of things acquires terminal data of the TCP communication protocol transmission equipment, the acquisition component generates a TCP acquisition drive to start acquisition and data format conversion, acquires an alarm condition occurrence position, and then sends the alarm condition occurrence position to the Topic of Kafka;
1.3 The audio and video reading component collects video data of the field camera, decodes video images, analyzes the images based on a content recognition algorithm, and screens out images related to fire occurrence reasons;
1.4 The vector data reading component acquires a building design diagram, extracts marked fire-fighting equipment entity objects, and associates alarm condition position data to acquire nearby fire-fighting equipment;
2) Orchestration design data processing task model
2.1 Removing blank space in the equipment serial number field, dragging and adding a character string emptying component, and selecting equipment serial numbers by selecting filling field and backfilling field parameters;
2.2 If there are several pieces of data with repeated equipment serial numbers, reserving one piece of data with latest warehousing time; dragging to increase a de-duplication component, wherein a de-duplication field selects a device serial number, a condition field selects a warehouse-in time, and a de-duplication condition selects de-duplication according to a condition maximum value;
2.3 The internet of things sensing data is associated with the processed equipment information data and the image decoded by the video data through the data converging component, so that associated data of fire occurrence time, occurrence position and occurrence reason are obtained;
2.4 After confirming the fire, sending a notification by adding a sending component;
2.5 The data converging component continues to correlate the related data obtained in the step 2.3) with the fire-fighting equipment near the police condition extracted by the vector data reading component;
2.6 A data table writing component is added in a dragging mode, and finally the processed data is written into the database.
3) Data transmission and data processing
After the design of the data processing task is finished, the data processing task model is executed, the model language converter converts the generated data processing task model into json files required by the Flink data processing language, the json files are submitted to the Flink engine for data transmission and processing, the time consumption and the number of data processing tasks can be checked in a log, and the processed data are finally written into a database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310748531.8A CN116796015A (en) | 2023-06-25 | 2023-06-25 | Visual multi-mode data acquisition, transmission and processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310748531.8A CN116796015A (en) | 2023-06-25 | 2023-06-25 | Visual multi-mode data acquisition, transmission and processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116796015A true CN116796015A (en) | 2023-09-22 |
Family
ID=88039302
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310748531.8A Pending CN116796015A (en) | 2023-06-25 | 2023-06-25 | Visual multi-mode data acquisition, transmission and processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116796015A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117724706A (en) * | 2024-02-06 | 2024-03-19 | 湖南盛鼎科技发展有限责任公司 | Method and system for batch-flow integrated flow real-time processing of heterogeneous platform mass data |
-
2023
- 2023-06-25 CN CN202310748531.8A patent/CN116796015A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117724706A (en) * | 2024-02-06 | 2024-03-19 | 湖南盛鼎科技发展有限责任公司 | Method and system for batch-flow integrated flow real-time processing of heterogeneous platform mass data |
| CN117724706B (en) * | 2024-02-06 | 2024-05-03 | 湖南盛鼎科技发展有限责任公司 | Method and system for batch-flow integrated flow real-time processing of heterogeneous platform mass data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10885100B2 (en) | Thumbnail-based image sharing method and terminal | |
| CA3040856C (en) | Methods and apparatus for false positive minimization in facial recognition applications | |
| WO2017005149A1 (en) | Game acceleration method and device | |
| Simoens et al. | Scalable crowd-sourcing of video from mobile devices | |
| CN105095211B (en) | The acquisition methods and device of multi-medium data | |
| US20210365343A1 (en) | Artificial Intelligence (AI)-Based Cloud Computing Safety Monitoring System | |
| US11341351B2 (en) | Methods and apparatus for facial recognition on a user device | |
| WO2020134684A1 (en) | Information retrieval method, apparatus, device and medium | |
| CN116796015A (en) | Visual multi-mode data acquisition, transmission and processing method | |
| US10902049B2 (en) | System and method for assigning multimedia content elements to users | |
| WO2022127259A1 (en) | Data cleaning method, apparatus and device, and storage medium | |
| CN111935663B (en) | Sensor data stream processing method, device, medium and electronic equipment | |
| Balchandani et al. | A deep learning framework for smart street cleaning | |
| CN110852306A (en) | Safety monitoring system based on artificial intelligence | |
| WO2023179161A1 (en) | Video frame rate control method and apparatus, and electronic device and storage medium | |
| CN105335408B (en) | A kind of extended method and related system of search term white list | |
| CN111062043A (en) | Medical image identification method and system based on edge calculation | |
| CN117540321A (en) | Data service monitoring system | |
| CN110188273B (en) | Information content notification method, device, server and readable medium | |
| KR20190029800A (en) | Terminal device, server and system for image recognition capable of protecting personal information | |
| CN115221453B (en) | Media resource management method, device, server and medium | |
| CN116561337A (en) | Network attack knowledge graph generation method and device, electronic equipment and storage medium | |
| CN113762040B (en) | Video identification method, device, storage medium and computer equipment | |
| CN113032775B (en) | Information processing method and information processing system | |
| WO2019127653A1 (en) | Method for identifying harmful video on basis of credits content and system therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |