CN116150129A - Sea-entry sewage outlet data reorganization evaluation method - Google Patents
Sea-entry sewage outlet data reorganization evaluation method Download PDFInfo
- Publication number
- CN116150129A CN116150129A CN202310416350.5A CN202310416350A CN116150129A CN 116150129 A CN116150129 A CN 116150129A CN 202310416350 A CN202310416350 A CN 202310416350A CN 116150129 A CN116150129 A CN 116150129A
- Authority
- CN
- China
- Prior art keywords
- data
- drain
- outlet
- file
- sewage outlet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/156—Query results presentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
- G06F16/166—File name conversion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to a sea sewage outlet data reorganization evaluation method, which belongs to the field of data processing and comprises the following steps: s1, designing a drain outlet archive database; s2, uploading a file formed by the data compression package of the sewage outlet to a sewage outlet archive database in real time, and automatically inputting historical archive data to the sewage outlet archive database; s3, multi-dimensional data joint retrieval is carried out, and a new task is created. The invention has the advantages that: the problems of multi-user collaborative operation, mobile terminal data hierarchical slow release, accurate navigation positioning, ultra-large file real-time transmission, efficient merging output of batch data, multi-source historical data sorting and warehousing, multi-dimensional joint retrieval and the like are solved.
Description
Technical Field
The invention relates to a sea sewage outlet data reorganization evaluation method, and belongs to the field of data processing.
Background
With the popularization of technical means such as cloud databases, CDN node acceleration, elastic search engines and the like, the technical methods of generating, integrating and applying the sewage outlet data in the sea are changed, and the data gradually enter the digital data era from the traditional paper data era. At present, the sea entry and drainage data reorganization evaluation starts to have the trend of mobile phone end input, cloud storage and PC end management, but when the sea entry and drainage quantity breaks through to five thousands, new mode methods are needed to be established in the aspects of multi-person collaborative operation, mobile end data hierarchical slow release, accurate navigation positioning, ultra-large file real-time transmission, batch data efficient merging output, multi-source historical data sorting and warehousing, multi-dimensional joint retrieval and the like.
The traditional sea sewage outlet management is mainly based on industrial drainage, the quantity is small, and the data is mainly based on text data. With the deep recognition of the sea-entering sewage outlets and the popularization of electronic products such as mobile phones, unmanned aerial vehicles and the like, the variety and the number of sea-entering sewage outlets which are managed in a incorporating way are increased greatly, electronic photos and video data are increased gradually, and the traditional paper machine account type reorganization evaluation mode is not suitable for the reorganization evaluation requirement under a new form.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention establishes a sea-entering sewage outlet data reorganization evaluation method, and provides a high-efficiency solution for provincial sea-entering sewage outlet supervision and management.
The invention provides a sea-entering sewage outlet data reorganization evaluation method, which comprises the following steps:
the sea-entering sewage outlet data reorganization evaluation method is characterized by comprising the following steps of:
s1, designing a drain outlet archive database;
s2, uploading a file formed by the data compression package of the sewage outlet to a sewage outlet archive database in real time, and automatically inputting historical archive data to the sewage outlet archive database;
s3, multi-dimensional data joint retrieval is carried out, and a new task is created.
The step S1 specifically comprises the following steps:
a drain outlet archive data table is designed in a mode of combining the transverse direction and the longitudinal direction, wherein the transverse data is a drain outlet fixed attribute archive table, and the longitudinal data is a drain outlet monitoring item table; non-normalized data of the sewage outlet data in the sea, multi-source data cannot contain consistent monitoring items, and meanwhile, the monitoring items are not contained in the database, and the monitoring item table is supplemented according to the monitoring items of the data to form a sewage outlet monitoring item table;
the drain outlet file data table comprises fixed file attribute contents of a drain outlet, wherein the fixed file attribute comprises a drain outlet code and a drain outlet name; the sewage outlet monitoring item list comprises all monitoring items of the sewage outlet;
the monitoring item includes: the sewage outlet longitude and latitude, the point location, the administrative division to which the sewage outlet belongs, the detailed address, the discharge condition, the sewage source, the sea entering mode, the surrounding environment, the sewage outlet type, the field photograph and video and description, the sewage outlet state, whether illegal, whether reasonable, whether vertical, whether labeling path, whether marketing number, sewage outlet remediation condition, sewage discharge amount, main pollutant type, responsible body and corresponding contact mode, sewage outlet setup and withdrawal time, field sensory and flow, COD, ammonia nitrogen, total phosphorus, salinity, pH, suspended matters, chloride, hexavalent chromium, organic carbon, BOD, arsenic, mercury, copper, lead, cadmium, chromium, zinc, nickel, oil, volatile phenol, fecal coliform number, volatile halogenated hydrocarbon, administrative auditing state and remarking condition.
In the step S2, when the file formed by the data compression packet of the sewage outlet is uploaded in real time, the specific steps are:
2-1, packaging a plurality of types of files into a compressed file:
；
2-2, segmenting the file with more than 10M in the memory by adopting a slice slicing method, fixedly segmenting fragments with the size of 10M each time, simultaneously calculating the sequence number of the current segmented fragments and the total number of the file slices, generating uuid and carrying out the segmentation and the sequence number of the fragments, and submitting the sequence number to a server through a constructed form; if the submitting fails, re-submitting the fragmented file every 2 seconds, and if the submitting is successful, continuing the next fragmented until all the fragmented files are submitted successfully;
2-3, extracting the fragmented files of each time by using the file extractor MutipartHttpServletRequest in a directional way, and checking and storing the fragmented files as temporary files according to uuid and the sequence numbers of the fragmented files
The method comprises the steps of carrying out a first treatment on the surface of the Creating a file stream, then circulating a fragment sequence number, sequentially writing fragments into a buffer area, setting pointer offset of the file stream, sequentially extracting byte data from the buffer area, writing the byte data into the created file stream, and simultaneously moving the pointer offset of the file stream backwards according to the length of the byte data until all fragments of data are written into the file stream in sequence; deleting the temporary file of the fragment, simultaneously saving the merged file path to a database, and adding +.>
Wherein, F1: 1 st slice file, F2: slice 2 file, F3: slice 3 file, F4: 4 th slice file, fn: the nth slice file, F, is the sum of the slice files.
In the step 2-1, the pictures and the videos are stored in a physical hard disk of a server, then the reported data are converted into a drain archive data entity by combining the paths of the pictures and the videos, according to the data in the entity, the drain monitoring item table is searched circularly to obtain the corresponding monitoring item ID, and finally the data in the entity and the monitoring item ID are correspondingly stored in a warehouse.
The step S3 specifically comprises the following steps:
3-1, inquiring all drain outlet sets by key information, and only reserving the key information of the drain outlet, wherein the key information comprises a drain outlet number and drain outlet longitude and latitude so as to realize rapid superposition of point positions in a map; when clicking or selecting the drain outlet for viewing, searching and displaying the drain outlet file by using the drain outlet code;
3-2, the map adopts an OpenLayers map, a built-in cluster method is used for superposing all the point location sets into the map, the pixel distance of point location aggregation is set to be 40, meanwhile, the pixel distance between the point locations is calculated, and the point locations smaller than 40 are aggregated and displayed.
In the step S2, when the history archive data is automatically recorded into the drain archive database, the specific steps are as follows:
2-4, traversing the search folder by using a GetDirectories method, splitting the folder, and obtaining the investigation time and the pollution discharge code in the corresponding investigation time;
2-5, traversing the files in the search folder by using a GetFiles method, and associating drain port codes and investigation time of the corresponding files;
2-6, moving the picture video file to the corresponding physical path by using a Move method, renaming the file by adding drain port codes and serial numbers in the checking time, and storing and warehousing data corresponding to the drain port codes of the corresponding file path.
In the step 3-1, specifically:
taking tasks as an operation set and a drain outlet as an operation unit, when a plurality of people operate drain outlet archival data simultaneously, firstly taking the drain outlet out of the operation set, locking the drain outlet by synchronized, preempting the drain outlet in time sequence, and searching from the operation set according to drain outlet codes when other people want to operate, wherein the drain outlet is taken out from the operation set, prompting that the drain outlet is being operated, and reassigning threads to store the current operation data in a draft data form; and after the data operation is finished, automatically checking file table data and draft table data of the same sewage outlet, and sequencing according to the role priority of the data operators.
In the step 3-1, the task and sewage outlet searching steps are as follows:
number with built-in drain outlet
Task nameScale->
Task area->
And investigation personnel->
As search conditions, multi-dimensional search is selected as needed, and other search conditions O n The system also comprises a drain port type, a drain condition, a sea entering mode, a drain port state and a data archiving mode;
according to actual business requirements, the investigation and monitoring of the drain outlet takes a task as a basic unit, and the information of the drain outlet to be investigated forms a data set, wherein a drain outlet basic attribute archive table and a monitoring item data table are associated; the sewage outlet archive basic attribute archive table comprises sewage outlet codes, national standard codes and names, the monitoring project data table comprises sewage outlet codes, sewage outlet longitude and latitude, point positions, administrative division, detailed addresses, emission conditions, sewage sources, sea entering modes, surrounding environments, sewage outlet types, site photos and videos and descriptions, sewage outlet states, whether illegal, whether reasonable, whether vertical plates, whether labeling paths, whether sales numbers, sewage outlet treatment conditions, sewage discharge amounts, main pollutant types, responsibility main bodies and corresponding contact modes, sewage outlet setting and withdrawal time, site senses and flow, COD, ammonia nitrogen, total phosphorus, salinity, pH, suspended matters, chlorides, hexavalent chromium, organic carbon, arsenic, mercury, copper, lead, cadmium, chromium, zinc, nickel, oils, volatile phenols, fecal coliform numbers, volatile halogenated hydrocarbons, administrative states and remarking conditions, and the sewage outlet task table comprises task names, belonging task areas, a drain outlet set, sewage outlet set codes, sewage outlet basic attribute table and user attribute data are searched by the sewage outlet basic attribute data table;
the related drain outlet basic attribute archive table, the monitoring item data table and the investigation task table are subjected to dynamic matching retrieval, and drain outlet data meeting the conditions are screened:
when the task is created, a task range is selected by an Excel file, an easy POI tool is used, drain outlet coding data in the task is extracted by using an importer Excel method, meanwhile, the data is written into a buffer zone, a drain outlet file data table is searched according to drain outlet coding comparison in the buffer zone, and a basic file data information generating task is obtained.
The monitoring items focused by different types of drain ports are different, a certain specific drain port has no information in a plurality of monitoring items in a database, the non-information monitoring items are meaningless for drain port evaluation, when drain port data are retrieved, drain port file data table data are retrieved according to drain port code traversal, monitoring items with non-empty monitoring item data in all retrieved data are combined into a basic set, finally the basic attribute field is combined with the monitoring item set to form a drain port attribute column, a data set of information matching and invalid monitoring items is formed, wherein the basic attribute field comprises drain port codes, national standard codes and names, the monitoring item set comprises a drain longitude and latitude, a point location, an administrative division, a detailed address, a discharge condition, a sewage source, a sea entering mode, a surrounding environment, a drain type, a field photograph and video and description, a drain state, whether illegal, reasonable, whether vertical, whether labeling a path, whether marketing, a drain remediation condition, a sewage discharge amount, a main pollutant type, a responsibility main body and a corresponding contact mode, drain setting and withdrawal time, a field sense and flow, COD, ammonia nitrogen, total phosphorus, salinity, pH, suspended matters, chloride, hexavalent chromium, organic carbon, BOD, arsenic, mercury, copper, lead, cadmium, chromium, zinc, nickel, oils, volatile phenols, fecal coliform number, volatile halogenated hydrocarbons, an administrative auditing state and a set of remarking conditions, and filling with empty data when attribute column data are missing; when the drain outlet data is exported, traversing and searching all drain outlet archive data table data according to the drain outlet coding set, merging monitoring items with non-empty monitoring item data in all the searched data into a basic set, finally taking the drain outlet coding and the name as basic attribute fields, merging and displaying drain outlet attribute columns by combining the monitoring item sets, and filling with empty data when the attribute column data is missing.
The invention has the advantages that: the problems of multi-user collaborative operation, mobile terminal data hierarchical slow release, accurate navigation positioning, ultra-large file real-time transmission, efficient merging output of batch data, multi-source historical data sorting and warehousing, multi-dimensional joint retrieval and the like are solved.
Drawings
FIG. 1 is a schematic flow chart of the multi-dimensional data joint evaluation of the invention.
Detailed Description
The invention will be further described with reference to specific embodiments, and advantages and features of the invention will become apparent from the description. These examples are merely exemplary and do not limit the scope of the invention in any way. It will be understood by those skilled in the art that various changes and substitutions of details and forms of the technical solution of the present invention may be made without departing from the spirit and scope of the present invention, but these changes and substitutions fall within the scope of the present invention.
Referring to fig. 1, the invention relates to a sea-entering sewage outlet data reorganization evaluation method, which comprises the following steps:
s1, designing a drain outlet archive database;
s2, uploading a file formed by the data compression package of the sewage outlet to a sewage outlet archive database in real time, and automatically inputting historical archive data to the sewage outlet archive database;
s3, multi-dimensional data joint retrieval is carried out, and a new task is created.
The step S1 specifically comprises the following steps: a drain outlet archive data table is designed in a mode of combining the transverse direction and the longitudinal direction, wherein the transverse data is a drain outlet fixed attribute archive table, and the longitudinal data is a drain outlet monitoring item table; non-normalized data of the sewage outlet data in the sea, multi-source data cannot contain consistent monitoring items, and meanwhile, the monitoring items are not contained in the database, and the monitoring item table is supplemented according to the monitoring items of the data to form a sewage outlet monitoring item table;
the drain outlet file data table comprises fixed file attribute contents of a drain outlet, wherein the fixed file attribute comprises a drain outlet code and a drain outlet name; the sewage outlet monitoring item list comprises all monitoring items of the sewage outlet;
the monitoring item includes: the sewage outlet longitude and latitude, the point location, the administrative division to which the sewage outlet belongs, the detailed address, the discharge condition, the sewage source, the sea entering mode, the surrounding environment, the sewage outlet type, the field photograph and video and description, the sewage outlet state, whether illegal, whether reasonable, whether vertical, whether labeling path, whether marketing number, sewage outlet remediation condition, sewage discharge amount, main pollutant type, responsible body and corresponding contact mode, sewage outlet setup and withdrawal time, field sensory and flow, COD, ammonia nitrogen, total phosphorus, salinity, pH, suspended matters, chloride, hexavalent chromium, organic carbon, BOD, arsenic, mercury, copper, lead, cadmium, chromium, zinc, nickel, oil, volatile phenol, fecal coliform number, volatile halogenated hydrocarbon, administrative auditing state and remarking condition.
In the step S2, when the file formed by the data compression packet of the sewage outlet is uploaded in real time, the specific steps are:
2-1, packaging a plurality of types of files into a compressed file:
；
The drain photos shot in the investigation process are 10M/piece basically, 6-8 drain openings are 6-8 in each, the unmanned aerial vehicle high-definition videos are 200M/piece, 1-2 drain openings are classified and arranged after being subjected to centralized investigation in the same district and market range, then data uploading is carried out, the daily investigation data basically takes GB as a data unit, a large amount of bandwidth can be occupied by direct packing uploading, meanwhile, the browser can be blocked possibly, after a compressed package is sliced, a single thread is started for data uploading, the bandwidth occupancy rate is reduced, and meanwhile, the system fluency is correspondingly improved.
In the step 2-1, the pictures and the videos are stored in a physical hard disk of a server, then the reported data are converted into a drain archive data entity by combining the paths of the pictures and the videos, according to the data in the entity, the drain monitoring item table is searched circularly to obtain corresponding monitoring item IDs, and finally the data in the entity and the monitoring item IDs are stored in a warehouse correspondingly.
2-2, segmenting the file with more than 10M in the memory by adopting a slice slicing method, fixedly segmenting fragments with the size of 10M each time, simultaneously calculating the sequence number of the current segmented fragments and the total number of the file slices, generating uuid and carrying out the segmentation and the sequence number of the fragments, and submitting the sequence number to a server through a constructed form; if the submission fails, the fragmented file is re-submitted every 2 seconds, and if the submission is successful, the next fragment is continued until all fragments are submitted successfully
2-3, extracting the fragmented files of each time by using the file extractor MutipartHttpServletRequest in a directional way, and checking and storing the fragmented files as temporary files according to uuid and the sequence numbers of the fragmented files
The method comprises the steps of carrying out a first treatment on the surface of the Creating a file stream, then cycling the shard sequence numbers, andsequentially writing the fragments into a buffer area, setting pointer offset of the file stream, sequentially extracting byte data from the buffer area, writing the byte data into the created file stream, and simultaneously moving the pointer offset of the file stream backwards according to the length of the byte data until all the fragment data are written into the file stream in sequence; deleting the temporary file of the fragment, simultaneously saving the merged file path to a database, and adding +.>
Wherein, F1: 1 st slice file, F2: slice 2 file, F3: slice 3 file, F4: 4 th slice file, fn: the nth slice file, F, is the sum of the slice files.
In the step S2, when the history archive data is automatically recorded into the drain archive database, the specific steps are as follows:
2-4, traversing the search folder by using a GetDirectories method, splitting the folder, and obtaining the investigation time and the pollution discharge code in the corresponding investigation time;
2-5, traversing the files in the search folder by using a GetFiles method, and associating drain port codes and investigation time of the corresponding files;
2-6, moving the picture video file to the corresponding physical path by using a Move method, renaming the file by adding drain port codes and serial numbers in the checking time, and storing and warehousing data corresponding to the drain port codes of the corresponding file path.
The step S3 specifically comprises the following steps:
3-1, inquiring all drain outlet sets by key information, and only reserving the key information of the drain outlet, wherein the key information comprises a drain outlet number and drain outlet longitude and latitude so as to realize rapid superposition of point positions in a map; when clicking or selecting the drain outlet for viewing, searching and displaying the drain outlet file by using the drain outlet code;
3-2, the map adopts an OpenLayers map, a built-in cluster method is used for superposing all the point location sets into the map, the pixel distance of point location aggregation is set to be 40, meanwhile, the pixel distance between the point locations is calculated, and the point locations smaller than 40 are aggregated and displayed.
In the step 3-1, specifically:
taking tasks as an operation set and a drain outlet as an operation unit, when a plurality of people operate drain outlet archival data simultaneously, firstly taking the drain outlet out of the operation set, locking the drain outlet by synchronized, preempting the drain outlet in time sequence, and searching from the operation set according to drain outlet codes when other people want to operate, wherein the drain outlet is taken out from the operation set, prompting that the drain outlet is being operated, and reassigning threads to store the current operation data in a draft data form; and after the data operation is finished, automatically checking file table data and draft table data of the same sewage outlet, and sequencing according to the role priority of the data operators.
The task and drain retrieval setting steps are as follows:
number with built-in drain outlet
Task name->
Task area->
And investigation personnel->
As search conditions, multi-dimensional search is selected as needed, and other search conditions O n The system also comprises a drain port type, a drain condition, a sea entering mode, a drain port state and a data archiving mode; according to actual business requirements, the investigation and monitoring of the drain outlet takes a task as a basic unit, and the information of the drain outlet to be investigated forms a data set, wherein a drain outlet basic attribute archive table and a monitoring item data table are associated; wherein the basic attribute file table of the sewage outlet file comprises a sewage outlet code, a national standard code and a name, and the monitoring item data table comprises a sewage outlet code, a sewage outlet longitude and latitude, a point position, an administrative division to which the sewage outlet belongs, a detailed address, a discharge condition, a sewage source, a sea entering mode, a surrounding environment, a sewage outlet type, a field photo and videoDescription, drain status, whether illegal, reasonable, whether vertical, whether labeling path, whether number sales, drain remediation status, sewage discharge, main pollutant type, responsibility main body and corresponding contact mode, drain setup and withdrawal time, on-site sensory and flow, COD, ammonia nitrogen, total phosphorus, salinity, pH, suspended matter, chloride, hexavalent chromium, organic carbon, BOD, arsenic, mercury, copper, lead, cadmium, chromium, zinc, nickel, oil, volatile phenol, fecal coliform group number, volatile halogenated hydrocarbon, administrative auditing status and remarking status, wherein a drain task table comprises task names, affiliated task areas, investigation personnel sets, drain code sets, a drain basic attribute profile table, a monitoring item data table and an investigation task table which are subjected to data association through drain code pairs for users to retrieve;
the related drain outlet basic attribute archive table, the monitoring item data table and the investigation task table are subjected to dynamic matching retrieval, and drain outlet data meeting the conditions are screened:
when the task is created, a task range is selected by an Excel file, an easy POI tool is used, drain outlet coding data in the task is extracted by using an importer Excel method, meanwhile, the data is written into a buffer zone, a drain outlet file data table is searched according to drain outlet coding comparison in the buffer zone, and a basic file data information generating task is obtained.
Wherein the monitoring items focused by different types of drain ports are different, a specific drain port is in a database and has no information in a plurality of monitoring items, no information monitoring item is meaningless for drain port evaluation, when drain port data are retrieved, drain port file data table data are retrieved according to drain port code traversal, monitoring items with non-empty monitoring item data in all retrieved data are combined into a basic set, finally the basic attribute field is combined with the monitoring item set to form a drain port attribute column, a data set of information matching and invalid monitoring items is formed, wherein the basic attribute field comprises drain port codes, national standard codes and names, the monitoring item set comprises a drain longitude and latitude, a point location, an administrative division, a detailed address, a discharge condition, a sewage source, a sea entering mode, a surrounding environment, a drain type, a field photograph and video and description, a drain state, whether illegal, reasonable, whether vertical, whether labeling a path, whether marketing, a drain remediation condition, a sewage discharge amount, a main pollutant type, a responsibility main body and a corresponding contact mode, drain setting and withdrawal time, a field sense and flow, COD, ammonia nitrogen, total phosphorus, salinity, pH, suspended matters, chloride, hexavalent chromium, organic carbon, BOD, arsenic, mercury, copper, lead, cadmium, chromium, zinc, nickel, oils, volatile phenols, fecal coliform number, volatile halogenated hydrocarbons, an administrative auditing state and a set of remarking conditions, and filling with empty data when attribute column data are missing; when the drain outlet data is exported, traversing and searching all drain outlet archive data table data according to the drain outlet coding set, merging monitoring items with non-empty monitoring item data in all the searched data into a basic set, finally taking the drain outlet coding and the name as basic attribute fields, merging and displaying drain outlet attribute columns by combining the monitoring item sets, and filling with empty data when the attribute column data is missing.
The invention takes accurate positioning attribute information as a principle, performs accurate matching positioning screening according to attribute conditions during searching, filters other useless or misleading conditions, makes complex comprehensive searching become more efficient, simultaneously completes huge searching within milliseconds, saves a great amount of time and provides better experience for users.
The foregoing is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art, who is within the scope of the present invention, should make equivalent substitutions or modifications according to the technical scheme of the present invention and the inventive concept thereof, and should be covered by the scope of the present invention.
Claims (9)
1. The sea-entering sewage outlet data reorganization evaluation method is characterized by comprising the following steps of:
s1, designing a drain outlet archive database;
s2, uploading a file formed by the data compression package of the sewage outlet to a sewage outlet archive database in real time, and automatically inputting historical archive data to the sewage outlet archive database;
s3, multi-dimensional data joint retrieval is carried out, and a new task is created.
2. The method for reorganizing and evaluating the data of the sewage outlet in the sea according to claim 1, wherein the step S1 is specifically:
a drain outlet archive data table is designed in a mode of combining the transverse direction and the longitudinal direction, wherein the transverse data is a drain outlet fixed attribute archive table, and the longitudinal data is a drain outlet monitoring item table; non-normalized data of the sewage outlet data in the sea, multi-source data cannot contain consistent monitoring items, and meanwhile, the monitoring items are not contained in the database, and the monitoring item table is supplemented according to the monitoring items of the data to form a sewage outlet monitoring item table;
the drain outlet file data table comprises fixed file attribute contents of a drain outlet, wherein the fixed file attribute comprises a drain outlet code and a drain outlet name; the sewage outlet monitoring item list comprises all monitoring items of the sewage outlet;
the monitoring item includes: the sewage outlet longitude and latitude, the point location, the administrative division to which the sewage outlet belongs, the detailed address, the discharge condition, the sewage source, the sea entering mode, the surrounding environment, the sewage outlet type, the field photograph and video and description, the sewage outlet state, whether illegal, whether reasonable, whether vertical, whether labeling path, whether marketing number, sewage outlet remediation condition, sewage discharge amount, main pollutant type, responsible body and corresponding contact mode, sewage outlet setup and withdrawal time, field sensory and flow, COD, ammonia nitrogen, total phosphorus, salinity, pH, suspended matters, chloride, hexavalent chromium, organic carbon, BOD, arsenic, mercury, copper, lead, cadmium, chromium, zinc, nickel, oil, volatile phenol, fecal coliform number, volatile halogenated hydrocarbon, administrative auditing state and remarking condition.
3. The method for reorganizing and evaluating the data of the sewage outlet in the sea according to claim 2, wherein in the step S2, when uploading a file formed by compressing and packaging the data of the sewage outlet in real time, the method specifically comprises:
2-1, packaging a plurality of types of files into a compressed file:
；
2-2, segmenting the file with more than 10M in the memory by adopting a slice slicing method, fixedly segmenting fragments with the size of 10M each time, simultaneously calculating the sequence number of the current segmented fragments and the total number of the file slices, generating uuid and carrying out the segmentation and the sequence number of the fragments, and submitting the sequence number to a server through a constructed form; if the submitting fails, re-submitting the fragmented file every 2 seconds, and if the submitting is successful, continuing the next fragmented until all the fragmented files are submitted successfully;
2-3, mutipartHttpServletRequest directed extraction Using File extractor each timeAnd checking according to uuid and fragment sequence number and storing as temporary file
The method comprises the steps of carrying out a first treatment on the surface of the Creating a file stream, then circulating a fragment sequence number, sequentially writing fragments into a buffer area, setting pointer offset of the file stream, sequentially extracting byte data from the buffer area, writing the byte data into the created file stream, and simultaneously moving the pointer offset of the file stream backwards according to the length of the byte data until all fragments of data are written into the file stream in sequence; deleting the temporary file of the fragment, simultaneously saving the merged file path to a database, and adding +.>
Wherein, F1: 1 st slice file, F2: slice 2 file, F3: slice 3 file, F4: 4 th slice file, fn: the nth slice file, F, is the sum of the slice files.
4. The method for reorganizing and evaluating the sewage outlet data into the sea according to claim 3, wherein in the step 2-1, the pictures and the videos are stored in a physical hard disk of a server, then the submitted data are converted into the sewage outlet archival data entity by combining the paths of the pictures and the videos, the sewage outlet monitoring item table is searched circularly according to the data in the entity to obtain the corresponding monitoring item ID, and finally the data in the entity and the monitoring item ID are stored and put in storage correspondingly.
5. The method for reorganizing and evaluating the data of the sewage outlet in the sea according to claim 4, wherein the step S3 is specifically:
3-1, inquiring all drain outlet sets by key information, and only reserving the key information of the drain outlet, wherein the key information comprises a drain outlet number and drain outlet longitude and latitude so as to realize rapid superposition of point positions in a map; when clicking or selecting the drain outlet for viewing, searching and displaying the drain outlet file by using the drain outlet code;
3-2, the map adopts an OpenLayers map, a built-in cluster method is used for superposing all the point location sets into the map, the pixel distance of point location aggregation is set to be 40, meanwhile, the pixel distance between the point locations is calculated, and the point locations smaller than 40 are aggregated and displayed.
6. The method for reorganizing and evaluating marine drain data according to claim 1, wherein in the step S2, when the history archive data is automatically recorded into the drain archive database, specifically:
2-4, traversing the search folder by using a GetDirectories method, splitting the folder, and obtaining the investigation time and the pollution discharge code in the corresponding investigation time;
2-5, traversing the files in the search folder by using a GetFiles method, and associating drain port codes and investigation time of the corresponding files;
2-6, moving the picture video file to the corresponding physical path by using a Move method, renaming the file by adding drain port codes and serial numbers in the checking time, and storing and warehousing data corresponding to the drain port codes of the corresponding file path.
7. The method for integrally compiling and evaluating the data of the sewage outlet in the sea according to claim 5, wherein in the step 3-1, specifically:
taking tasks as an operation set and a drain outlet as an operation unit, when a plurality of people operate drain outlet archival data simultaneously, firstly taking the drain outlet out of the operation set, locking the drain outlet by synchronized, preempting the drain outlet in time sequence, and searching from the operation set according to drain outlet codes when other people want to operate, wherein the drain outlet is taken out from the operation set, prompting that the drain outlet is being operated, and reassigning threads to store the current operation data in a draft data form; and after the data operation is finished, automatically checking file table data and draft table data of the same sewage outlet, and sequencing according to the role priority of the data operators.
8. The method for the complete compiling and evaluating of the data of the sewage outlet into the sea according to claim 7, wherein in the step 3-1, the task and the sewage outlet searching steps are as follows:
number with built-in drain outlet
Task name->
Task area->
And investigation personnel->
As search conditions, multi-dimensional search is selected as needed, and other search conditions O n The system also comprises a drain port type, a drain condition, a sea entering mode, a drain port state and a data archiving mode;
according to actual business requirements, the investigation and monitoring of the drain outlet takes a task as a basic unit, and the information of the drain outlet to be investigated forms a data set, wherein a drain outlet basic attribute archive table and a monitoring item data table are associated; the sewage outlet archive basic attribute archive table comprises sewage outlet codes, national standard codes and names, the monitoring project data table comprises sewage outlet codes, sewage outlet longitude and latitude, point positions, administrative division, detailed addresses, emission conditions, sewage sources, sea entering modes, surrounding environments, sewage outlet types, site photos and videos and descriptions, sewage outlet states, whether illegal, whether reasonable, whether vertical plates, whether labeling paths, whether sales numbers, sewage outlet treatment conditions, sewage discharge amounts, main pollutant types, responsibility main bodies and corresponding contact modes, sewage outlet setting and withdrawal time, site senses and flow, COD, ammonia nitrogen, total phosphorus, salinity, pH, suspended matters, chlorides, hexavalent chromium, organic carbon, arsenic, mercury, copper, lead, cadmium, chromium, zinc, nickel, oils, volatile phenols, fecal coliform numbers, volatile halogenated hydrocarbons, administrative states and remarking conditions, and the sewage outlet task table comprises task names, belonging task areas, a drain outlet set, sewage outlet set codes, sewage outlet basic attribute table and user attribute data are searched by the sewage outlet basic attribute data table;
the related drain outlet basic attribute archive table, the monitoring item data table and the investigation task table are subjected to dynamic matching retrieval, and drain outlet data meeting the conditions are screened:
when the task is created, a task range is selected by an Excel file, an easy POI tool is used, drain outlet coding data in the task is extracted by using an importer Excel method, meanwhile, the data is written into a buffer zone, a drain outlet file data table is searched according to drain outlet coding comparison in the buffer zone, and a basic file data information generating task is obtained.
9. The method for the complete compiling and evaluating the data of the sewage outlet into the sea according to claim 8, wherein the monitoring items focused by different types of sewage outlets are different, a specific sewage outlet is in a condition that a database stores no information in a plurality of monitoring items, no information monitoring item is meaningless for the sewage outlet evaluation, when the sewage outlet data is searched, the data of a file data table of the sewage outlet is searched according to the sewage outlet code traversal, the monitoring items with the data of the monitoring items not being empty in all the searched data are combined into a basic set, finally, the basic attribute field is combined with the monitoring item set to form a data set of the sewage outlet attribute column, the data set of the invalid monitoring items are matched and removed, wherein the basic attribute field comprises the sewage outlet code, the national standard code and the name, the monitoring item set comprises a drain longitude and latitude, a point location, an administrative division, a detailed address, a discharge condition, a sewage source, a sea entering mode, a surrounding environment, a drain type, a field photograph and video and description, a drain state, whether illegal, reasonable, whether vertical, whether labeling a path, whether marketing, a drain remediation condition, a sewage discharge amount, a main pollutant type, a responsibility main body and a corresponding contact mode, drain setting and withdrawal time, a field sense and flow, COD, ammonia nitrogen, total phosphorus, salinity, pH, suspended matters, chloride, hexavalent chromium, organic carbon, BOD, arsenic, mercury, copper, lead, cadmium, chromium, zinc, nickel, oils, volatile phenols, fecal coliform number, volatile halogenated hydrocarbons, an administrative auditing state and a set of remarking conditions, and filling with empty data when attribute column data are missing; when the drain outlet data is exported, traversing and searching all drain outlet archive data table data according to the drain outlet coding set, merging monitoring items with non-empty monitoring item data in all the searched data into a basic set, finally taking the drain outlet coding and the name as basic attribute fields, merging and displaying drain outlet attribute columns by combining the monitoring item sets, and filling with empty data when the attribute column data is missing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310416350.5A CN116150129B (en) | 2023-04-19 | 2023-04-19 | Sea-entry sewage outlet data reorganization evaluation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310416350.5A CN116150129B (en) | 2023-04-19 | 2023-04-19 | Sea-entry sewage outlet data reorganization evaluation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116150129A true CN116150129A (en) | 2023-05-23 |
| CN116150129B CN116150129B (en) | 2023-07-07 |
Family
ID=86341041
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310416350.5A Active CN116150129B (en) | 2023-04-19 | 2023-04-19 | Sea-entry sewage outlet data reorganization evaluation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116150129B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118039021A (en) * | 2024-04-11 | 2024-05-14 | 四川省铁路建设有限公司 | Sewage pollutant detection and alarm method and system |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1545067A (en) * | 2003-11-24 | 2004-11-10 | 宏 廖 | A method for compressing digitalized archive file using computer |
| TW200732930A (en) * | 2006-02-17 | 2007-09-01 | Jian-Shin Ye | Integration management method and system of computer files and readable record medium thereof |
| CN105630786A (en) * | 2014-10-27 | 2016-06-01 | 航天信息股份有限公司 | Car purchase tax electronic archive uploading, storing and querying system and method |
| US20180060177A1 (en) * | 2015-10-23 | 2018-03-01 | Oracle International Corporation | Pluggable database archive |
| CN109241022A (en) * | 2018-09-11 | 2019-01-18 | 天津理工大学 | A kind of archive management system and its ant search algorithm based on blue-ray storage |
| CN112632009A (en) * | 2020-12-29 | 2021-04-09 | 航天信息股份有限公司 | Electronic file processing method and device, storage medium and electronic equipment |
| CN115033552A (en) * | 2022-04-20 | 2022-09-09 | 生态环境部南京环境科学研究所 | Site soil pollutant environment and toxicity database construction method |
| CN115033905A (en) * | 2022-06-17 | 2022-09-09 | 江西睿创科技有限公司 | Wisdom archives management system platform based on thing networking |
-
2023
- 2023-04-19 CN CN202310416350.5A patent/CN116150129B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1545067A (en) * | 2003-11-24 | 2004-11-10 | 宏 廖 | A method for compressing digitalized archive file using computer |
| TW200732930A (en) * | 2006-02-17 | 2007-09-01 | Jian-Shin Ye | Integration management method and system of computer files and readable record medium thereof |
| CN105630786A (en) * | 2014-10-27 | 2016-06-01 | 航天信息股份有限公司 | Car purchase tax electronic archive uploading, storing and querying system and method |
| US20180060177A1 (en) * | 2015-10-23 | 2018-03-01 | Oracle International Corporation | Pluggable database archive |
| CN109241022A (en) * | 2018-09-11 | 2019-01-18 | 天津理工大学 | A kind of archive management system and its ant search algorithm based on blue-ray storage |
| CN112632009A (en) * | 2020-12-29 | 2021-04-09 | 航天信息股份有限公司 | Electronic file processing method and device, storage medium and electronic equipment |
| CN115033552A (en) * | 2022-04-20 | 2022-09-09 | 生态环境部南京环境科学研究所 | Site soil pollutant environment and toxicity database construction method |
| CN115033905A (en) * | 2022-06-17 | 2022-09-09 | 江西睿创科技有限公司 | Wisdom archives management system platform based on thing networking |
Non-Patent Citations (3)
| Title |
|---|
| WELLINGTON FRANCISCO DA SILVA;RENAN A.C.MELO;MARCELLA GROSSO;GABRIELA R.PEREIRA;DOUGLAS BRESSAN RIFFEL;: "Active Thermography Data-Processing Algorithm for Nondestructive Testing of Materials", IEEE ACCESS, pages 175054 - 175062 * |
| 王兰成;黄永勤;: "信息化条件下图片资料档案库规范化建设研究", 浙江档案, no. 05, pages 7 - 11 * |
| 花红霞;李朝晖;田曦;章瑞清;: "新一代天气雷达资料整编软件的应用", 智能城市, pages 46 - 47 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118039021A (en) * | 2024-04-11 | 2024-05-14 | 四川省铁路建设有限公司 | Sewage pollutant detection and alarm method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116150129B (en) | 2023-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20180365489A1 (en) | Automatically organizing images | |
| CN109446344B (en) | Intelligent analysis report automatic generation system based on big data | |
| US8990346B2 (en) | System and method for location monitoring based on organized geofeeds | |
| EP2122490B1 (en) | A method and system for integrating a social network and data repository to enable map creation | |
| US20170024415A1 (en) | Automatic discovery of popular landmarks | |
| CN111259006A (en) | Universal distributed heterogeneous data integrated physical aggregation, organization, release and service method and system | |
| CN102930060B (en) | A kind of method of database quick indexing and device | |
| CN116150129B (en) | Sea-entry sewage outlet data reorganization evaluation method | |
| KR20060026924A (en) | Tagging method and system for digital data | |
| US9026519B2 (en) | Clustering web pages on a search engine results page | |
| CN110362740B (en) | Water conservancy portal information hybrid recommendation method | |
| CN106682003A (en) | Path segmentation mapping method and device for distributed memory namespace | |
| Weber et al. | Journalism history, web archives, and new methods for understanding the evolution of digital journalism | |
| CN110795397B (en) | Automatic identification method for catalogue and file type of geological data packet | |
| CN113655968A (en) | Unstructured data storage method | |
| CN104462349A (en) | File processing method and file processing device | |
| CN110795520B (en) | Automatic identification method for association relation between digital geological data packet directory and file | |
| KR102491599B1 (en) | System for providing unified portal service for the search of cultural heritage | |
| CN108121760A (en) | A kind of mining analysis towards OGC geographic information services data is with recommending method | |
| CN114117174A (en) | Multi-format data screening management system based on big data | |
| Ardizzone et al. | Extracting touristic information from online image collections | |
| CN112966704A (en) | Urban memory resource classification labeling method and system | |
| CN111400579A (en) | Intelligent hardware search engine system | |
| Charrada et al. | Development of a database with ground truth for old documents analysis | |
| Deniziak et al. | Content Based Image Retrieval Using Modified Scalable Distributed Two-Layer Data Structure. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |