CN117056551A - File aggregation method and device for driving path, computer equipment and storage medium - Google Patents

File aggregation method and device for driving path, computer equipment and storage medium Download PDF

Info

Publication number
CN117056551A
CN117056551A CN202310833922.XA CN202310833922A CN117056551A CN 117056551 A CN117056551 A CN 117056551A CN 202310833922 A CN202310833922 A CN 202310833922A CN 117056551 A CN117056551 A CN 117056551A
Authority
CN
China
Prior art keywords
time
real
file
snapshot
node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310833922.XA
Other languages
Chinese (zh)
Other versions
CN117056551B (en
Inventor
请求不公布姓名
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Real AI Technology Co Ltd
Original Assignee
Beijing Real AI Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Real AI Technology Co Ltd filed Critical Beijing Real AI Technology Co Ltd
Priority to CN202310833922.XA priority Critical patent/CN117056551B/en
Publication of CN117056551A publication Critical patent/CN117056551A/en
Application granted granted Critical
Publication of CN117056551B publication Critical patent/CN117056551B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/587Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • G06F16/24552Database cache management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/29Geographical information databases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/51Indexing; Data structures therefor; Storage structures
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1095Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1097Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • H04L67/566Grouping or aggregating service requests, e.g. for unified processing
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • H04L67/568Storing data temporarily at an intermediate stage, e.g. caching
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/18Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Remote Sensing (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Library & Information Science (AREA)
  • Multimedia (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention provides a file aggregation method, a device, computer equipment and a storage medium of a driving path, relating to the technical field of image processing, wherein the method comprises the following steps: processing longitude and latitude of image acquisition equipment in the area to be monitored into GeoHash codes; obtaining a plurality of subareas according to the GeoHash codes; in the distributed system, mapping one computing node and a region to be monitored into a central node, and mapping a plurality of computing nodes and a plurality of sub-regions into a plurality of region nodes one by one; acquiring each snapshot through a central node and sending the snapshot to a regional node mapped by a subarea to which image acquisition equipment belongs; in each regional node, comparing each received snapshot with the acquired real-time file, and according to the comparison result, comparing each snapshot with the history file, and when the comparison fails, storing each snapshot as the real-time file; and when the comparison is successful, feeding back the identification information of the history file and each snapshot which are successfully compared to the central node.

Description

File aggregation method and device for driving path, computer equipment and storage medium
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method and apparatus for aggregating files in a driving path, a computer device, and a storage medium.
Background
The ten-thousand-way file gathering is required to track the personnel track in real time under the condition of multiple cameras, and the ten-thousand-way file gathering refers to cameras with ten-thousand ways as orders of magnitude, and the specific number of ways is different, so that under the condition of multiple cameras, if files gathering (file gathering for short) of a specific personnel driving path is expected to be realized, the file gathering is difficult to realize by only one server at present, the capacity of one server is limited, the single-node processing capacity of one server is only in hundred ways, and the file gathering of multiple ways of cameras with ten-thousand orders of magnitude cannot be realized at all; in addition, when the file aggregation is performed by means of a single node of one server, if the server fails, the file cannot be recovered, and smooth completion of the file aggregation cannot be ensured.
Disclosure of Invention
In view of the above, the embodiment of the invention provides a file aggregation method for a driving path, so as to solve the technical problems that in the prior art, a single node cannot realize ten-thousand-path file aggregation and a single node failure cannot be recovered. The method comprises the following steps:
acquiring the longitude and latitude of the geographic position of at least one image acquisition device in the region to be monitored, and processing the longitude and latitude of the geographic position of the image acquisition device into a GeoHash code;
Dividing the geographic position of the image acquisition equipment with the same first preset number of prefix characters in the GeoHash code into a sub-area so as to divide the area to be monitored into a plurality of sub-areas;
in the distributed system, mapping a computing node with the area to be monitored, wherein the computing node is called a central node, mapping a plurality of computing nodes with a plurality of subareas one by one, each computing node in the plurality of computing nodes is called an area node, and distributing a cache block for each area node in the same cache region according to the geographic position adjacent relation among the plurality of subareas, so that the cache blocks of the area nodes mapped by adjacent subareas are adjacent cache blocks;
acquiring each snapshot generated by the image acquisition equipment in the area to be monitored through the central node, determining a corresponding subarea according to a GeoHash code of the geographic position of the image acquisition equipment for generating each snapshot, and distributing each snapshot to an area node mapped with the determined subarea, wherein each snapshot comprises a set of pictures generated by the same image acquisition equipment for the same individual at preset time;
In each regional node, acquiring real-time files from a self cache block and an adjacent cache block, comparing each snapshot received by the regional node with the acquired real-time files to obtain a first comparison result, comparing each snapshot with the history files according to the first comparison result to obtain a second comparison result, and storing each snapshot in the self cache block as the real-time files when the second comparison result is that the comparison fails; when the second comparison result is that the comparison is successful, the identification information of the history file and each snapshot which are successfully compared are fed back to the central node;
and merging each snapshot into the corresponding history file according to the identification information of the history file successfully compared in the central node.
The embodiment of the invention also provides a file aggregation device of the driving path, which aims to solve the technical problem that in the prior art, a single node cannot realize ten-thousand-path file aggregation and a single node failure cannot be recovered. The device comprises:
the coding module is used for acquiring the longitude and latitude of the geographic position of at least one image acquisition device in the area to be monitored and processing the longitude and latitude of the geographic position of the image acquisition device into a GeoHash code;
The region dividing module is used for dividing the geographic position of the image acquisition equipment with the same first preset number of prefix characters in the GeoHash code into a sub-region so as to divide the region to be monitored into a plurality of sub-regions;
the node mapping module is used for mapping one computing node with the area to be monitored in the distributed system, the computing node is called a central node, a plurality of computing nodes and a plurality of subareas are mapped one by one, each computing node in the plurality of computing nodes is called an area node, and a cache block is distributed for each area node in the same cache region according to the geographic position adjacent relation among the plurality of subareas, so that the cache blocks of the area nodes mapped by adjacent subareas are adjacent cache blocks;
the central node operation module is used for acquiring each snapshot generated by the image acquisition equipment in the area to be monitored through the central node, determining a corresponding subarea according to the GeoHash code of the geographic position of the image acquisition equipment for generating each snapshot, and distributing each snapshot to the area node mapped with the determined subarea, wherein each snapshot comprises a set of pictures generated by the same image acquisition equipment for the same individual at preset time;
The regional node operation module is used for acquiring real-time files from the self cache block and the adjacent cache blocks in each regional node, comparing each snapshot received by the regional node with the acquired real-time files to obtain a first comparison result, comparing each snapshot with the history files according to the first comparison result to obtain a second comparison result, and storing each snapshot into the self cache block as the real-time file when the second comparison result is that the comparison fails; when the second comparison result is that the comparison is successful, the identification information of the history file and each snapshot which are successfully compared are fed back to the central node;
and the central node operation module is further used for merging each snapshot into the corresponding history file according to the identification information of the history file successfully compared in the central node.
The embodiment of the invention also provides computer equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the file aggregation method of any running path is realized when the processor executes the computer program, so that the technical problems that ten-thousand-path file aggregation cannot be realized by a single node and single-node faults cannot be recovered in the prior art are solved.
The embodiment of the invention also provides a computer readable storage medium which stores a computer program for executing the file aggregation method of any running path, so as to solve the technical problems that a single node cannot realize ten-thousand-path file aggregation and a single node fault cannot be recovered in the prior art.
Compared with the prior art, the beneficial effects that above-mentioned at least one technical scheme that this description embodiment adopted can reach include at least: the method comprises the steps of processing longitude and latitude of a geographic position of an image acquisition device into GeoHash codes, dividing a region to be monitored into a plurality of subareas based on the GeoHash codes, mapping the subareas with a plurality of computing nodes in a distributed system one by one to obtain a plurality of region nodes mapped with each subarea respectively, mapping the region to be monitored with one computing node of the distributed system to obtain a central node, finally, acquiring each snapshot generated by the image acquisition device in the region to be monitored through the central node, distributing each snapshot to the region node mapped by the subarea of the image acquisition device, comparing each snapshot received by the regional node with the acquired real-time files in parallel, comparing each snapshot with the history files according to a first comparison result, and caching or merging each snapshot into the corresponding history files according to a second comparison result of each snapshot and the history files. The method has the advantages that the capture generated by the image acquisition equipment is subjected to gear gathering processing by adopting a plurality of regional nodes of the distributed system in parallel, the gear gathering capability of the capture with high concurrence is improved, the capture gear gathering under the condition of ten-thousand cameras can be supported, and the efficiency of ten-thousand gear gathering is improved; meanwhile, a plurality of regional nodes perform file aggregation processing on the snap shots generated by the image acquisition equipment in parallel, so that the defect that file aggregation cannot be recovered due to single-node faults can be avoided, and even if a certain regional node fault occurs, other regional nodes can still perform file aggregation processing normally, thereby being beneficial to ensuring smooth completion of file aggregation; in addition, the subareas are obtained based on the GeoHash codes of the geographic positions of the image acquisition devices, the number of the subareas can be flexibly and real-timely adjusted based on different requirements according to the adjustment of the same prefix character number of the GeoHash codes, the number of the image acquisition devices can be flexibly increased or reduced in each subarea, the subareas can be newly increased according to the geographic positions of the newly increased image acquisition devices, the number of the area nodes mapped with the subareas one by one can be flexibly and real-timely adjusted, and the expansibility and the high availability are improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for aggregating files of a driving path according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a distributed system according to an embodiment of the present application;
fig. 3 is a schematic flow chart of a gear gathering process for each snapshot by each regional node according to an embodiment of the present application;
fig. 4 is a schematic diagram of a zipper alignment according to an embodiment of the present application;
FIG. 5 is a block diagram of a computer device according to an embodiment of the present application;
fig. 6 is a block diagram of a file aggregation device for a driving path according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in detail below with reference to the accompanying drawings.
Other advantages and effects of the present application will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present application with reference to specific examples. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. The application may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present application. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
In an embodiment of the present invention, a method for aggregating files of a driving path is provided, as shown in fig. 1, where the method includes:
step S101: acquiring the longitude and latitude of the geographic position of at least one image acquisition device in the region to be monitored, and processing the longitude and latitude of the geographic position of the image acquisition device into a GeoHash code;
step S102: dividing the geographic position of the image acquisition equipment with the same first preset number of prefix characters in the GeoHash code into a sub-area so as to divide the area to be monitored into a plurality of sub-areas;
step S103: in the distributed system, mapping a computing node with the area to be monitored, wherein the computing node is called a central node, mapping a plurality of computing nodes with a plurality of subareas one by one, each computing node in the plurality of computing nodes is called an area node, and distributing a cache block for each area node in the same cache region according to the geographic position adjacent relation among the plurality of subareas, so that the cache blocks of the area nodes mapped by adjacent subareas are adjacent cache blocks;
step S104: acquiring each snapshot generated by the image acquisition equipment in the area to be monitored through the central node, determining a corresponding subarea according to a GeoHash code of the geographic position of the image acquisition equipment for generating each snapshot, and distributing each snapshot to an area node mapped with the determined subarea, wherein each snapshot comprises a set of pictures generated by the same image acquisition equipment for the same individual at preset time;
Step S105: in each regional node, acquiring real-time files from a self cache block and an adjacent cache block, comparing each snapshot received by the regional node with the acquired real-time files to obtain a first comparison result, comparing each snapshot with the history files according to the first comparison result to obtain a second comparison result, and storing each snapshot in the self cache block as the real-time files when the second comparison result is that the comparison fails; when the second comparison result is that the comparison is successful, the identification information of the history file and each snapshot which are successfully compared are fed back to the central node;
step S106: and merging each snapshot into the corresponding history file according to the identification information of the history file successfully compared in the central node.
As can be seen from the flow shown in fig. 1, in the embodiment of the present invention, the capture generated by the image acquisition device is subjected to the file gathering processing by using multiple area nodes of the distributed system in parallel, so that the file gathering capability of capturing with high concurrence is improved, so that capturing files under the condition of ten-thousand cameras can be supported, and further the efficiency of ten-thousand files gathering is improved; meanwhile, a plurality of regional nodes perform file aggregation processing on the snap shots generated by the image acquisition equipment in parallel, so that the defect that file aggregation cannot be recovered due to single-node faults can be avoided, and even if a certain regional node fault occurs, other regional nodes can still perform file aggregation processing normally, thereby being beneficial to ensuring smooth completion of file aggregation; in addition, the subareas are obtained based on the GeoHash codes of the geographic positions of the image acquisition devices, the number of the subareas can be flexibly and real-timely adjusted based on different requirements according to the adjustment of the same prefix character number of the GeoHash codes, the number of the image acquisition devices can be flexibly increased or reduced in each subarea, the subareas can be newly increased according to the geographic positions of the newly increased image acquisition devices, the number of the area nodes mapped with the subareas one by one can be flexibly and real-timely adjusted, and the expansibility and the high availability are improved.
In specific implementation, the file aggregation method of the driving path can be applied to a file aggregation scene of a human activity track, and at the moment, the individual characteristic of each picture in the snapshot can be the biological characteristic of a person; the method can also be used for a gear gathering scene of the moving track of a moving object, wherein the moving object can be traffic equipment such as automobiles, mobile equipment such as mobile robots and the like, and at the moment, the individual characteristic of each picture in the snapshot can be the identification characteristic of the moving object, such as license plate numbers, id codes of the mobile robots and the like.
In particular, the GeoHash code is a spatial address coding method, which can code two-dimensional spatial latitude and longitude data into a character string, for example, the latitude and longitude is [31.1932993,121.43960190000007], and the final result of the GeoHash code is wtw37q. And forming a grid region by using the length of the character string as the precision and calculating the longitude and latitude of the same precision, wherein the character string formed by the longitude and latitude of a certain grid region under certain precision under the condition that the precision is consistent (namely, the first preset number of prefix characters in the GeoHash code of the longitude and latitude of the certain grid region are the same). The application provides a method for processing longitude and latitude of a geographic position of an image acquisition device into a GeoHash code, and dividing the geographic position of the image acquisition device with the same first preset number of prefix characters in the GeoHash code into a sub-area so as to divide an area to be monitored into a plurality of sub-areas.
In the implementation, in the process of dividing the subareas, the geographical position of the image acquisition device with the same prefix characters in the first preset number in the GeoHash code is divided into a subarea, and in order to simultaneously consider the calculation amount and the accuracy of the file gathering, the numerical value of the first preset number can be determined according to the specific precision requirement, the area range size related to the track and the like, the smaller the numerical value of the first preset number is, the more the number of the image acquisition devices included in each subarea is, the larger the geographical space area distributed by the image acquisition device included in each subarea is, the larger the calculation amount of the nodes relative to each area is, and the lower the accuracy of the file gathering is. For example, the first preset number may be 5, 6, etc.
In particular, the image capturing device may be any device capable of capturing images, such as a video camera, a camera, or the like.
In specific implementation, the subregion is divided based on GeoHash coding, so that expansibility can be improved. For example, since the number of area nodes is determined according to the number of sub-areas, the deployment of the area nodes can be completed by modifying the relevant settings of the distributed system in case of a change in the number of sub-areas. In addition, in the distributed system, the camera can be dynamically increased by observing the information of the camera to which each snapshot belongs, and the socket connection condition between the central node and the regional node of the regional mapping to which the increased camera belongs can be adjusted accordingly, so that the dynamic capacity expansion of the regional node of the distributed system can be completed. For example, if the GeoHash code of the newly added camera falls in the existing subarea, only the information of the camera in the corresponding area node needs to be modified; if the GeoHash code falls outside the existing subarea, the subarea is newly added, the area node is newly added, and the camera is distributed to the newly added area node.
In the implementation, as shown in fig. 2, after mapping the multiple sub-areas with the multiple computing nodes, multiple area nodes are obtained, and a buffer block is allocated to each area node in the same buffer area, where each buffer block is used to store a real-time file of a corresponding area node, and the buffer blocks of the area nodes mapped by adjacent sub-areas are adjacent buffer blocks, that is, the buffer blocks corresponding to the area nodes mapped by the sub-areas with adjacent geographic positions also have a relationship of adjacent positions. After kafka (a high throughput, durable and distributed message queue system for publishing and subscribing) consumes and takes data (i.e. a snapshot), the central node automatically sends the data to corresponding regional nodes according to the conditions of a subarea and the like of a camera generated by the snapshot, each regional node carries out relevant file gathering processing on the received snapshot, each regional node feeds back file gathering processing results to the central node, the central node continuously monitors an uploading client, and if the file gathering results are uploaded, the results are sent to kafka or mysql for storage according to the needs.
In particular implementations, in a distributed system, a central node and a regional node may be initialized by, for example,
1. Requesting an API node (or only a script), calculating the partition condition of the subareas according to the GeoHash codes of all cameras, and determining the number of the subareas;
2. according to the number of the subareas, firstly setting up a central node for running the central node mirror image in a distributed system (for example, k8 s), and starting up area nodes for setting up a plurality of running area node mirror images at the central node;
3. the central node and the regional nodes are connected by sockets to perform initialized setting, which comprises the following steps:
a. establishing a channel between a central node and an area node;
b. distributing cameras of the subareas to corresponding area nodes;
c. notifying the regional node of the number of its neighboring regional nodes in order to subscribe to the cache block (e.g., redis cache block);
4. after the setup is completed, the region node will start to continuously request data from the center node (i.e., snapshot) until a stop instruction of the center node is received.
In particular, once the distributed system starts to operate, the system-level termination is only performed if the following occurs:
control the central node to stop, clear and exit through the API node (or just a script); or,
the central node receives the stop information from kafka, cleans up and exits; or,
The central node exits abnormally. The system restart is performed under the termination condition of the system layer.
In particular, the buffer area used for allocating the buffer block to each regional node may be a shared buffer, for example, a redis, which is a data structure server stored in a memory and may be used as a database, a cache and a message queue agent. In this embodiment, it is proposed that a redis is used to allocate a cache block for each area node, so that each area node corresponds to a redis cache block, and further, data exchange and synchronization between adjacent area nodes can be performed based on the redis cache blocks.
In specific implementation, the above-mentioned distributed system may be any distributed system capable of implementing computation, for example, k8s is short for Kubernetes, which is used as a bottom management architecture of the distributed system.
In the specific implementation, in order to realize accurate file gathering of the snapshots in each regional node, the process of gathering the files of each snapshot is provided to comprise two stages of comparison with the real-time files and comparison with the history files, and each snapshot is gathered according to the comparison results of the two stages. The real-time file refers to a temporary file generated in the T+0 real-time file gathering system. The history file refers to a history snapshot file gathering result obtained by the T+1 file gathering service operation, and the real-time file is used as a formal file and becomes a part of the formal file after being combined into the formal file. Archives refer to a collection of snapshots calculated during the system operation, each archive is considered in the algorithm as all snapshots corresponding to a person or a moving object equivalent to a body, but the actual result may have a person multiple and a person multiple. One or more snapshots may be within a file. The file aggregation is a short term of file aggregation, and refers to a process of judging whether a snapshot can be put into a file or become a new file.
In the specific implementation, in the process of comparing with the real-time file, in order to improve the accuracy and the comparison efficiency of the comparison, a process of realizing the comparison of the snapshot and the real-time file through a plurality of parallel threads is proposed, for example,
setting a plurality of parallel threads in each regional node to execute a process of comparing each snapshot received by the regional node with the acquired real-time file;
in each thread, aiming at each snapshot, sequencing the pictures according to the sequence from the large to the small of the individual characteristic quality of each picture to obtain a first sequence, and determining the pictures before the corresponding ranking of the second preset number in the first sequence as first key frames to obtain the second preset number of first key frames;
for each obtained real-time file, sequencing the pictures according to the sequence from big to small of the individual characteristic quality of each picture to obtain a second sequence, determining the pictures before the second preset number is correspondingly ranked in the second sequence as second key frames, and determining the second preset number of the second key frames of each real-time file;
calculating first similarity of individual features between each first key frame and each second key frame respectively to obtain a plurality of first similarity;
Sequencing the first similarities according to a sequence from large to small to obtain a third sequence, and determining the first similarities before the arrangement corresponding to a third preset number in the third sequence as key similarities to obtain the third preset number of key similarities;
counting each real-time archive according to the number of the key similarity corresponding to each real-time archive;
traversing each real-time file in sequence from large to small according to the counting, determining effective real-time files, and when the number of the effective real-time files is 0, determining that the first comparison result is failure in comparison, and not processing each snapshot; when the number of the effective real-time files is not 0, the first comparison result is that the comparison is successful, each effective real-time file and each snapshot are combined to form a new real-time file, and each effective real-time file is deleted from the corresponding cache block, wherein the effective real-time file is a real-time file which is not combined currently and is not held by other threads currently.
In specific implementation, as shown in fig. 3, in each regional node, a process of comparing a snapshot received from a central node with a real-time file is performed through a plurality of parallel threads, in this process, the snapshots may be allocated and scheduled to each thread for processing according to the receiving order of the snapshots, or the received snapshots may be put into a queue, and then the snapshots may be scheduled to each thread for processing based on the queue.
In the embodiment, as shown in fig. 3, it is proposed that after receiving each snapshot from a central node, the regional node performs data analysis on additional information (for example, folprint information, which reflects a unique number of a track to which the snapshot belongs) carried by each snapshot, so as to determine whether each snapshot is a track-carrying snapshot (i.e., whether each snapshot has a track to which the track belongs, if yes, the snapshot is a track-carrying snapshot, if no, the snapshot is a track-free snapshot), and if not, each snapshot is directly scheduled to each thread according to a receiving order or after being put into a queue, and then processed based on the queue order; if yes, placing the snapshot with the track into a snapshot buffer pool, screening out a representative snapshot from the snapshots of a plurality of tracks which are received in a preset time period and belong to the same track according to track information of the snapshot with the track, and scheduling the screened snapshots to each thread for processing according to a receiving sequence or a queue sequence; meanwhile, each snapshot may belong to multiple tracks at the same time, and when one snapshot is screened out in a certain track and scheduled to each thread for processing, the snapshot is deleted from other tracks to which the snapshot belongs.
During implementation, in the process of comparing the snapshot with the real-time file, the real-time file can be counted and the effective real-time file can be determined through the following steps:
1. multiple threads in each regional node can generate multiple merging demands at the same time, and in order to improve the concurrency degree to the greatest extent, a competition strategy is adopted;
a. calculating the similarity of k key frames of each real-time file and the snap shot k key frames to obtain a plurality of similarities corresponding to each real-time file, and taking top k times k similarities from the similarities corresponding to all the real-time files as key similarities;
b. searching the real-time files to which the key k is belonged for the key k, and correspondingly counting +1 for the real-time files to which each key similarity is belonged;
c. real-time files that compete for real-time file ordering from more to less by count.
d. The effective real-time file refers to a real-time file which is not merged and not held by a certain thread, and the count/key frame number k can be greater than a certain threshold;
2. setting an uncombined valid bit for each real-time archive;
limiting each real-time file to be modified by one thread, locking, and successfully competing a certain real-time file (obtaining the lock of the file), wherein after the certain real-time file is modified or combined by one thread, the valid bit of the corresponding real-time file is set to be invalid, so that whether the corresponding real-time file is combined or not can be judged based on the state of the valid bit; if the competition of a certain real-time file is unsuccessful, competing for the next real-time file; if none of the real-time files is successful, i.e., there is no valid real-time file, file merging is not performed.
In the specific implementation, in the process of determining a key frame for each snapshot and real-time file, when the application object of the file aggregation method of the driving path is a person, the individual characteristic quality score of each picture can be a face quality score, and the size of the face quality score is in direct proportion to the shooting completeness and definition of the face characteristic; when the application object of the file aggregation method of the driving path is a moving object, the individual characteristic quality score of each picture can be the quality score of the object identifier, and the size of the quality score of the object identifier is in direct proportion to the shooting completeness and definition of the object identifier (such as a license plate number, an ID number and the like).
In the implementation, in each regional node, in the process of comparing the current snapshot with the real-time file, each regional node may acquire the real-time file from one or more neighboring regional nodes, as shown in fig. 3 (the neighboring regional cache shown in fig. 3 represents the cache block corresponding to the neighboring regional node), take the real-time file acquired from 8 neighboring regional nodes of the regional node as an example, at this time, the 8 neighboring regional nodes synchronize the real-time file in the current cache block to the regional node, and further compare the real-time file in the cache block of the regional node with the acquired real-time file respectively with the current snapshot, for example, compare 3 real-time files such as the real-time file 1 of the regional node, the real-time file 2 of the neighboring regional node 5, and the real-time file 3 of the neighboring regional node 6 with the current snapshot, take the successful comparison as an example, for example, then combine the current snapshot with the real-time file 1, 3 as an effective real-time file, form a new real-time file, store the new real-time file into the block of the regional node, and simultaneously, and delete the real-time file 1, 3 from the neighboring regional node and the local snapshot 1, the real-time file 3 from the regional node and the current snapshot 1, and the real-time file 3, or the other regional file are prevented from being repeatedly deleted (or the result of the combined with the real-time file 1). If the comparison between the 3 real-time files and the current snapshot fails, namely, no effective real-time file exists in the 3 real-time files, the current snapshot is regarded as the real-time file at the moment, and no change processing is performed.
In particular, in order to further improve the accuracy of comparison between the snapshot and the real-time file, in this embodiment, it is proposed to count the real-time file by combining the key similarity and the space-time factor, so as to accurately determine the effective real-time file. For example, from the key similarities corresponding to each of the real-time archives, a number of the key similarities for which the spatio-temporal information between the first key frame and the second key frame meets a first time threshold and meets a first distance threshold is determined, and the number is determined as a count of each of the real-time archives.
In the specific implementation, in the process of combining the snapshot and the real-time file to form a new real-time file, in order to further improve the accuracy of the combining process and further improve the precision of file aggregation, in this embodiment, a time-space requirement is provided to combine the snapshot and the effective real-time file. For example, merging the effective real-time files of the first rank and each snapshot according to the order from the big count to the small count, after forming the current real-time file, traversing each other effective real-time file in turn according to the order from the big count to the small count, and executing the following steps in a circulating way:
Judging whether the track of the effective real-time file traversed currently and the track of the current real-time file meet space-time preset requirements or not;
if not, ending the circulation, and determining the current real-time file as the new real-time file; if so, merging the effective real-time file currently traversed with the current real-time file, and regarding the formed file as a new current real-time file.
In particular, in order to accurately determine whether the track of the currently traversed effective real-time file and the track of the current real-time file meet the space-time preset requirement, it is proposed to use a "zip-type" for track comparison, for example,
according to the time stamp sequence of the pictures, mixing and sorting the pictures in the effective real-time file traversed currently and the pictures in the current real-time file;
for each picture in the current real-time file, when two pictures on the front and rear rows of the picture are pictures in the effective real-time file which is traversed currently, calculating the time difference between the time stamp of the picture and the time stamp of the two pictures on the front and rear rows and the distance between the picture and the affiliated image acquisition equipment; when two pictures on the front and rear rows of the picture and the picture with the same time stamp as the picture are pictures in the effective real-time file which is traversed currently, calculating the time difference between the time stamp of the picture and the time stamp of the two pictures on the front and rear rows and the distance between the picture and the image acquisition equipment to which the picture with the same time stamp belongs, and calculating the distance between the picture and the image acquisition equipment to which the picture with the same time stamp belongs; when one of the two pictures on the front and rear rows of the picture is a picture in the effective real-time archive currently traversed and the other picture is a picture in the current real-time archive, calculating the time difference between the picture and the time stamp of the one picture and the distance between the picture and the affiliated image acquisition equipment, and calculating the time difference between the time stamp of the other picture and the picture on the adjacent row of the picture and the distance between the picture and the affiliated image acquisition equipment;
Judging whether the distance corresponding to each picture in the current real-time file is smaller than a second distance threshold value and whether the time difference corresponding to each picture is smaller than a second time threshold value, if yes, judging that the track of the effective real-time file traversed currently and the track of the current real-time file meet space-time preset requirements; if not, judging that the track of the effective real-time file currently traversed and the track of the current real-time file do not meet the space-time preset requirement.
In the implementation, as shown in fig. 4, a track a represents a track of the currently traversed effective real-time file, and numbers 1, 2, 3 and … … corresponding to the track a represent each picture included in the currently traversed effective real-time file; track B represents the track of the current real-time file, and the corresponding digital numbers 1, 2, 3 and … … of track B represent each picture included in the current real-time file; for example, the pictures 1 and 5 in the track B are the case that two pictures on the front and rear rows of the picture are the pictures in the effective real-time archive currently traversed, and the picture 4 in the track B is the case that two pictures on the front and rear rows of the picture and the picture with the same timestamp as the picture are both the pictures in the effective real-time archive currently traversed; the pictures 2 and 3 in the track B are the pictures in the effective real-time file which is currently traversed, one picture in the two pictures in the front and rear rows of the picture is the picture in the effective real-time file which is currently traversed, and the other picture is the picture in the real-time file, so that whether the track of the effective real-time file which is currently traversed and the track of the current real-time file meet the space-time preset requirement can be judged by judging whether the distance corresponding to each picture in the current real-time file is smaller than a second distance threshold value and whether the time difference corresponding to each picture is smaller than a second time threshold value.
In a specific implementation, after comparing the snapshot with the real-time file, the snapshot needs to be compared with the history file according to a first comparison result, so as to determine a final file gathering result, and in this embodiment, in order to improve the precision of the file gathering result, a process of comparing the snapshot with the history file is proposed by the following steps:
for each history file, sequencing the pictures according to the sequence from big to small of individual characteristic quality of each picture to obtain a fourth sequence, determining the pictures before the second preset number is ranked corresponding to the second preset number in the fourth sequence as third key frames, and determining the second preset number of the third key frames of each history file;
when the first comparison result is that the comparison fails, calculating second similarity of individual features between each first key frame and each third key frame of each history file to obtain a plurality of second similarity, and selecting a fourth preset number of second similarity before arrangement from the sequence of the second similarity from large to small to sum, wherein the sum is the similarity sum of each history file;
When the maximum similarity sum is greater than a preset threshold, determining that the second comparison result is successful, determining the history file corresponding to the maximum similarity sum as the history file which is successfully compared, and feeding back identification information of the history file which is successfully compared and each snapshot to the central node; when the maximum similarity sum is smaller than the preset threshold value, determining that the second comparison result is a comparison failure, and storing each snapshot in a self cache block as a real-time file; or,
when the first comparison result is that the comparison is successful, sequencing the pictures in the new real-time file according to the sequence from big to small of the individual characteristic quality scores of each picture to obtain a fifth sequence, and determining the pictures before the second preset number is correspondingly ranked in the fifth sequence as a fourth key frame;
calculating third similarity of individual features between each fourth key frame and each third key frame of each history file to obtain a plurality of third similarity, and selecting third similarity before ranking corresponding to the fourth preset number from the sequence of the plurality of third similarity from large to small to sum, wherein the sum is the similarity sum of each history file;
When the maximum similarity sum is greater than the preset threshold value, determining that the second comparison result is successful in comparison, determining the history file corresponding to the maximum similarity sum as the history file which is successful in comparison, and feeding back identification information (such as the serial number of the history file and the like) of the history file which is successful in comparison and the new real-time file to the central node; and when the maximum similarity sum is smaller than the preset threshold value, determining that the second comparison result is failed in comparison, and storing the new real-time file as a real-time file in a self cache block.
In specific implementation, in the process of comparing the snapshot (or new real-time file) table with the history file, the comparison can be performed in the following manner:
1. when calculating the similarity between the key frames of the snapshot or the new real-time file and the history file, if the individual is a person, the individual features for calculating the similarity can only consider the face features (the confidence of the human features is not high) so as to improve the precision;
2. calculating the similarity of k key frames of the snapshot or new real-time files (namely the second preset number) and k key frames of each history file, performing top k search on the calculated multiple similarities, counting the sum of the similarity of top k under each history file, and sequentially arranging the sum of each history file from large to small;
3. Traversing the arranged results, and finding a history file with the first sum larger than a preset threshold value as a successfully-compared history file;
4. the identification information of the successfully compared history file and the snapshot (or the new real-time file) are sent to the central node.
In specific implementation, as shown in fig. 3, in the process of comparing with the history file, according to the first comparison result of the snapshot and the real-time file, the snapshot (or the new real-time file) is compared with the history file, at this time, the history file may adopt a real-name history file and/or an anonymous history file, fig. 3 is used to adopt the real-name history file and the anonymous history file, when the snapshot (or the new real-time file) is successfully compared with the real-name history file, the identification information of the snapshot (or the new real-time file) and the successfully compared real-name history file is fed back to the central node; if the comparison fails, comparing the snapshot (or the new real-time archive) with the anonymous historical archive, if the comparison is successful, feeding back the identification information of the snapshot (or the new real-time archive) with the anonymous historical archive which is successfully compared to the central node, and if the comparison fails, storing the snapshot (or the new real-time archive) into a cache block of the local node.
In particular, in order to further improve the high availability of the file aggregation method of the driving path, a method for restarting the inactivated area node is provided, so as to avoid that the inactivation of a certain area node affects the progress or the accuracy of file aggregation. For example, the central node puts each snapshot distributed to each regional node into a queue, wherein each snapshot is provided with a unique ID;
When each snapshot is distributed to the regional node mapped with the determined subarea through the central node, receiving the ID of the earliest received snapshot fed back by the regional node;
when the inactivated area node exists in the plurality of area nodes, screening the ID of each snapshot before the received ID and the ID of each snapshot cached in a cache block of the inactivated area node from the IDs of each snapshot in the queue through the central node, and determining each snapshot corresponding to the rest IDs as an unprocessed snapshot of the inactivated area node;
restarting a new regional node through the central node, and establishing a binding relation between the new regional node and a cache block of the inactivated regional node;
and redistributing the unprocessed snapshots of the inactivated regional nodes to the regional nodes mapped with the determined subareas according to the sequence of the time stamps through the central node.
In particular, in order to discover and restart an inactivated area node in time, the central node may also detect the inactivated area node by:
1. the central node distributes snapshot to the regional nodes, and the regional nodes do not respond within the preset time length, or the preset time length does not receive a new data request of the regional nodes, or the regional nodes with the preset time length do not send keep-alive messages to the central node, so that the regional nodes can be judged to be invalid.
The preset time period can be determined after comprehensive consideration in the aspects of network, load and the like, and the time beyond the preset time period is ensured to be the cause of node faults.
2. It is necessary to determine whether the inactivated area node is restarted by k8s, and if not, a k8s forced restart is performed.
3. And closing old connection with the socket of the inactivated regional node by the central node, monitoring the connection of the new regional node, and sending a node restarting instruction to the new regional node after the connection is successful.
In particular, in order to further improve the high availability of the file aggregation method of the driving path, a method for recovering operation for the inactivation of the central node is provided, so that the operation can be automatically recovered after the inactivation of the central node, and the connection with the previous regional node is established. For example, the central nodes can form a central node cluster, only one central node in a plurality of central nodes in each time cluster provides service to the outside, the rest central nodes are distributed on different hosts for redundant processing, and the data synchronization among the central nodes is kept. Or, a proxy mechanism can be used, after the central node outside the central node cluster is deactivated, a new central node can be rapidly selected to continue service, so that the service is not interrupted, and only the information needs to be forwarded to the new central node. Or the agent not only forwards the information, but also caches the information so as to cope with partial data loss during service switching. Alternatively, the agent may choose a mature framework such as a relatively mature ngix to ensure high availability. The data synchronization and information caching between the central nodes also depend on redis, so that high availability is ensured.
In this embodiment, a computer device is provided, as shown in fig. 5, including a memory 501, a processor 502, and a computer program stored in the memory and capable of running on the processor, where the processor implements the file aggregation method of any of the above travel paths when executing the computer program.
In particular, the computer device may be a computer terminal, a server or similar computing means.
In the present embodiment, there is provided a computer-readable storage medium storing a computer program for executing the archive aggregation method of any of the travel paths described above.
In particular, computer-readable storage media, including both permanent and non-permanent, removable and non-removable media, may be used to implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable storage media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
Based on the same inventive concept, the embodiment of the invention also provides a file gathering device of a driving path, as described in the following embodiment. Since the principle of the file aggregation device of the driving path for solving the problem is similar to that of the file aggregation method of the driving path, the implementation of the file aggregation device of the driving path can be referred to the implementation of the file aggregation method of the driving path, and the repetition is not repeated. As used below, the term "unit" or "module" may be a combination of software and/or hardware that implements the intended function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
Fig. 6 is a block diagram of a file aggregation device for a driving path according to an embodiment of the present invention, and as shown in fig. 6, the device includes:
the encoding module 601 is configured to obtain a longitude and latitude of a geographic location where at least one image capturing device is located in an area to be monitored, and process the longitude and latitude of the geographic location where the image capturing device is located into a GeoHash code;
the region dividing module 602 is configured to divide a geographic location where the image acquisition device with the same prefix characters of a first preset number in the GeoHash code is located into a sub-region, so as to divide the region to be monitored into a plurality of sub-regions;
The node mapping module 603 is configured to map a computing node with the area to be monitored in the distributed system, where the computing node is called a central node, map a plurality of computing nodes with a plurality of sub-areas one by one, each computing node in the plurality of computing nodes is called an area node, and allocate a cache block for each of the area nodes in a same cache area according to a geographic position adjacent relation among the plurality of sub-areas, so that the cache blocks of the area nodes mapped by adjacent sub-areas are adjacent cache blocks;
a central node operation module 604, configured to obtain, by using the central node, each snapshot generated by the image capturing device in the area to be monitored, determine a corresponding sub-area according to a GeoHash code that generates a geographic location where the image capturing device of each snapshot is located, and distribute each snapshot to an area node mapped to the determined sub-area, where each snapshot includes a set of pictures generated by the same image capturing device for the same individual at a preset time;
the regional node operation module 605 is configured to obtain real-time files from a cache block and adjacent cache blocks of each regional node, compare each snapshot received by the regional node with the obtained real-time files to obtain a first comparison result, compare each snapshot with a history file according to the first comparison result to obtain a second comparison result, and store each snapshot in the cache block of the regional node as the real-time file when the second comparison result is that the comparison fails; when the second comparison result is that the comparison is successful, the identification information of the history file and each snapshot which are successfully compared are fed back to the central node;
The central node operation module 604 is further configured to combine, in the central node, each snapshot into the corresponding history file according to the identification information of the history file that is successfully compared.
In one embodiment, zone node run module 605 includes:
the first comparison unit is used for setting a plurality of parallel threads in each regional node to execute the process of comparing each snapshot received by the first comparison unit with the acquired real-time file; in each thread, aiming at each snapshot, sequencing the pictures according to the sequence from the large to the small of the individual characteristic quality of each picture to obtain a first sequence, and determining the pictures before the corresponding ranking of the second preset number in the first sequence as first key frames to obtain the second preset number of first key frames; for each obtained real-time file, sequencing the pictures according to the sequence from big to small of the individual characteristic quality of each picture to obtain a second sequence, determining the pictures before the second preset number is correspondingly ranked in the second sequence as second key frames, and determining the second preset number of the second key frames of each real-time file; calculating first similarity of individual features between each first key frame and each second key frame respectively to obtain a plurality of first similarity; sequencing the first similarities according to a sequence from large to small to obtain a third sequence, and determining the first similarities before the arrangement corresponding to a third preset number in the third sequence as key similarities to obtain the third preset number of key similarities; counting each real-time archive according to the number of the key similarity corresponding to each real-time archive; traversing each real-time file in sequence from large to small according to the counting, determining effective real-time files, and when the number of the effective real-time files is 0, determining that the first comparison result is failure in comparison, and not processing each snapshot; when the number of the effective real-time files is not 0, the first comparison result is that the comparison is successful, each effective real-time file and each snapshot are combined to form a new real-time file, and each effective real-time file is deleted from the corresponding cache block, wherein the effective real-time file is a real-time file which is not combined currently and is not held by other threads currently.
In one embodiment, the first comparing unit is further configured to determine, from the key similarities corresponding to each of the real-time files, a number of the key similarities for which the spatio-temporal information between the first key frame and the second key frame meets a first time threshold and meets a first distance threshold, and determine the number as a count of each of the real-time files.
In one embodiment, the first comparing unit is further configured to combine the effective real-time file of the first rank and each snapshot according to the order from the big count to the small count, form the current real-time file, sequentially traverse each other effective real-time files according to the order from the big count to the small count, and perform the following steps in a circulating manner: judging whether the track of the effective real-time file traversed currently and the track of the current real-time file meet space-time preset requirements or not; if not, ending the circulation, and determining the current real-time file as the new real-time file; if so, merging the effective real-time file currently traversed with the current real-time file, and regarding the formed file as a new current real-time file.
In one embodiment, the first comparing unit is further configured to mix and sort the pictures in the effective real-time archive currently traversed with the pictures in the current real-time archive according to the time stamp sequence of the pictures; for each picture in the current real-time file, when two pictures on the front and rear rows of the picture are pictures in the effective real-time file which is traversed currently, calculating the time difference between the time stamp of the picture and the time stamp of the two pictures on the front and rear rows and the distance between the picture and the affiliated image acquisition equipment; when two pictures on the front and rear rows of the picture and the picture with the same time stamp as the picture are pictures in the effective real-time file which is traversed currently, calculating the time difference between the time stamp of the picture and the time stamp of the two pictures on the front and rear rows and the distance between the picture and the image acquisition equipment to which the picture with the same time stamp belongs, and calculating the distance between the picture and the image acquisition equipment to which the picture with the same time stamp belongs; when one of the two pictures on the front and rear rows of the picture is a picture in the effective real-time archive currently traversed and the other picture is a picture in the current real-time archive, calculating the time difference between the picture and the time stamp of the one picture and the distance between the picture and the affiliated image acquisition equipment, and calculating the time difference between the time stamp of the other picture and the picture on the adjacent row of the picture and the distance between the picture and the affiliated image acquisition equipment; judging whether the distance corresponding to each picture in the current real-time file is smaller than a second distance threshold value and whether the time difference corresponding to each picture is smaller than a second time threshold value, if yes, judging that the track of the effective real-time file traversed currently and the track of the current real-time file meet space-time preset requirements; if not, judging that the track of the effective real-time file currently traversed and the track of the current real-time file do not meet the space-time preset requirement.
In one embodiment, zone node run module 605 includes:
the second comparison unit is used for sequencing the pictures according to the sequence from large to small of the individual characteristic quality scores of each picture aiming at each history file to obtain a fourth sequence, determining the pictures before the second preset number is correspondingly ranked in the fourth sequence as third key frames, and determining the second preset number of the third key frames of each history file; when the first comparison result is that the comparison fails, calculating second similarity of individual features between each first key frame and each third key frame of each history file to obtain a plurality of second similarity, and selecting a fourth preset number of second similarity before arrangement from the sequence of the second similarity from large to small to sum, wherein the sum is the similarity sum of each history file; when the maximum similarity sum is greater than a preset threshold, determining that the second comparison result is successful, determining the history file corresponding to the maximum similarity sum as the history file which is successfully compared, and feeding back identification information of the history file which is successfully compared and each snapshot to the central node; when the maximum similarity sum is smaller than the preset threshold value, determining that a second comparison result is a comparison failure, and storing each snapshot in a self cache block as a real-time file; or when the first comparison result is that the comparison is successful, sequencing the pictures in the new real-time archive according to the sequence from the large to the small of the individual characteristic quality of each picture to obtain a fifth sequence, and determining the pictures before the ranking corresponding to the second preset number in the fifth sequence as a fourth key frame; calculating third similarity of individual features between each fourth key frame and each third key frame of each history file to obtain a plurality of third similarity, and selecting third similarity before ranking corresponding to the fourth preset number from the sequence of the plurality of third similarity from large to small to sum, wherein the sum is the similarity sum of each history file; when the maximum similarity sum is larger than the preset threshold, determining that the second comparison result is successful, determining the history file corresponding to the maximum similarity sum as the history file which is successfully compared, and feeding back identification information of the history file which is successfully compared and the new real-time file to the central node; and when the maximum similarity sum is smaller than the preset threshold value, determining that the second comparison result is a comparison failure, and storing the new real-time file into a self cache block as a real-time file.
In one embodiment, the apparatus further comprises:
the node restarting module is used for putting each snapshot distributed to each regional node into a queue through the central node, wherein each snapshot is provided with a unique ID; when each snapshot is distributed to the regional node mapped with the determined subarea through the central node, receiving the ID of the earliest received snapshot fed back by the regional node; when the inactivated area node exists in the plurality of area nodes, screening the ID of each snapshot before the received ID and the ID of each snapshot cached in a cache block of the inactivated area node from the IDs of each snapshot in the queue through the central node, and determining each snapshot corresponding to the rest IDs as an unprocessed snapshot of the inactivated area node; restarting a new regional node through the central node, and establishing a binding relation between the new regional node and a cache block of the inactivated regional node; and redistributing the unprocessed snapshots of the inactivated regional nodes to the regional nodes mapped with the determined subareas according to the sequence of the time stamps through the central node.
The embodiment of the invention realizes the following technical effects: the method has the advantages that the capture generated by the image acquisition equipment is subjected to gear gathering processing by adopting a plurality of regional nodes of the distributed system in parallel, the gear gathering capability of the capture with high concurrence is improved, the capture gear gathering under the condition of ten-thousand cameras can be supported, and the efficiency of ten-thousand gear gathering is improved; meanwhile, a plurality of regional nodes perform file aggregation processing on the snap shots generated by the image acquisition equipment in parallel, so that the defect that file aggregation cannot be recovered due to single-node faults can be avoided, and even if a certain regional node fault occurs, other regional nodes can still perform file aggregation processing normally, thereby being beneficial to ensuring smooth completion of file aggregation; in addition, the subareas are obtained based on the GeoHash codes of the geographic positions of the image acquisition devices, the number of the subareas can be flexibly and real-timely adjusted based on different requirements according to the adjustment of the same prefix character number of the GeoHash codes, the number of the image acquisition devices can be flexibly increased or reduced in each subarea, the subareas can be newly increased according to the geographic positions of the newly increased image acquisition devices, the number of the area nodes mapped with the subareas one by one can be flexibly and real-timely adjusted, and the expansibility and the high availability are improved.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may alternatively be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than what is shown or described, or they may be separately fabricated into individual integrated circuit modules, or a plurality of modules or steps in them may be fabricated into a single integrated circuit module. Thus, embodiments of the invention are not limited to any specific combination of hardware and software.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, and various modifications and variations can be made to the embodiments of the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A method for aggregating files of a travel path, comprising:
acquiring the longitude and latitude of the geographic position of at least one image acquisition device in the region to be monitored, and processing the longitude and latitude of the geographic position of the image acquisition device into a GeoHash code;
dividing the geographic position of the image acquisition equipment with the same first preset number of prefix characters in the GeoHash code into a sub-area so as to divide the area to be monitored into a plurality of sub-areas;
in the distributed system, mapping a computing node with the area to be monitored, wherein the computing node is called a central node, mapping a plurality of computing nodes with a plurality of subareas one by one, each computing node in the plurality of computing nodes is called an area node, and distributing a cache block for each area node in the same cache region according to the geographic position adjacent relation among the plurality of subareas, so that the cache blocks of the area nodes mapped by adjacent subareas are adjacent cache blocks;
acquiring each snapshot generated by the image acquisition equipment in the area to be monitored through the central node, determining a corresponding subarea according to a GeoHash code of the geographic position of the image acquisition equipment for generating each snapshot, and distributing each snapshot to an area node mapped with the determined subarea, wherein each snapshot comprises a set of pictures generated by the same image acquisition equipment for the same individual at preset time;
In each regional node, acquiring real-time files from a self cache block and an adjacent cache block, comparing each snapshot received by the regional node with the acquired real-time files to obtain a first comparison result, comparing each snapshot with the history files according to the first comparison result to obtain a second comparison result, and storing each snapshot in the self cache block as the real-time files when the second comparison result is that the comparison fails; when the second comparison result is that the comparison is successful, the identification information of the history file and each snapshot which are successfully compared are fed back to the central node;
and merging each snapshot into the corresponding history file according to the identification information of the history file successfully compared in the central node.
2. The method for aggregating files of a traveling path according to claim 1, wherein in each of the regional nodes, a real-time file is acquired from a cache block of the regional node and an adjacent cache block, and each snapshot received by the regional node is compared with the acquired real-time file to obtain a first comparison result, comprising:
setting a plurality of parallel threads in each regional node to execute a process of comparing each snapshot received by the regional node with the acquired real-time file;
In each thread, aiming at each snapshot, sequencing the pictures according to the sequence from the large to the small of the individual characteristic quality of each picture to obtain a first sequence, and determining the pictures before the corresponding ranking of the second preset number in the first sequence as first key frames to obtain the second preset number of first key frames;
for each obtained real-time file, sequencing the pictures according to the sequence from big to small of the individual characteristic quality of each picture to obtain a second sequence, determining the pictures before the second preset number is correspondingly ranked in the second sequence as second key frames, and determining the second preset number of the second key frames of each real-time file;
calculating first similarity of individual features between each first key frame and each second key frame respectively to obtain a plurality of first similarity;
sequencing the first similarities according to a sequence from large to small to obtain a third sequence, and determining the first similarities before the arrangement corresponding to a third preset number in the third sequence as key similarities to obtain the third preset number of key similarities;
Counting each real-time archive according to the number of the key similarity corresponding to each real-time archive;
traversing each real-time file in sequence from large to small according to the counting, determining effective real-time files, and when the number of the effective real-time files is 0, determining that the first comparison result is failure in comparison, and not processing each snapshot; when the number of the effective real-time files is not 0, the first comparison result is that the comparison is successful, each effective real-time file and each snapshot are combined to form a new real-time file, and each effective real-time file is deleted from the corresponding cache block, wherein the effective real-time file is a real-time file which is not currently combined and is not currently held by other threads.
3. The profile aggregation method of travel paths according to claim 2, wherein counting each of the real-time profiles according to the number of key similarities corresponding to each of the real-time profiles comprises:
and determining the number of the key similarity, which is in accordance with a first time threshold and a first distance threshold, of the space-time information between the first key frame and the second key frame from the key similarity corresponding to each real-time archive, and determining the number as the count of each real-time archive.
4. The method of claim 2, wherein combining each of the effective real-time profile and each snapshot to form a new real-time profile comprises:
merging the effective real-time files of the first rank and each snapshot according to the sequence from big to small in count to form a current real-time file, traversing each other effective real-time file in sequence according to the sequence from big to small in count, and executing the following steps in a circulating way:
judging whether the track of the effective real-time file traversed currently and the track of the current real-time file meet space-time preset requirements or not;
if not, ending the circulation, and determining the current real-time file as the new real-time file; if so, merging the effective real-time file currently traversed with the current real-time file, and regarding the formed file as a new current real-time file.
5. The profile aggregation method of travel paths according to claim 2, wherein determining whether the track of the effective real-time profile currently traversed and the track of the current real-time profile satisfy a space-time preset requirement comprises:
according to the time stamp sequence of the pictures, mixing and sorting the pictures in the effective real-time file traversed currently and the pictures in the current real-time file;
For each picture in the current real-time file, when two pictures on the front and rear rows of the picture are pictures in the effective real-time file which is traversed currently, calculating the time difference between the time stamp of the picture and the time stamp of the two pictures on the front and rear rows and the distance between the picture and the affiliated image acquisition equipment; when two pictures on the front and rear rows of the picture and the picture with the same time stamp as the picture are pictures in the effective real-time file which is traversed currently, calculating the time difference between the time stamp of the picture and the time stamp of the two pictures on the front and rear rows and the distance between the picture and the image acquisition equipment to which the picture with the same time stamp belongs, and calculating the distance between the picture and the image acquisition equipment to which the picture with the same time stamp belongs; when one of the two pictures on the front and rear rows of the picture is a picture in the effective real-time archive currently traversed and the other picture is a picture in the current real-time archive, calculating the time difference between the picture and the time stamp of the one picture and the distance between the picture and the affiliated image acquisition equipment, and calculating the time difference between the time stamp of the other picture and the picture on the adjacent row of the picture and the distance between the picture and the affiliated image acquisition equipment;
Judging whether the distance corresponding to each picture in the current real-time file is smaller than a second distance threshold value and whether the time difference corresponding to each picture is smaller than a second time threshold value, if yes, judging that the track of the effective real-time file traversed currently and the track of the current real-time file meet space-time preset requirements; if not, judging that the track of the effective real-time file currently traversed and the track of the current real-time file do not meet the space-time preset requirement.
6. The method for aggregating files of a traveling path according to claim 2, wherein each snapshot is compared with a history file according to the first comparison result to obtain a second comparison result, and when the second comparison result is that the comparison fails, each snapshot is stored in a self cache block as a real-time file; and when the second comparison result is that the comparison is successful, feeding back the identification information of the history file and each snapshot which are successfully compared to the central node, wherein the method comprises the following steps:
for each history file, sequencing the pictures according to the sequence from big to small of individual characteristic quality of each picture to obtain a fourth sequence, determining the pictures before the second preset number is ranked corresponding to the second preset number in the fourth sequence as third key frames, and determining the second preset number of the third key frames of each history file;
When the first comparison result is that the comparison fails, calculating second similarity of individual features between each first key frame and each third key frame of each history file to obtain a plurality of second similarity, and selecting a fourth preset number of second similarity before arrangement from the sequence of the second similarity from large to small to sum, wherein the sum is the similarity sum of each history file;
when the maximum similarity sum is greater than a preset threshold, determining that the second comparison result is successful, determining the history file corresponding to the maximum similarity sum as the history file which is successfully compared, and feeding back identification information of the history file which is successfully compared and each snapshot to the central node; when the maximum similarity sum is smaller than the preset threshold value, determining that the second comparison result is a comparison failure, and storing each snapshot in a self cache block as a real-time file; or,
when the first comparison result is that the comparison is successful, sequencing the pictures in the new real-time file according to the sequence from big to small of the individual characteristic quality scores of each picture to obtain a fifth sequence, and determining the pictures before the second preset number is correspondingly ranked in the fifth sequence as a fourth key frame;
Calculating third similarity of individual features between each fourth key frame and each third key frame of each history file to obtain a plurality of third similarity, and selecting third similarity before ranking corresponding to the fourth preset number from the sequence of the plurality of third similarity from large to small to sum, wherein the sum is the similarity sum of each history file;
when the maximum similarity sum is greater than the preset threshold value, determining that the second comparison result is successful in comparison, determining the history file corresponding to the maximum similarity sum as the history file which is successful in comparison, and feeding back identification information of the history file which is successful in comparison and the new real-time file to the central node; and when the maximum similarity sum is smaller than the preset threshold value, determining that the second comparison result is failed in comparison, and storing the new real-time file as a real-time file in a self cache block.
7. The archive aggregation method of a travel path according to any one of claims 2 to 6, further comprising:
placing each snapshot distributed to each regional node into a queue through the central node, wherein each snapshot is provided with a unique ID;
When each snapshot is distributed to the regional node mapped with the determined subarea through the central node, receiving the ID of the earliest received snapshot fed back by the regional node;
when the inactivated area node exists in the plurality of area nodes, screening the ID of each snapshot before the received ID and the ID of each snapshot cached in a cache block of the inactivated area node from the IDs of each snapshot in the queue through the central node, and determining each snapshot corresponding to the rest IDs as an unprocessed snapshot of the inactivated area node;
restarting a new regional node through the central node, and establishing a binding relation between the new regional node and a cache block of the inactivated regional node;
and redistributing the unprocessed snapshots of the inactivated regional nodes to the regional nodes mapped with the determined subareas according to the sequence of the time stamps through the central node.
8. A profile aggregation device for a travel path, comprising:
the coding module is used for acquiring the longitude and latitude of the geographic position of at least one image acquisition device in the area to be monitored and processing the longitude and latitude of the geographic position of the image acquisition device into a GeoHash code;
The region dividing module is used for dividing the geographic position of the image acquisition equipment with the same first preset number of prefix characters in the GeoHash code into a sub-region so as to divide the region to be monitored into a plurality of sub-regions;
the node mapping module is used for mapping one computing node with the area to be monitored in the distributed system, the computing node is called a central node, a plurality of computing nodes and a plurality of subareas are mapped one by one, each computing node in the plurality of computing nodes is called an area node, and a cache block is distributed for each area node in the same cache region according to the geographic position adjacent relation among the plurality of subareas, so that the cache blocks of the area nodes mapped by adjacent subareas are adjacent cache blocks;
the central node operation module is used for acquiring each snapshot generated by the image acquisition equipment in the area to be monitored through the central node, determining a corresponding subarea according to the GeoHash code of the geographic position of the image acquisition equipment for generating each snapshot, and distributing each snapshot to the area node mapped with the determined subarea, wherein each snapshot comprises a set of pictures generated by the same image acquisition equipment for the same individual at preset time;
The regional node operation module is used for acquiring real-time files from the self cache block and the adjacent cache blocks in each regional node, comparing each snapshot received by the regional node with the acquired real-time files to obtain a first comparison result, comparing each snapshot with the history files according to the first comparison result to obtain a second comparison result, and storing each snapshot into the self cache block as the real-time file when the second comparison result is that the comparison fails; when the second comparison result is that the comparison is successful, the identification information of the history file and each snapshot which are successfully compared are fed back to the central node;
and the central node operation module is further used for merging each snapshot into the corresponding history file according to the identification information of the history file successfully compared in the central node.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the archive aggregation method of the travel path of any one of claims 1 to 7 when the computer program is executed.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program that executes the archive aggregation method of the travel path according to any one of claims 1 to 7.
CN202310833922.XA 2023-07-07 2023-07-07 File aggregation method and device for driving path, computer equipment and storage medium Active CN117056551B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310833922.XA CN117056551B (en) 2023-07-07 2023-07-07 File aggregation method and device for driving path, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310833922.XA CN117056551B (en) 2023-07-07 2023-07-07 File aggregation method and device for driving path, computer equipment and storage medium

Publications (2)

Publication Number Publication Date
CN117056551A true CN117056551A (en) 2023-11-14
CN117056551B CN117056551B (en) 2024-04-02

Family

ID=88668151

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310833922.XA Active CN117056551B (en) 2023-07-07 2023-07-07 File aggregation method and device for driving path, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN117056551B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117453942A (en) * 2023-12-21 2024-01-26 北京瑞莱智慧科技有限公司 File aggregation method, device, computer equipment and medium for driving path

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200379955A1 (en) * 2019-06-03 2020-12-03 EMC IP Holding Company LLC Incremental metadata aggregation for a file storage system
US20210173811A1 (en) * 2019-12-04 2021-06-10 Commvault Systems, Inc. Optimizing the restoration of deduplicated data stored in multi-node replicated file systems
CN114329058A (en) * 2021-12-29 2022-04-12 重庆紫光华山智安科技有限公司 Image gathering method and device and electronic equipment
CN114743050A (en) * 2022-04-08 2022-07-12 浙江大华技术股份有限公司 Driver portrait pre-classification method, image classification device and storage medium
CN115495606A (en) * 2022-08-11 2022-12-20 厦门市美亚柏科信息股份有限公司 Image gathering method and system
CN115687249A (en) * 2022-12-30 2023-02-03 浙江大华技术股份有限公司 Image gathering method and device, terminal and computer readable storage medium
CN115880754A (en) * 2022-12-07 2023-03-31 浙江大华技术股份有限公司 Multi-gear combination method and device and electronic equipment
CN116168222A (en) * 2023-03-15 2023-05-26 浙江大华技术股份有限公司 Track recognition method and device, storage medium and electronic equipment

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200379955A1 (en) * 2019-06-03 2020-12-03 EMC IP Holding Company LLC Incremental metadata aggregation for a file storage system
US20210173811A1 (en) * 2019-12-04 2021-06-10 Commvault Systems, Inc. Optimizing the restoration of deduplicated data stored in multi-node replicated file systems
CN114329058A (en) * 2021-12-29 2022-04-12 重庆紫光华山智安科技有限公司 Image gathering method and device and electronic equipment
CN114743050A (en) * 2022-04-08 2022-07-12 浙江大华技术股份有限公司 Driver portrait pre-classification method, image classification device and storage medium
CN115495606A (en) * 2022-08-11 2022-12-20 厦门市美亚柏科信息股份有限公司 Image gathering method and system
CN115880754A (en) * 2022-12-07 2023-03-31 浙江大华技术股份有限公司 Multi-gear combination method and device and electronic equipment
CN115687249A (en) * 2022-12-30 2023-02-03 浙江大华技术股份有限公司 Image gathering method and device, terminal and computer readable storage medium
CN116168222A (en) * 2023-03-15 2023-05-26 浙江大华技术股份有限公司 Track recognition method and device, storage medium and electronic equipment

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
YIFAN CHEN ET AL.: "Prefetch and Cache Replacement Based on Thompson Sampling for Satellite IoT Network", 《ICC 2021 - IEEE INTERNATIONAL CONFERENCE ON COMMUNICATIONS》, 23 June 2021 (2021-06-23), pages 1 - 6, XP033953669, DOI: 10.1109/ICC42927.2021.9500508 *
叶隽毅 等: "探析公共安全视图智能分析系统的数据血缘", 《中国安防》, no. 06, 1 June 2020 (2020-06-01), pages 110 - 113 *
李杰 等: "大数据分析技术助力公安典型应用场景精细化治理", 《警察技术》, no. 04, 7 July 2020 (2020-07-07), pages 73 - 76 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117453942A (en) * 2023-12-21 2024-01-26 北京瑞莱智慧科技有限公司 File aggregation method, device, computer equipment and medium for driving path
CN117453942B (en) * 2023-12-21 2024-03-19 北京瑞莱智慧科技有限公司 File aggregation method, device, computer equipment and medium for driving path

Also Published As

Publication number Publication date
CN117056551B (en) 2024-04-02

Similar Documents

Publication Publication Date Title
US10838777B2 (en) Distributed resource allocation method, allocation node, and access node
CN117056551B (en) File aggregation method and device for driving path, computer equipment and storage medium
WO2020147331A1 (en) Micro-service monitoring method and system
CN108985162A (en) Object real-time tracking method, apparatus, computer equipment and storage medium
EP2966835A1 (en) Cooperative caching method and apparatus
DE112010003453T5 (en) Method, unit and system for searching for virtual resources
CN106936622A (en) A kind of distributed memory system upgrade method and device
CN107092686B (en) File management method and device based on cloud storage platform
CN111291062B (en) Data synchronous writing method and device, computer equipment and storage medium
US8645555B2 (en) System and method for stateless, fault tolerance and load balanced data collection using overlay namespaces
CN101533416A (en) Method and system for managing video and audio file
CN113055461B (en) ZooKeeper-based unmanned cluster distributed cooperative command control method
CN114564302B (en) GPU resource allocation method, system, equipment and medium
CN109756429B (en) Bandwidth allocation method and device
CN103973791A (en) Rapid uniform storage system of internet of things
CN109063121A (en) Date storage method, device, computer equipment and computer storage medium
CN116107828A (en) Main node selection method, distributed database and storage medium
CN108174147B (en) Automatic optimization allocation method for video source and tracking server in video monitoring system
CN107122136B (en) Capacity obtaining method and device
CN117061512A (en) Distributed information management method and system based on big data
KR102390976B1 (en) Method and system for receiving scheduling of satellite image
CN112887224A (en) Traffic scheduling processing method and device, electronic equipment and storage medium
CN104102557A (en) Cloud computing platform data backup method based on clustering
CN111913927A (en) Data writing method and device and computer equipment
CN114493602B (en) Block chain transaction execution method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant