CN102833580B - Based on HD video application system and the method for infiniband - Google Patents
Based on HD video application system and the method for infiniband Download PDFInfo
- Publication number
- CN102833580B CN102833580B CN201210348249.2A CN201210348249A CN102833580B CN 102833580 B CN102833580 B CN 102833580B CN 201210348249 A CN201210348249 A CN 201210348249A CN 102833580 B CN102833580 B CN 102833580B
- Authority
- CN
- China
- Prior art keywords
- video
- file
- storage
- infiniband
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
<b/><b/><b/> the invention provides a kind of HD video application system based on infiniband, comprise monitoring business portion, HD video storage part and video identification analysis portion, wherein monitoring business portion is for obtaining HD video data and assigning the order transferred HD video video recording and carry out video identification processing operation; HD video storage part connects monitoring business portion, and its medium cloud bridge middleware is by infiniband network and storage server communication; Video identification analysis portion connects described HD video storage part by infiniband network, and comprise management server node, computing cluster and the LUSTRE storage cluster based on infiniband, the described LUSTRE storage cluster based on infiniband is object-based distributed file system, is responsible for a large amount of intermediate data storing the generation of described computing cluster.Infiniband network application in storage environment and parallel processing environment, is suitable for the application of high-definition video monitoring mass memory and complicated recognition technology by the scheme that the present invention proposes.
Description
Technical field
The present invention relates to video field, in particular, relate to a kind of HD video application system based on infiniband and HD video recognition methods.
Background technology
Traditional monitoring mode can not meet the industries such as government " safe city ", financial sector, higher education, supervision, prison, Wen Bo to the demand of security protection, have networking, intellectuality, digitlization, long-range feature network video monitor and control system then become new application trend, and form efficient, safe, an advanced Network Video Surveillance system.
In network video monitor and control system, H.264 coding compression algorithm is successfully promoted, and along with progressively accreditation, the reduction of price and the perfect of function of user, must welcome the epoch that high-definition network camera dominates future video monitoring field.Current statistics shows, high-resolution image-quality video stores more valuable, and especially in the supervisory control system of public security type, finally play a role 90% is all record a video by the playback of high definition.And based on the high-definition network camera of standard H.264 compress technique, the code stream of the full realtime graphic of one road 720P high definition is 3Mbps, the memory space that record a video needs for one hour is approximately about 1.35G (if 1080P, video record is usually pressed 5Mbps and calculated, one hour needs 2.25G).This shows that high-definition network camera needs the storage of greater room.
Video file belongs to unstructured data, has stronger dependence to the network bandwidth.If 200 road 720P mega pixel full real-time high-definition video camera (taking 3Mbps by the video recording of every road to calculate) is installed in front end, take total bandwidth resource 60% ~ 80%, so kilomega network can effectively be supported completely.If change 1080P into, video recording bandwidth press 5M or even high standard 6M every road when calculating, and whole transmission network bandwidth design just almost needs to double.
In sum, from hardware net camera interface technology, the H.264 aspect such as coding techniques and cost, kilomega network meets front end applications environment completely.But in background process, kilomega network environment is just unable to do what one wishes.Background process is most typical is exactly the application of concentrating storage and video processing technique.
In concentrated storage, storage system is convergent point, all high-definition cameras by while information be stored in system, very large pressure is caused to the whole network bandwidth.In large-scale deployment, ten thousand mbit ethernets also seem unable to do what one wishes, and infiniband technology single port starting is at present exactly 40,000 million and nanosecond time delay, and comparing Ethernet has lower transmission delay and provide higher bandwidth ability.From storage protocol, the IPSAN framework that Ethernet adopts have employed ISCSI agreement, and only have 40%-60% in kilomega network environment actual transmissions efficiency, more cpu resource participates in unpacking with in recombination of TCP/IP.Although current 10Gb Ethernet releases TOE(TCP/IPOffloadEngine, TCP/IP load offload engine) hardware-accelerated design, its limited relieving capacity and considerable price are larger obstacles.And be SRP(SCSIRDMAProtocol based on the storage protocol of infiniband technology, SCSI remote direct memory access protocal), SRP agreement is packed by scsi command in InfiniBand, allow scsi command by RDMA (Remotedirectmemoryaccess, remote direct memory is accessed) communicate between different systems, realize memory device and share and RDMA communication service.RDMA technology makes CPU process communication bag just account for 3% of whole system resource, and the resource that major part is valuable gives user job, and 97% flow communication is realized to remote memory access by local internal memory.Infiniband is by IPoIB technology in addition, realizes ICP/IP protocol at infiniband network design, make much based on the application of TCP/IP to enjoy without the need to changing code infiniband Netowrk tape to the high-quality service of high bandwidth low delay.
In Video processing, video identification technology is very important link.At the train station, airport, subway, bus station, the crowd is dense the place such as exhibition, grasping crowd's quantity, density etc. at any time, to make correct decisions to departments such as commander, safety checks significant in good time.Rely on people to stare at the change of display monitoring surveyed area completely, due to reasons such as the inevasible fatigue of monitor staff, carelessness, the as easy as rolling off a log suspicious figure of allowing passes through.Adopt the direction that image processing method realizes intellectualized detection, analysis is safe examination system intelligent development from now on.
Video identification technology can be divided into real-time analysis and ex-post analysis, consider the feature of video data, no matter be real-time analysis or ex-post analysis, its algorithm is all complicated, and efficient parallel algorithm often, it produces a large amount of intermediate data with a large amount of calculating, and these intermediate data need multiple node read simultaneously and write.This is typical computing cluster environmental characteristic, and the network environment that parallel algorithm is wished is high speed, stable, reliable and low delay.Infiniband is applicable to the cluster environment of such computation-intensive completely.
Although infiniband technology is disposed comparatively extensive in high-performance computing sector and data center, the outstanding ability of its high bandwidth and low delay have also been obtained extensive accreditation, but how itself and current video monitoring is stored and discriminance analysis combine with technique is blank always.
Summary of the invention
In order to overcome the problems referred to above, the present invention proposes a kind of HD video application system based on infiniband, comprise monitoring business portion, HD video storage part and video identification analysis portion, wherein monitoring business portion, for obtaining HD video data and assigning the order transferred HD video video recording and carry out video identification processing operation; HD video storage part connects described monitoring business portion, described HD video storage part comprises cloud bridge middleware cluster and storage server cluster, and adopting IBNAS framework, described cloud bridge middleware cluster passes through infiniband network based on remote direct memory access protocal and described storage server trunking communication; Video identification analysis portion connects described HD video storage part by infiniband network, described video identification analysis portion also comprises management server node, computing cluster and the LUSTRE storage cluster based on infiniband, described computing cluster is responsible for carrying out identifying processing by submitting to video identification parallel parsing algorithm after high-definition video file decoding, the described LUSTRE storage cluster based on infiniband is object-based distributed file system, is responsible for a large amount of intermediate data storing the generation of described computing cluster.
Further, described monitoring business portion comprises high-definition monitoring image unit and service maintenance unit, wherein high-definition monitoring image unit is responsible for obtaining HD video data, and based on ICP/IP protocol, obtained HD video data acquisition gigabit Ethernet is converged to cloud bridge middleware cluster by IP operation private network; Described service maintenance unit primary responsibility assigns the order transferred HD video video recording and carry out video identification processing operation, by IP, it safeguards that private network converges to described cloud bridge middleware cluster.
Further, described HD video storage part has back-up system, described back-up system comprises the storage system of cloud bridge middleware for subsequent use and connection thereof, an infiniband synchronization link is had to realize the data syn-chronization mirror image ability between active system and standby system between primary and backup cloud bridge middleware, when the cloud bridge middleware in active system or its storage system connected break down, replacement active system and described monitoring business portion and video identification analysis portion carry out alternately by standby system.
Further, the described LUSTRE storage cluster based on infiniband comprises meta data server and object storage server, described management server node is the interface of described service maintenance unit submit job job invocation, and video identification job task submits to described computing cluster node by described management server node through infiniband network; Described meta data server is responsible for the metadata providing whole file system to described computing cluster node, the view of structure, management document distribution, allow client-access object, manage the NameSpace of whole file system, safeguard the bibliographic structure of whole file system, user right, and be responsible for the data consistency of maintaining file system; The storage of real data is responsible for by described object storage server, and what process between all computing cluster nodes and physical store is mutual.
Further, configuration two meta data servers in described LUSTRE storage cluster, one of them is as backup, and two meta data servers adopt the mode sharing storage to deposit metadata.
The invention allows for a kind of HD video recognition methods based on infiniband, comprising:
Step 1: by IP, service maintenance unit safeguards that private network issues job instruction;
Step 2: after management server node receives described job instruction, is converted to the initial conditions of parallelism recognition program by described job instruction, by infiniband net distribution to computing cluster node calculate;
Step 3: described computing cluster node is searched for according to described initial conditions and located video file required on storage server, then extract this video file and decode and obtain frame of video, each frame of video is incorporated as the initial conditions of new parallelism recognition program, is again calculated in described computing cluster node by video identification parallel parsing algorithm;
Step 4: a large amount of intermediate object programs that described video identification parallel parsing algorithm produces are buffered in LUSTRE storage cluster, when first establishment file, meta data server in described computing cluster node visit LUSTRE storage cluster, described meta data server definition and log file leave the positional information of object storage server in, when first access file, meta data server in described computing cluster node visit LUSTRE storage cluster, the object stores service positional information at locating file place;
Step 5, during first establishment file, the object storage server that described meta data server sets up data file by instruction notification in object storage server cluster creates data file, and described object storage server has created rear feedback target meta data server;
Step 6, described computing cluster node, after obtaining depositing the positional information of intermediate data file, accesses the intermediate data in described object storage server cluster in a parallel fashion;
Step 7, after described computing cluster node has run described video identification parallel parsing algorithm, has submitted to described management server node by end product;
Step 8, described management server node submits to client by after the finish message of described end product.
Further, the described instruction in step 1 comprises the scope target that can search for.
Further, the described end product in step 6 comprises and identifies the associated video files information successfully or identify that unsuccessful reason is fed back.
The invention allows for a kind of access method of the HD video based on infiniband network, comprise: when preserving video file, the video file of generation to be saved to the virtual folder in cloud bridge middleware by the high-definition monitoring image unit in monitoring business portion by kilomega network, virtual folder is mapped as the file of NFS NFS4 identification by the virtualization modules of cloud bridge middleware, is saved in the space of storage server distribution by infiniband network and remote direct memory access technique by video file; When transferring video file, the service maintenance unit in described monitoring business portion directly transfers video file by kilomega network in the virtual folder of cloud bridge middleware, virtual folder is mapped as the file of NFS NFS4 identification by the virtualization modules of cloud bridge middleware, and the video record file transferred by actual needs by infiniband network and remote direct memory access technique is taken out from the physical space that storage server is deposited.
Also the problem that intermediate data deposits can be solved well based on the NFS NFS4 of infiniband and parallel file system LUSTRE.Video identification technology is when possessing good parallel algorithm ability, and collocation adopts the computing cluster system of infiniband, and its operating efficiency will improve greatly.
The scheme that the present invention proposes, utilizes very at a high speed and reliable infiniband network storage environment and parallel processing environment, is very applicable to high-definition video monitoring mass memory and complicated recognition technology is applied.
Accompanying drawing explanation
Fig. 1 is the system block diagram of the HD video application system based on infiniband.
Fig. 2 is the Organization Chart of HD video storage part.
Fig. 3 is the HD video storage part with back-up system.
Fig. 4 (a) and (b) are respectively the cloud bridge middleware being in Mobile Forms shown in Fig. 3 or storage system when occurring delaying machine, the switching schematic diagram of back-up system.
Fig. 5 is the structure chart of LUSTRE parallel file system.
Fig. 6 is the functional block diagram of HD video identification part.
Fig. 7 is the access flow chart of HD video.
Fig. 8 is the video identification flow chart of HD video.
Embodiment
Be described principle of the present invention and feature below in conjunction with accompanying drawing, example, only for explaining the present invention, is not intended to limit scope of the present invention.
The present invention proposes a kind of HD video application system based on infiniband, as shown in Figure 1.Described HD video application system comprises monitoring business portion, HD video storage part and video identification analysis portion.
Monitoring business portion comprises high-definition monitoring image unit and service maintenance unit, wherein high-definition monitoring image unit is responsible for obtaining HD video data, and mainly based on the TCP/IP technology of main flow, the data acquisition gigabit Ethernet that high-definition monitoring image unit obtains is converged to cloud bridge middleware cluster by IP operation private network.The order of video identification processing operation is transferred and is assigned in the video recording of service maintenance unit primary responsibility HD video, by IP, the client in service maintenance unit safeguards that private network (100,000,000 or gigabit Ethernet) converges to cloud bridge middleware cluster.
What HD video storage part adopted is IBSAN framework, and cloud bridge middleware cluster is by infiniband network and storage server trunking communication.Infiniband has the feature of high bandwidth (single port 40Gbps) and low delay (end-to-end 300 nanoseconds time delay) as transport vehicle.In IBNAS, adopt NFS4 file system can realize based on the communication of RDMA technology between cloud bridge middleware cluster and storage server cluster, its medium cloud bridge middleware cluster is as NFS4 client, and storage server cluster is as NFS4 server.The process participating in communication traffic due to CPU only accounts for 3%, and it not only releases CPU greatly, and between memory-to-memory at a high speed, reading and write operation make I/O transmission real without bottleneck under based on the infiniband network architecture.
Fig. 2 shows the Organization Chart of HD video storage part.High-definition monitoring image unit does bearer network with gigabit Ethernet, is placed on the virtual Shared Folders of cloud bridge middleware by the video file produced by ICP/IP protocol.Cloud bridge middleware adopts NFS4 agreement, through infiniband network, with carry file form, back end storage system is mapped the disk come to organize, then be the virtual Shared Folders arrived that front end high definition monitoring camera unit is seen by Storage Virtualization module converts.In whole storage area, employing be NFS NFS4 based on infiniband, it not only inherits all advantages of NFS file system, the more important thing is and supports infiniband network, support the technology of RDMA simultaneously.
Cloud bridge middleware cluster provides Shared Folders access stencil towards monitoring business portion, simultaneously cloud bridge middleware cluster as the client of IBNAS by the technology access storage server of RDMA.
Storage server cluster, as NFS4 file server, provides shared memory space by infiniband network at a high speed.Storage server cluster supports mass storage capacity, because the network bandwidth adopts the infiniband of 40Gb, not only can support SSD high speed solid hard disk, also supports Large Copacity 3.5 cun of mechanical hard disks of enterprise-level or civil SAS or SATA simultaneously.
Video identification analysis portion mainly comprises management server node, computing cluster and LUSTRE storage cluster.Described management server node is the interface of described service maintenance unit submit job job invocation, and video identification job task submits to described computing cluster node by described management server node through infiniband network.The task of computing cluster submits to video identification routine analyzer after high-definition video file being decoded to carry out identifying processing.Video identification routine analyzer algorithmically often adopts Parallel Design thinking, and algorithm complexity is comparatively large, and has higher concurrency, and when general image, Video processing, view data copy can consume a large amount of cpu resources; A large amount of intermediate object program can be produced when calculating in addition, share to all computing nodes.
LUSTRE storage cluster is object-based distributed file system, and its main task is a large amount of intermediate data storing computing cluster generation.LUSTRE is optimized for large files (as video file) read-write, LUSTRE file system relies on transmission network bandwidth very much, and the low delay of infiniband and high bandwidth feature provide good network support for LUSTRE file system, improve the communication speed of computing node, thus improve whole file system performance.In addition, the LUSTRE file system based on infiniband also supports RDMA technology, makes it on Internet Transmission without any bottleneck.
Fig. 3 shows the HD video storage part with backup, thus cloud bridge middleware and storage server also can realize the failover capability of high speed infiniband network environment.There is an infiniband synchronization link between active and standby cloud bridge middleware, mainly realize the data syn-chronization mirror image ability between active and standby cloud bridge middleware and the storage system connected separately.Storage system that primary cloud bridge middleware is connected with it forms Mobile Forms, directly and front end high definition monitoring camera unit mutual.It is dynamic that cloud bridge middleware for subsequent use and its storage system connected form non-live, realized and active system synchronous mirror by infiniband synchronization link.
When the primary cloud bridge middleware being in Mobile Forms shown in Fig. 3 occurs delaying machine, the storage system of the dynamic cloud bridge middleware for subsequent use of non-live and connection thereof will become Mobile Forms, carry out alternately, as shown in Figure 4 (a) with front end high definition monitoring camera unit.When the primary storage system being in Mobile Forms shown in Fig. 3 occurs delaying machine, the storage system of the dynamic cloud bridge middleware for subsequent use of non-live and connection thereof will become Mobile Forms, carry out alternately, as shown in Figure 4 (b) with front end high definition monitoring camera unit.
Before the structure that HD video identification part is described, first LUSTRE parallel file system is described.LUSTRE file system adopts object-based storage technology, based on the storage scheme that distributed lock management mechanism and metadata and storage data are separated, provide the NameSpace of the overall situation, eliminate the data multiple duplication of traditional distributed file system, be convenient to the work that group system is acted in agreement.LUSTRE optimizes for the read-write of large files, the independence of high performance data throughput, global data shared environment, data storage location can be provided for group system, node failure is provided redundancy scheme and cluster reset or server and network failure time quick Resume service, meet the needs of mass memory preferably.
LUSTRE parallel file system is primarily of three part compositions: client (client, i.e. computing cluster node), object storage server (OSS, and meta data server (MDS ObjectStorageSever), MetaDataServer), as shown in Figure 5.Client provides the access to file system by the POSIX interface (PortableOperatingSystemInterface, portable operating system interface) of standard to user.For client, LUSTRE is a transparent file system, and he is without the need to knowing the position at concrete data place, can data pellucidly in travel all over file system.Client carries out the mutual of file data with object storage server OSS, comprises the read-write of file data, the change etc. of object properties; Client carries out the mutual of metadata with meta data server MDS, comprises directory management, NameSpace management etc.
Object storage server OSS is responsible for the storage of real data, and what process between all clients and physical store is mutual.The storage adopted in object storage server OSS is based on object (ObjectBased), and all object datas are put on physical storage device by object storage server OSS, and complete the management to each object.Realized alternately by device driver between object storage server OSS and the physical storage device of reality.By the effect of driver, LUSTRE can inherit new physical store technology and file system, realizes the expansion to physical storage device.
Meta data server MDS is responsible for the metadata providing whole file system to client, the view of structure, management document distribution, allow client-access object, manage the NameSpace of whole file system, safeguard the bibliographic structure of whole file system, user right, and be responsible for the data consistency of maintaining file system.By the file of meta data server MDS and directory access management, LUSTRE can control client to the establishment of file system file, deletion, amendment and the access control such as establishment, deletion, amendment to catalogue.By meta data server MDS, client obtains the object storage server OSS at data place, and connect with it, after this read-write operation just carries out between client is with object storage server OSS, unless there are the amendment to NameSpace, no longer will there is relation with meta data server MDS, this reduces the load of meta data server MDS.When multiple client, existing owing to there being multiple object storage server OSS. above-mentioned mode of operation is just converted to parallel work-flow to the access of file system, thus can improve performance preferably.In LUSTRE. client uses and writes back formula cache to ensure the consistency of metadata.
Client, object storage server OSS and meta data server MDS tri-parts, except the function of respective uniqueness, are shared each other and are such as locked, ask the modules such as process, Message Transmission.LUSTRE is the system of a high modularization, and three parts can work on one node, also can at different node points.
Preferably, in order to improve the reliability of system, LUSTRE is that meta data server MDS provides failover function.LUSTRE system can configure two meta data server MDS, and one of them is as backup.Two servers adopt the mode sharing storage to deposit metadata.After certain meta data server MDS breaks down, backup server can take over its service, ensures the normal operation of system.In order to meet the needs of high-performance energy computing system, LUSTRE is optimized for the read-write of large files, for group system provides higher IO throughput.
The performance of LUSTRE file system and autgmentability are all very outstanding, and compared with traditional memory file system, object storage system has following advantage:
(1) parallel data access.Based in the LUSTRE file system that object stores, each user's node directly can access each object storage server OSS, and allows to multiple object storage server OSS concurrent reading and concurrent writing, maximizes the throughput of unique user node.Like this, the performance of whole storage system increases linearly along with the increase of object storage server OSS number.
(2) distributed meta data.NAS system uses a concentrated file server as meta data manager, and some SAN file system then adopt concentrated lock manager, and such metadata management will become a bottleneck.And the meta data manager bottleneck in other shared memory systems is not had based on the LUSTRE file system that object stores.Meta data server MDS separates by the LUSTRE file system stored based on object, and the metadata management work of 90% is distributed in multiple object storage server OSS, effectively eliminates the meta data manager bottleneck problem existed in traditional shared memory systems.
(3) intelligent.Because the storage administration assembly in traditional file systems is transferred in object storage server OSS, make object storage server OSS be provided with powerful management function, system storage managerial role can be simplified.
Because HD video identification part exists the parallel algorithm of large amount of complex, therefore in HD video identification part, adopt LUSTRE, the functional block diagram of HD video identification part as shown in Figure 6.HD video identification part comprises management server node, computing cluster node, meta data server MDS and object storage server OSS, wherein object storage server OSS and meta data server MDS constitutes LUSTRE storage cluster, whole regional deployment LUSTRE parallel file system.Management server node is the interface of service maintenance unit submit job job invocation, and video identification job task submits to computing cluster node by management server node by infiniband network at a high speed; Computing cluster node, according to the task run parallel algorithm program submitted to, completes the process of video file search, decoding, analysis, generation job result; Meta data server and object storage server are the required assemblies of LUSTRE parallel file system, and major function is a large amount of intermediate data that storage computing node produces in the calculation.The video file of HD video storage part storage service, because each computing cluster node is also the client (NFS4 client) of storage server cluster, computing cluster node can be transferred any business video file easily and carry out treatment and analysis.
In order to improve recognition efficiency, the suggestion of video identification parallel algorithm adopts larger cluster scale to go to calculate, and select to adopt LUSTRE, main cause is the performance of LUSTRE excellence and the design of extensibility:
1) Metadata Service and the I/O performance bottleneck being separated the system of decreasing of serving, simultaneously in order to improve extensibility, must reduce the information of system centre Service Management, and reducing the load of central server.LUSTRE is just separated the metadata of file at the beginning of design with data.Meta data server only manages the higher metadata of coherence request.Hundreds of I/O server provides concurrent I/O service.The polymerization I/O performance of system linearly can increase along with the increase of I/O server.The restriction of central server performance to the system expandability reduces greatly.
2) unit that OO storage LUSTRE file stores is not the block of fixed size, but variable size, informative file object.Each file object to correspond on I/O server an independent file.The file size of LUSTRE is no longer subject to the restriction of disk unit and disk file system, maximumly can reach 320TB.
3) I/O server virtual is become can independently process the smart machine of I/O mistake, fault recovery and security control by intelligent storage equipment LUSTRE.The disposal ability be dispersed on I/O equipment can be utilized fully like this, reduce the load of meta data server as much as possible.
4) file itemize storage file can be left in multiple virtual memory facilities by itemize.If identical file is accessed by multiple client simultaneously, the method that itemize stores can disperse I/O to ask, and improves the performance of file access.
5) the LNET module of LUSTRE file system supports infiniband express network completely on transmission network, support the data access technology of RDMA, thus do not need to consider communication data packet transmission problem completely as the cpu resource of the client of computing node, be absorbed in service application completely.In addition, LUSTRE optimizes for the read-write of large files, provide in cluster environment high performance data throughput, global data shared environment, data storage location independence and redundancy scheme is provided to node failure, and cluster reset or server and network failure time quick Resume service, well meet HPCC field store needs.
6) outstanding performance and stability.LUSTRE is the cluster file system of a PetaSeale level.7 cluster-based storage that have employed based on LUSTRE are had in whole world Top10 supercomputer.
Below the system configuration based on the HD video application system of infiniband is understood in detail.The part run related to of following this system works of explanation.
Fig. 7 shows the access flow chart of HD video.The video file of generation to be saved to the virtual folder in cloud bridge middleware by high-definition monitoring image unit by kilomega network, virtual folder is mapped as the file of NFS4 identification by the virtualization modules of cloud bridge middleware, is saved in the space of storage server distribution by video file by infiniband network and RDMA technology.
It is that service maintenance unit directly transfers video file by kilomega network in the virtual folder of cloud bridge middleware that video is transferred, virtual folder is mapped as the file of NFS4 identification by the virtualization modules of cloud bridge middleware, and the video record file transferred by actual needs by infiniband network and RDMA technology is taken out from the physical space that storage server is deposited.
Fig. 8 has shown the video identification flow chart of HD video.Be described as follows:
Step 1: by IP, service maintenance unit safeguards that private network issues job instruction, instruction can comprise the scope target that can search for, as recognition image information, object time conditions information etc.
Step 2: after management server node receives service order, is converted to the initial conditions of parallelism recognition program by job instruction by software, by infiniband net distribution to computing cluster node calculate.
Step 3: computing cluster node is first according to concurrent job initial conditions, as according to target thing time conditions is searched for and located video file required on storage server, then extract this file and decode and obtain frame of video, each frame of video is incorporated as the initial conditions of new parallelism recognition program, is again calculated in computing cluster by video identification parallel algorithm.
Step 4: video identification Paralyzer is run and produced a large amount of intermediate object program, and need shared memory space buffer memory at a high speed, system adopts efficient parallel memorizing LUSTRE storage system.During first establishment file, computing cluster is as LUSTRE client-access meta data server, and meta data server definition and log file leave the relevant information of object storage server OSS position in.During maiden visit file, computing cluster as LUSTRE client-access meta data server, the OSS positional information at locating file place.
Step 5, during first establishment file, meta data server is created data file by the object storage server setting up data file in instruction notification object storage server cluster, and object storage server has created rear feedback target meta data server.
Step 6, computing cluster is as LUSTRE client after obtaining depositing the positional information of intermediate data file, and the intermediate data in a parallel fashion in access object storage server cluster, concurrent access makes operating efficiency greatly improve.
Step 7, after computing cluster node has run video identification routine analyzer, submits to management server node by end product.Result mainly comprises and identifies the associated video files information successfully or identify that unsuccessful reason is fed back.
Step 8, management server node submits client transactions object information to after being arranged by object information.
The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (7)
1., based on a HD video application system of infiniband, comprise monitoring business portion, HD video storage part and video identification analysis portion, wherein
Monitoring business portion, for obtaining HD video data and assigning the order transferred HD video video recording and carry out video identification processing operation;
Described monitoring business portion comprises high-definition monitoring image unit and service maintenance unit, wherein high-definition monitoring image unit is responsible for obtaining HD video data, and based on ICP/IP protocol, obtained HD video data acquisition gigabit Ethernet is converged to cloud bridge middleware cluster by IP operation private network; Described service maintenance unit primary responsibility is assigned and is transferred HD video video recording and carry out the order of video identification processing operation, described in the order transferring HD video video recording and carry out video identification processing operation safeguard that private network converges to described cloud bridge middleware cluster by IP;
HD video storage part connects described monitoring business portion, described HD video storage part comprises cloud bridge middleware cluster and storage server cluster, and adopt IBNAS framework, described cloud bridge middleware cluster passes through infiniband network based on remote direct memory access protocal and described storage server trunking communication, described cloud bridge middleware cluster provides Shared Folders access stencil towards monitoring business portion, simultaneously cloud bridge middleware cluster as the client of IBNAS by the technology access storage server of RDMA;
Video identification analysis portion connects described HD video storage part by infiniband network, described video identification analysis portion also comprises management server node, computing cluster and the LUSTRE storage cluster based on infiniband, described computing cluster is responsible for carrying out identifying processing by submitting to video identification parallel parsing algorithm after high-definition video file decoding, the described LUSTRE storage cluster based on infiniband is object-based distributed file system, is responsible for a large amount of intermediate data storing the generation of described computing cluster;
The described LUSTRE storage cluster based on infiniband comprises meta data server and object storage server, described management server node is the interface of described service maintenance unit submit job job invocation, and video identification job task submits to described computing cluster node by described management server node through infiniband network; Described meta data server is responsible for the metadata providing whole file system to described computing cluster node, the view of structure, management document distribution, allow client-access object, manage the NameSpace of whole file system, safeguard the bibliographic structure of whole file system, user right, and be responsible for the data consistency of maintaining file system; The storage of real data is responsible for by described object storage server, and what process between all computing cluster nodes and physical store is mutual.
2. a kind of HD video application system based on infiniband according to claim 1, described HD video storage part has back-up system, described back-up system comprises the storage system of cloud bridge middleware for subsequent use and connection thereof, an infiniband synchronization link is had to realize the data syn-chronization mirror image ability between active system and back-up system between primary and backup cloud bridge middleware, when the cloud bridge middleware in active system or its storage system connected break down, replacement active system and described monitoring business portion and video identification analysis portion carry out alternately by back-up system.
3. a kind of HD video application system based on infiniband according to claim 2, configuration two meta data servers in described LUSTRE storage cluster, one of them is as backup, and two meta data servers adopt the mode sharing storage to deposit metadata.
4., based on a HD video recognition methods for the HD video application system of infiniband according to claim 1, comprising:
Step 1: by IP, service maintenance unit safeguards that private network issues job instruction;
Step 2: after management server node receives described job instruction, is converted to the initial conditions of parallelism recognition program by described job instruction, by infiniband net distribution to computing cluster node calculate;
Step 3: described computing cluster node is searched for according to described initial conditions and located video file required on storage server, then extract this video file and decode and obtain frame of video, each frame of video is incorporated as the initial conditions of new parallelism recognition program, is again calculated in described computing cluster node by video identification parallel parsing algorithm;
Step 4: a large amount of intermediate object programs that described video identification parallel parsing algorithm produces are buffered in LUSTRE storage cluster, when first establishment file, meta data server in described computing cluster node visit LUSTRE storage cluster, described meta data server definition and log file leave the positional information of object storage server in, when first access file, meta data server in described computing cluster node visit LUSTRE storage cluster, the object stores service positional information at locating file place;
Step 5, during first establishment file, the object storage server that described meta data server sets up data file by instruction notification in object storage server cluster creates data file, and described object storage server has created rear feedback target meta data server;
Step 6, described computing cluster node, after obtaining depositing the positional information of intermediate data file, accesses the intermediate data in described object storage server cluster in a parallel fashion;
Step 7, after described computing cluster node has run described video identification parallel parsing algorithm, has submitted to described management server node by end product;
Step 8, described management server node submits to client by after the finish message of described end product.
5. method according to claim 4, the described instruction in step 1 comprises the scope target that can search for.
6. method according to claim 4, the described end product in step 7 comprises and identifies the associated video files information successfully or identify that unsuccessful reason is fed back.
7., based on an access method for the HD video of the HD video application system of infiniband according to claim 1, comprising:
When preserving video file, the video file of generation to be saved to the virtual folder in cloud bridge middleware by the high-definition monitoring image unit in monitoring business portion by kilomega network, virtual folder is mapped as the file of NFS NFS4 identification by the virtualization modules of cloud bridge middleware, is saved in the space of storage server distribution by infiniband network and remote direct memory access technique by video file;
When transferring video file, the service maintenance unit in described monitoring business portion directly transfers video file by kilomega network in the virtual folder of cloud bridge middleware, virtual folder is mapped as the file of NFS NFS4 identification by the virtualization modules of cloud bridge middleware, and the video record file transferred by actual needs by infiniband network and remote direct memory access technique is taken out from the physical space that storage server is deposited.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210348249.2A CN102833580B (en) | 2012-09-19 | 2012-09-19 | Based on HD video application system and the method for infiniband |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210348249.2A CN102833580B (en) | 2012-09-19 | 2012-09-19 | Based on HD video application system and the method for infiniband |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102833580A CN102833580A (en) | 2012-12-19 |
| CN102833580B true CN102833580B (en) | 2016-03-02 |
Family
ID=47336505
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210348249.2A Expired - Fee Related CN102833580B (en) | 2012-09-19 | 2012-09-19 | Based on HD video application system and the method for infiniband |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102833580B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104092759B (en) * | 2014-07-14 | 2017-10-13 | 南京斯坦德云科技股份有限公司 | A kind of distributed cloud storage service device group system and storage method |
| CN104375889B (en) * | 2014-10-29 | 2018-04-03 | 北京思特奇信息技术股份有限公司 | A kind of Web layers switching system and method |
| CN105068758B (en) * | 2015-07-23 | 2018-06-19 | 清华大学 | Towards the Distributed File System Data I/O optimization methods of parallel data acquisition |
| CN105791888A (en) * | 2016-03-09 | 2016-07-20 | 浪潮软件股份有限公司 | Video analyzing method and video analyzing device |
| CN108123984A (en) * | 2016-11-30 | 2018-06-05 | 天津易遨在线科技有限公司 | A kind of memory database optimizes server cluster framework |
| CN106657076B (en) * | 2016-12-26 | 2020-04-07 | 北京神州绿盟信息安全科技股份有限公司 | TCP service implementation method and device for network name space |
| CN107203639A (en) * | 2017-06-09 | 2017-09-26 | 联泰集群（北京）科技有限责任公司 | Parallel file system based on High Performance Computing |
| CN110381165A (en) * | 2019-08-06 | 2019-10-25 | 段磊 | A kind of shared memory systems of the high-availability cluster based on video monitoring |
| CN110445994B (en) * | 2019-08-21 | 2021-10-15 | 成都索贝数码科技股份有限公司 | Method for realizing large-format multilayer real-time editing based on interframe segmentation clustering |
| CN113031852A (en) * | 2019-12-25 | 2021-06-25 | 阿里巴巴集团控股有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111274176B (en) * | 2020-01-15 | 2022-04-22 | 联想(北京)有限公司 | Information processing method, electronic equipment, system and storage medium |
| CN111277801B (en) * | 2020-03-03 | 2021-08-31 | 西安烽火软件科技有限公司 | Monitoring video storage system based on content and application method |
| CN113535068B (en) * | 2020-04-21 | 2024-09-17 | 华为技术有限公司 | Data reading method and system |
| CN116228433B (en) * | 2023-01-03 | 2024-05-17 | 易方达基金管理有限公司 | Method, apparatus, device and readable storage medium for returning bond combination performance |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1892586A (en) * | 2005-06-28 | 2007-01-10 | 惠普开发有限公司 | Centralized hot-pluggable video and kvm |
| CN101621424A (en) * | 2009-05-07 | 2010-01-06 | 曙光信息产业(北京)有限公司 | Intelligent monitoring method of Infiniband exchanger |
| CN101632090A (en) * | 2005-02-08 | 2010-01-20 | 国际商业机器公司 | The system and method that is used for selective image capture, transmission and reconstruct |
| CN102223388A (en) * | 2010-05-10 | 2011-10-19 | 北京月新时代科技有限公司 | Cluster memory |
-
2012
- 2012-09-19 CN CN201210348249.2A patent/CN102833580B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101632090A (en) * | 2005-02-08 | 2010-01-20 | 国际商业机器公司 | The system and method that is used for selective image capture, transmission and reconstruct |
| CN1892586A (en) * | 2005-06-28 | 2007-01-10 | 惠普开发有限公司 | Centralized hot-pluggable video and kvm |
| CN101621424A (en) * | 2009-05-07 | 2010-01-06 | 曙光信息产业(北京)有限公司 | Intelligent monitoring method of Infiniband exchanger |
| CN102223388A (en) * | 2010-05-10 | 2011-10-19 | 北京月新时代科技有限公司 | Cluster memory |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102833580A (en) | 2012-12-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102833580B (en) | Based on HD video application system and the method for infiniband | |
| US10929428B1 (en) | Adaptive database replication for database copies | |
| US11567674B2 (en) | Low overhead resynchronization snapshot creation and utilization | |
| US11921679B2 (en) | Synchronizing configuration of partner objects across distributed storage systems using transformations | |
| US10545994B2 (en) | Data replication method and storage system | |
| CN103312791B (en) | Internet of Things isomeric data storage means and system | |
| Jian-Hua et al. | Cloud computing-based data storage and disaster recovery | |
| CN103763383A (en) | Integrated cloud storage system and storage method thereof | |
| US11836115B2 (en) | Gransets for managing consistency groups of dispersed storage items | |
| CN102413172B (en) | Parallel data sharing method based on cluster technology and apparatus thereof | |
| CN106502823A (en) | data cloud backup method and system | |
| CA2910270A1 (en) | Efficient read replicas | |
| CN105468296B (en) | Nothing based on virtual platform shares memory management method | |
| CN103268318A (en) | Distributed key value database system with strong consistency and read-write method thereof | |
| CN103166785A (en) | Distributed type log analysis system based on Hadoop | |
| US11449260B2 (en) | Persistent hole reservation | |
| CN111813332A (en) | High-performance, high-expansion and high-safety intelligent distributed storage system | |
| US11714782B2 (en) | Coordinating snapshot operations across multiple file systems | |
| US8516044B1 (en) | Systems and methods for scalable heterogeneous and homogeneous unified enterprise storage clusters | |
| US20200285655A1 (en) | Non-disruptive transition to synchronous replication state | |
| CN103384267A (en) | Parastor200 parallel storage management node high availability method based on distributed block device | |
| Park et al. | An analysis of replication enhancement for a high availability cluster | |
| US9152687B2 (en) | Cross domain locking in a distributed environment | |
| Zhou et al. | ECStor: A flexible enterprise-oriented cloud storage system based on GlusterFS | |
| Guo-song et al. | Design and implementation of iSCSI Out-of-band storage virtualization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C53 | Correction of patent of invention or patent application | ||
| CB02 | Change of applicant information |
Address after: Six floor, No. 1 building, East Midlands Guanghua Road, Baixia District of Nanjing city in Jiangsu province 210014 Applicant after: NANJING STD CLOUD TECHNOLOGY CO.,LTD. Address before: Six floor, No. 1 building, East Midlands Guanghua Road, Baixia District of Nanjing city in Jiangsu province 210014 Applicant before: Nanjing Standard Communications Co.,Ltd. |
|
| COR | Change of bibliographic data |
Free format text: CORRECT: APPLICANT; FROM: NANJING STANDARD COMMUNICATIONS CO., LTD. TO: NANJING SITANDE CLOUD TECHNOLOGY CO., LTD. |
|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20160302 |