CN104023058B - Management and distribution method and system for storage resources - Google Patents

Management and distribution method and system for storage resources Download PDF

Info

Publication number
CN104023058B
CN104023058B CN201410245074.1A CN201410245074A CN104023058B CN 104023058 B CN104023058 B CN 104023058B CN 201410245074 A CN201410245074 A CN 201410245074A CN 104023058 B CN104023058 B CN 104023058B
Authority
CN
China
Prior art keywords
ip
storage
encoder
slice
datanode
Prior art date
Application number
CN201410245074.1A
Other languages
Chinese (zh)
Other versions
CN104023058A (en
Inventor
成浩
Original Assignee
浙江宇视科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 浙江宇视科技有限公司 filed Critical 浙江宇视科技有限公司
Priority to CN201410245074.1A priority Critical patent/CN104023058B/en
Publication of CN104023058A publication Critical patent/CN104023058A/en
Application granted granted Critical
Publication of CN104023058B publication Critical patent/CN104023058B/en

Links

Abstract

本发明公开了一种存储资源的管理和分配方法,包括:数据管理节点DataNode接收IP编码器的分配存储空间请求;所述DataNode按照第一策略为所述IP编码器选择存储资源,并根据第二策略为所述IP编码器分配所选存储资源中的存储空间片(slice),以供所述IP编码器根据所选存储资源的iSCSI挂载信息,通过iSCSI挂载存储资源,根据所分配的存储空间slice的起始地址、空间大小进行iSCSI数据写入。 The present invention discloses a method of storing the management and allocation of resources, comprising: a data management node receives the IP DataNode encoder allocates storage request; DataNode said selected memory resources for the IP encoder according to a first strategy, and in accordance with the first two selected memory resource allocation policy storage slice (slice) is the IP encoder, the encoder for use by the IP iSCSI mount according to the selected information storage resources, storage resources via iSCSI mount, according to the allocated storage slice start address, data is written for the iSCSI space. 本发明同时还公开了一种存储资源的管理和分配系统。 The present invention also discloses a system for the management and allocation of storage resources.

Description

一种存储资源的管理和分配方法和系统 Management and distribution method and system for storage resources

技术领域 FIELD

[0001] 本发明涉及互联网的媒体流i SCSI块直存技术领域,尤其涉及一种存储资源的管理和分配方法和系统。 [0001] relates to the Internet media stream i block of the present invention in particular to a SCSI storage resource management and allocation method and system for direct deposit technical field.

背景技术 Background technique

[0002]随着高清在监控业务系统中的应用,对监控录像存储的可靠性、性能等要求越来越高。 [0002] With the application of high-definition monitoring business systems, surveillance video storage for reliability, performance and other requirements higher and higher. 而前端互联网协议(IP,Internet Protocol)编码器到IP存储局域网(IPSAN,IPStorage Area Network)的端到端的媒体流块直存技术就是一种较好的录像存储方案。 The front-end Internet Protocol (IP, Internet Protocol) IP encoder to a storage area network (IPSAN, IPStorage Area Network) end to end media stream block direct deposit technology is a better video storage solutions. 如图1所示,为相关技术中IP编码器到IPSAN的端到端的媒体流块直存示意图,在存储方式上,媒体流块直存的数据管理方式抛弃了媒体服务器;在IP网络的基础上,在编码器中集成了iSCSI模块,使得编码器可以基于iSCSI协议端到端的将录像数据写入IPSAN存储设备中。 1, the related art of the encoder to the end of the IP media stream IPSAN schematic block direct deposit, on storage media data stream block direct deposit management abandoned media server; IP network on the basis of the integrated iSCSI module in the encoder, the encoder such that the video data may be based on the iSCSI protocol IPSAN end will be written in the storage device. 当监控点的图像需要进行集中存储时,前端IP编码器与IPSAN存储设备建立iSCSI连接,将IPSAN存储设备的存储资源通过iSCSI协议挂载到IP编码器本地,然后对存储视频流进行iSCSI协议封装,直接采用数据块的方式将媒体数据写入IPSAN存储设备中。 Monitoring points when the image needs to be centralized storage, the encoder and the front end IP storage device IPSAN iSCSI connection establishment, storage resources IPSAN storage device mounted to a local IP encoder iSCSI protocol, and then stores the video stream encapsulated iSCSI protocol , direct use of the data block write data to the media way IPSAN storage device.

[0003] 而在媒体流数据块直存技术的基础上,对存储空间的分配管理成为了一个关键问题,摄像机录像存储空间需要满足用户指定的录像留存期的需求,同时可以支持灵活扩容,另外需要提高空间利用率,避免存储空间浪费。 [0003] The basis of the media stream data block direct deposit technology, distribution management for storage space has become a key issue, camcorder video storage space required to meet the needs of video retention period specified by the user, and can support flexible expansion, in addition We need to improve space utilization, avoid wasting storage space.

[0004] 相关技术中,监控iSCSI块存储的空间分配机制,是基于逻辑卷管理(LVM,LogicVolume Manage)的存储虚拟化技术,即在屏蔽底层物理存储实体的情况下,为每个摄像机单独分配一段逻辑存储空间,该逻辑存储空间称为存储资源。 [0004] In the related art, space allocation monitor iSCSI block storage mechanism is based on the Logical Volume Manager (LVM, LogicVolume Manage) storage virtualization technology, i.e. in the case of shielding the underlying physical storage entity, assigned separately for each camera section of the logical storage space, called the logical memory space of storage resources. 块存储资源管理的分层结构如图2所示,所谓存储资源,是通过iSCSI Target (标签)的逻辑单元号(LUN,Logic UnitNumber)与LVM层的逻辑卷进行绑定,从而为iSCSI存储业务提供的一段逻辑地址空间。 , The so-called storage resource block storage hierarchy 2 in FIG resource management, is bound by the Target iSCSI (tag) of the logical unit number (LUN, Logic UnitNumber) LVM logical volume layer, so as iSCSI storage service It provided some logical address space. 而LVM层是由物理卷(PV)、卷组(VG)和逻辑卷(LV)等不同实体组成。 LVM layer is formed while the physical volume (PV), the volume group (VG) and logical volume (LV), such as different entities. 物理卷是与物理阵列或物理磁盘相对应的实体,卷组是所有加入本卷组的物理卷的集合;而逻辑卷则是在卷组上分配的一块逻辑地址空间,在i SCSI存储应用中,与i SCSI TargeM^LUN进行绑定。 Physical volume is a physical or physical disk array corresponding to the entity, the volume group added to all physical volumes in this volume group set; logical volume is allocated on a logical address space in the volume group, the i SCSI storage applications , and i SCSI TargeM ^ LUN binding.

[0005] 通过上述方案,可以为上层存储业务提供一种虚拟化的存储空间管理机制,实现了对底层实际的物理磁盘阵列的屏蔽。 [0005] By the above-described embodiment, it is possible to provide a virtualized storage space for the upper storage business management system, to achieve shielding of the actual physical underlying disk array. LVM层目前只支持逻辑卷的分配、删除、扩容,不支持单个逻辑卷的容量缩减,同时逻辑卷本身不提供小粒度、可分配管理的存储单元,因此逻辑卷本质上是一种空间独享的技术。 LVM logical layer only supports allocation, deletion, expansion, does not support a single logical volume capacity reduction, while the small particle size does not provide logical volume, managing the storage unit can be allocated, so the space is essentially a logical exclusive Technology. 在监控iSCSI块直存业务中为摄像机分配存储资源,对于LVM层,实际是在物理卷组成的卷组空间上分配一块逻辑卷,并将逻辑卷与为摄像机iSCSI直存所创建的[Target,LUN]进行绑定。 Monitoring direct deposit service iSCSI block allocation of storage resources for the camera, for LVM layer, an actually allocated logical volume group on the space of physical volumes, logical volume and direct deposit created for the camera iSCSI [the Target, LUN] bind. 这样,在前端IP编码器的iSCSI Initiator (iSCSI发起)端与IPSAN存储设备上的[Target,LUN]建立ISCSI连接后,可以将IPSAN存储设备上分配的逻辑卷映射为前端本地的一个逻辑盘,以自定义的块数据格式写入数据到逻辑盘,从而实现将媒体流数据通过iSCSI协议直存到存储设备中。 Thus, the front end of the encoder IP iSCSI Initiator (iSCSI initiated) [Target, LUN] IPSAN on the end of the storage device ISCSI connection is established, may be allocated on a logical storage device IPSAN mapped to a distal end of the local logical disk, writing data to the block data to the logical disk custom format, to achieve the linear media stream data stored in the storage device through the iSCSI protocol.

[0006]目前的iSCSI块存储的空间分配机制,是一种将单个存储资源空间一次性分配给单个摄像机、并由该摄像机独享写入该空间的机制。 [0006] Spatial distribution mechanism of the current block storage iSCSI, a disposable single storage resource allocating space for a single camera, the camera by exclusive write mechanism of the space. 对于告警联动类存储、或者存储码率动态变化的存储业务,由于无法提前预估出摄像机存储空间的大小,因此不适合采用这种预先为单个摄像机分配固定存储空间的机制。 For dynamic alarm linkage storage class, storage or stored bit rate service, since the camera can not predict the size of the storage space in advance, this mechanism is not suitable for pre-allocate storage space as a single fixed camera. 在很多情况下,一些老的监控局点会新购置一些存储设备、或者在原有存储设备上增加磁盘,但是该方案无法支持对已有摄像机存储资源在系统新增存储设备、磁盘时的自动扩容,在系统升级改造上存在较多不便。 In many cases, some of the old game point monitoring will purchase some of the new storage device, or increase the disk storage device in the original, but the program does not support automatic expansion of existing storage resources in the camera system to add storage devices, disk , present on the system upgrade, more inconvenience.

发明内容 SUMMARY

[0007] 为解决现有存在的技术问题,本发明实施例提供一种存储资源的管理和分配方法和系统。 [0007] To solve the problems in the prior art, embodiments of the present invention provide management and distribution method and system storage resources.

[0008] 本发明实施例提供了一种存储资源的管理和分配方法,该方法包括: [0008] The embodiments of the present invention provides a method of storing the management and allocation of resources, the method comprising:

[0009] 数据管理节点DataNode接收IP编码器的分配存储空间请求; [0009] Data management node receives the IP DataNode encoder allocates storage request;

[0010] 所述DataNode按照第一策略为所述IP编码器选择存储资源,并根据第二策略为所述IP编码器分配所选存储资源中的存储空间片Slice,以供所述IP编码器根据所选存储资源的iSCSI挂载信息,通过iSCSI挂载存储资源,根据所分配的存储空间slice的起始地址、空间大小进行iSCSI数据写入。 [0010] The DataNode selected according to the first policy IP encoder memory resources, and allocating storage resources selected memory Slice sheet as the IP encoder according to a second policy for the IP encoder the iSCSI mount the selected information storage resources, storage resources via iSCSI mount performs data writing according to the allocated iSCSI storage slice start address space.

[0011] 其中,所述第一策略包括: [0011] wherein said first policy comprises:

[0012] 如果当前正在分配slice的存储资源上仍有空闲状态的slice,则继续选择当前存储资源;且, [0012] If the current storage resources are allocated slice on slice still idle, then continue to select the current storage resources; and,

[0013] 如果当前正在分配slice的存储资源的占用slice达到所述存储资源总slice的第一预设比例以上,则选择下一剩余存储空间最大的存储资源。 The first preset ratio [0013] If the current slice of the storage resources are allocated slice to a total occupation of the memory resource of the slice or higher, selects the next largest remaining storage space of the storage resource.

[00M] 其中,该方法进一步包括:所述DataNode对存储资源以slice为单位进行切分,并通过空闲si ice队列和占用si ice队列对存储资源中的si ice进行管理; [00M] wherein, the method further comprising: said slice DataNode storage resources in units of segmentation, and management of the storage resources by si ice idle queue and si ice si ice queue occupancy;

[0015] 所述第二策略包括:在所述IP编码器请求的存储资源中选择空闲slice队列中的slice分配给所述IP编码器,且在应答成功后,将本次分配的slice更新为占用状态并调度到占用队列中。 [0015] The second policy comprises: selecting the free slice assignment slice queue storage resources of the IP encoder IP request to the encoder, and after the success of the response, the slice allocated for the updating of this occupancy status and schedule queue occupancy.

[0016] 其中,该方法进一步包括:在收到所述IP编码器发送的分配新slice的请求时,所述DataNode根据所述IP编码器分配新s I ice的请求,判断所述原si ice所属的存储资源中slice的占用率是否达到第三预设比例,如果未达到第三预设比例,则继续为所述IP编码器分配相同存储资源中的空闲slice,如果达到第三预设比例,则为所述IP编码器选择所述DataNode的其他存储资源; [0016] wherein, the method further comprising: upon receipt of a new request for allocating the IP slice encoder transmitted the allocation request new DataNode s I ice encoder in accordance with the IP, it is judged the original si ice storage resource occupancy slice belongs has reached a third preset threshold, if the third predetermined ratio has not reached, continue allocating the same resources stored in the free slice encoder to the IP, if the ratio reaches the third preset , compared to the encoder selects the IP DataNode other storage resources;

[0017] 如果所述DataNode的所有存储资源的slice占用率都达到所述第三预设比例,则删除所述DataNode中的老化数据并将所删数据对应的slice挂回到空闲slice队列,为所述IP编码器选择剩余slice空间最大的存储资源,并分配所选存储资源的空闲slice给所述IP编码器; [0017] If all slice usage of the storage resources DataNode has reached the third predetermined ratio, and to delete the aging data slice corresponding to the deleted data in the DataNode linked to an idle slice queue, is the encoder selects the remaining slice IP largest storage space resources, storage resources and allocates the selected slice to the idle IP encoder;

[0018] 其中,所述分配新slice的请求,是由所述IP编码器在所分配slice的空间被写入比例达到第二预设比例时,向所述DataNode发送的。 [0018] wherein, the slice of the new allocation request, the IP is the proportion of the encoder is written in the second predetermined ratio of space allotted slice, it is transmitted to the DataNode.

[0019] 其中,该方法进一步包括: [0019] wherein the method further comprises:

[°02°] 在所述IP编码器向DataNode发送分配存储空间请求之前,所述IP编码器向Master管理节点请求分配DataNode; [° 02 °] IP prior to said encoder transmits a request to allocate storage space for DataNode, requesting allocation of the IP encoder to DataNode Master management node;

[0021] 所述Master管理节点根据第三策略为所述IP编码器分配所述DataNode。 [0021] The Master node to assign the management of the IP DataNode encoder according to the third strategy.

[0022] 其中,所述第三策略包括: [0022] wherein said third policy comprises:

[0023] 根据各DataNode的剩余存储空间、剩余输入输出1处理能力,并基于负载均衡、网络直连因素,为所述IP编码器分配DataNode。 [0023] The DataNode remaining storage space of each of the remaining processing capacity of input and output 1, and based on load balancing, network attached factors assigned to said IP DataNode encoder.

[0024] 其中,所述Master管理节点包括主Master管理节点和辅Master管理节点,该方法进一步包括: [0024] wherein the node includes a master management Master Master Master secondary management node and the management node, the method further comprising:

[0025] 所述辅Master管理节点用于对所述主Master管理节点进行冗余保护,在所述主Master管理节点故障时,所述辅Master管理节点切换为主Master管理节点。 [0025] The secondary management Master Master node for the master node for redundancy management, when the Master management master node fails, the secondary switch to the master node management Master Master management node.

[0026] 其中,该方法进一步包括:所述Master管理节点在所述DataNode故障或断网时,为所述故障或断网的DataNode所对应的IP编码器分配新的可用DataNode。 [0026] wherein the method further comprises: the management node when the Master DataNode failure or disconnection, allocates a new DataNode is available DataNode failure or disconnection of the corresponding IP encoder.

[0027] 本发明实施例还提供了一种存储资源的管理和分配系统,该系统包括:IP编码器和数据管理节点DataNode,其中, Example [0027] The present invention further provides a system for the management and allocation of storage resources, the system comprising: IP encoder and a data management node DataNodes, wherein

[0028] 所述IP编码器用于,向所述DataNode发送分配存储空间请求; [0028] The IP encoder for transmitting an allocation request to the storage space DataNodes;

[0029] 所述DataNode用于,按照第一策略为所述IP编码器选择存储资源,并根据第二策略为所述IP编码器分配所选存储资源中的存储空间片slice; [0029] The DataNode for selecting resources for storing the IP encoder according to a first policy and the selected slice allocated storage space sheet storage resources of the IP encoder according to a second policy;

[0030] 所述IP编码器还用于,根据所选存储资源的iSCSI挂载信息,通过iSCSI挂载存储资源,根据所分配的存储空间slice的起始地址、空间大小进行iSCSI数据写入。 [0030] The encoder is further configured IP, iSCSI mount according to the selected information storage resources, storage resources via iSCSI mount performs data writing according to the allocated iSCSI storage slice start address space.

[0031] 其中,所述DataNode进一步用于,按照以下第一策略为所述IP编码器选择存储资源: [0031] wherein the DataNode further configured to select the storage resources for the first IP encoder according to the following strategy:

[0032] 如果当前正在分配slice的存储资源上仍有空闲状态的slice,则继续选择当前存储资源;且, [0032] If the current storage resources are allocated slice on slice still idle, then continue to select the current storage resources; and,

[0033] 如果当前正在分配slice的存储资源的占用slice达到所述存储资源总slice的第一预设比例以上,则选择下一剩余存储空间最大的存储资源。 The first preset ratio [0033] If the current slice of the storage resources are allocated slice to a total occupation of the memory resource of the slice or higher, selects the next largest remaining storage space of the storage resource.

[0034] 其中,所述DataNode进一步用于,对存储资源以si ice为单位进行切分,并通过空闲slice队列和占用slice队列对存储资源中的slice进行管理; [0034] wherein the DataNode further configured si ice storage resources to be segmented in units, and the free slice by slice queue occupancy and queue storage resources of the slice management;

[0035] 所述DataNode根据以下第二策略分配slice:在所述IP编码器请求的存储资源中选择空闲slice队列中的slice分配给所述IP编码器,且在应答成功后,将本次分配的slice更新为占用状态并调度到占用队列中。 [0035] The second policy is assigned according to the following DataNode slice: the free slice assignment queue selected slice in the storage resource of the IP encoder IP request to the encoder, and after the success of the response, the distribution of this the slice is updated to the occupied state and an occupancy schedule queue.

[0036] 其中,所述IP编码器进一步用于,在所分配slice的空间被写入比例达到第二预设比例时,向所述DataNode请求分配新的s I ice; [0036] wherein said encoder further configured IP is written in the ratio reaches a second predetermined proportion of the space allotted slice of time, the request to allocate a new DataNode s I ice;

[0037] 所述DataNode进一步用于,根据所述IP编码器分配新sI ice的请求,判断所述原slice所属的存储资源中slice的占用率是否达到第三预设比例,如果未达到第三预设比例,则继续为所述IP编码器分配相同存储资源中的空闲slice,如果达到第三预设比例,则为所述IP编码器选择所述DataNode的其他存储资源; [0037] Further for the DataNode, allocation request in accordance with the new IP sI ice encoder, determining whether the occupancy rate of the primary storage resources slice belongs slice reaches a third preset proportion, if not reached the third preset ratio, the same allocation of storage resources to continue the free slice encoder to the IP, if it reaches a third predetermined ratio, compared with the other IP encoder selects the DataNode of storage resources;

[0038] 如果所述DataNode的所有存储资源的slice占用率都达到所述第三预设比例,则删除所述DataNode中的老化数据并将所删数据对应的slice挂回到空闲slice队列,为所述IP编码器选择剩余slice空间最大的存储资源,并分配所选存储资源的空闲slice给所述IP编码器。 [0,038] If the occupancy rate of all storage resources slice the DataNode has reached the third predetermined ratio, and then deleting the aging of the deleted data corresponding to the data slice is linked to an idle DataNode slice queue, is the encoder selects the remaining slice IP largest storage space resources, storage resources and allocates the selected slice to the idle IP encoder.

[0039] 其中,该系统还包括:Master管理节点; [0039] wherein the system further comprises: Master management node;

[0040] 所述IP编码器进一步用于,在向所述DataNode发送分配存储空间请求之前,向所述Master管理节点请求分配DataNode; [0040] The IP encoder is further configured to, before sending the request to allocate storage space DataNode, allocation request to the Master DataNode management node;

[0041] 所述Master管理节点用于,在收到所述IP编码器分配DataNode的请求后,根据第三策略为所述IP编码器分配所述DataNode。 [0041] The Master node for managing, after receiving the request DataNode encoder IP assignment, the assigned IP DataNode encoder according to the third strategy.

[0042] 其中,所述Master管理节点进一步用于,根据以下第三策略为IP编码器分配DataNode: [0042] wherein, the Master node manager is further configured to assign IP DataNode third encoder according to the following strategy:

[0043] 根据各DataNode的剩余存储空间、剩余输入输出1处理能力,并基于负载均衡、网络直连因素,为所述IP编码器分配DataNode。 [0,043] The DataNode remaining storage space of each of input and output 1 remaining processing capacity, and based on load balancing, network attached factors assigned to said IP encoder DataNode.

[0044] 其中,所述Master管理节点包括主Master管理节点和辅Master管理节点; [0044] wherein the node includes a master management Master Master Master secondary management node and management node;

[0045] 所述辅Master管理节点用于对所述主Master管理节点进行冗余保护,在所述主Master管理节点故障时,所述辅Master管理节点切换为主Master管理节点。 [0045] The secondary management Master Master node for the master node for redundancy management, when the Master management master node fails, the secondary switch to the master node management Master Master management node.

[0046] 其中,所述Master管理节点进一步用于,在所述DataNode故障或断网时,为所述故障或断网的DataNode所对应的IP编码器分配新的可用DataNode。 [0046] wherein, the Master node manager is further configured to, when the DataNode disconnection or failure as a failure or disconnection of the DataNode corresponding to the encoder allocates a new IP DataNode available.

[0047] 本发明实施例所提供的一种存储资源的管理和分配方法和系统,在保持媒体流iSCSI直存的高性能优势的前提下,实现了存储空间的单元切分与弹性动态分配机制,满足告警联动存储、变码率存储等无法预估摄像机存储空间的场景需求,达到最大程度提高存储空间利用率的效果;还满足了已有摄像机存储空间在系统增加新设备、新磁盘后的自动扩容需求;支持摄像机存储空间在存储设备集群任意一到多台设备上的分配;基于分布式存储集群机制,实现了对Master管理节点、任意DataNode的故障容错机制,最大程度上保障了IP编码器的录像存储不中断,提高了录像存储业务的可靠性。 [0047] The management and distribution method and system storage resources provided by the embodiment according to the present invention, the media stream while maintaining a straight iSCSI storage performance advantages achieved with the elastic segmentation unit dynamic allocation of storage space to meet the alarm linkage memory, storage and other variable rate demand can not predict the scene camera storage space, to maximize storage space utilization effect; but also to meet the existing camera system to increase storage space in new equipment, new disk automatic expansion needs; support any camera storage space allocation on a device to multiple storage devices in a cluster; based on distributed memory cluster mechanism, the realization of Master node management, fault tolerance mechanism in any DataNode maximum extent protect the IP coding video memory is not interrupted, improve the reliability of video storage business.

附图说明 BRIEF DESCRIPTION

[0048]图1为相关技术中IP编码器到IPSAN的端到端的媒体流块直存示意图; [0048] FIG. 1 is a related art encoder-to-end IP media flow IPSAN schematic block direct deposit;

[0049]图2为相关技术中块存储资源管理的分层结构示意图; [0049] FIG. 2 is a schematic block storage hierarchy of the resource management related art;

[0050]图3为本发明实施例的一种存储资源的管理和分配方法的流程图; [0050] FIG 3 a flow diagram of a method for managing and allocating storage resources embodiment of the present invention;

[0051]图4为本发明实施例的一种存储资源的管理和分配系统的结构示意图; [0051] FIG. 4 is a schematic configuration management and distribution system according to an embodiment of the storage resources embodiment of the present invention;

[0052]图5为本发明实施例的一种硬件组网示意图; [0052] FIG. 5 networking schematic diagram of a hardware embodiment of the present invention;

[0053]图6为本发明实施例的另一种存储资源的管理和分配方法的流程图。 [0053] The flowchart of FIG. 6 and dispensing management method of storage resources of another embodiment of the present invention.

具体实施方式 Detailed ways

[0054]下面结合附图和具体实施例对本发明的技术方案进一步详细阐述。 Drawings and specific embodiments set forth in detail further aspect of the present invention [0054] The following binding.

[0055] 本发明实施例提供的一种存储资源的管理和分配方法,如图3所示,主要包括: [0055] A memory management and resource allocation method according to an embodiment of the present invention, shown in Figure 3, including:

[0056] 步骤301,数据管理节点DataNode接收IP编码器的分配存储空间请求。 [0056] Step 301, the data management node receives the IP DataNode encoder memory allocation request.

[0057] 本发明实施例的DataNode部署于IPSAN存储设备中,用于管理其所属IPSAN存储设备的存储资源,DataNode对其所属IPSAN存储设备的存储资源以片(slice)为单位进行切分,并通过空闲slice队列和占用slice队列对存储资源中的slice进行管理,未被分配的slice为空闲状态并加入到空闲队列中,已被分配的slice为占用状态并调度到占用队列中。 DataNode embodiment [0057] The present invention is deployed in IPSAN storage device for storing their respective resource management IPSAN storage device, storage resources relevant to their DataNode storage device IPSAN slice (Slice) for segmentation units, and idle slice by slice queue occupancy and queue storage resources management slice, the slice is not allocated to the idle state and added to the free list have been allocated for the slice occupied state and an occupancy schedule queue.

[0058] IP编码器可以包括具有编码功能且将编码后数据封装成IP数据包的编码器,还可以包括带有摄像功能,并将拍摄后的图像数据编码并封装成IP数据包的网络摄像机IPC。 [0058] IP encoder may include a coding function and the encoded data is encapsulated into IP packets encoder may further include an imaging function with, the image data captured and encoded encapsulated into an IP packet network camera IPC.

[0059] IP编码器需要存储空间进行iSCSI数据写入时,向DataNode发送分配存储空间请求。 When the [0059] IP encoder needs the storage space for the iSCSI data write, send a request to allocate storage space for DataNode.

[0060] 步骤302 ,DataNode按照第一策略为IP编码器选择存储资源,并根据第二策略为IP编码器分配所选存储资源中的存储空间slice,以供IP编码器根据所选存储资源的iSCSI挂载信息,通过iSCSI挂载存储资源,根据所分配的存储空间slice的起始地址、空间大小进行iSCSI数据写入。 [0060] Step 302, DataNode storage resource according to a first policy to select an IP encoder, and allocating the selected resource stored in the storage space for the IP slice encoder according to a second policy for IP encoder according to the selected storage resources iSCSI mount information via iSCSI mount storage resources, perform data writing according to the allocated iSCSI storage slice start address space.

[0061] DataNode根据IP编码器的请求,为所述IP编码器选择存储资源,并在所选存储资源中分配存储空间slice; IP编码器根据所选存储资源的iSCSI挂载信息,通过iSCSI挂载存储资源,根据所分配的存储空间slice的起始地址、空间大小进行iSCSI数据写入。 [0061] DataNode encoder according to the request of IP, the IP selected as the encoder memory resources, and allocating storage space in the selected slice in the storage resource; IP encoders iSCSI mount according to the selected information storage resources, linked via iSCSI carrier storage resources, based on the slice allocated storage start address, data is written for the iSCSI space.

[0062] 其中,所述第一策略包括: [0062] wherein said first policy comprises:

[0063] 如果当前正在分配slice的存储资源上仍有空闲状态的slice,则继续选择当前存储资源;且, [0063] If the current storage resources are allocated slice on slice still idle, then continue to select the current storage resources; and,

[0064] 如果当前正在分配slice的存储资源的占用slice达到所述存储资源总slice的第一预设比例以上,则选择下一剩余存储空间最大的存储资源。 The first preset ratio [0064] If the current slice of the storage resources are allocated slice to a total occupation of the memory resource of the slice or higher, selects the next largest remaining storage space of the storage resource.

[0065] 也就是说,优先选择当前分配slice所在的存储资源。 [0065] In other words, preferred storage resources currently allocated slice is located. 其中,第一预设比例可以根据实际需要进行设定,例如可以设定为90%。 Wherein the first predetermined ratio may be set according to actual needs, for example, may be set to 90%.

[0066] 所述第二策略包括:在所述IP编码器请求的存储资源中选择空闲slice队列中的slice分配给所述IP编码器,且在应答成功后,将本次分配的slice更新为占用状态并调度到占用队列中。 [0066] The second policy comprises: selecting the free slice assignment slice queue storage resources of the IP encoder IP request to the encoder, and after the success of the response, the slice allocated for the updating of this occupancy status and schedule queue occupancy.

[0067] 本发明实施例中,IP编码器在所分配slice的空间被写入比例(即在分配给IP编码器的slice空间中已经写入数据的slice空间所占的比例)达到第二预设比例时,可以向DataNode请求分配新的si ice,同时IP编码器在原slice中继续数据写入;其中,所述第二预设比例可以根据实际需要进行设定,例如可以设定为90%; Embodiment, the encoder is written IP space allocation ratio of the slice (i.e., slice data has been written in the space allocated to the encoder in the IP space occupied ratio slice) embodiment [0067] The present invention achieves the second pre- when the ratio set to be a request to allocate a new DataNode si ice, while IP data is written in the encoder continues the original slice; and wherein the second predetermined ratio may be set according to actual needs, for example, may be set to 90% ;

[0068] 在收到所述IP编码器发送的分配新si ice的请求时,DataNode判断所述原si ice所属的存储资源中slice的占用率(即所述存储资源中已经写入数据的slice空间占所述存储资源总slice空间的比例)是否达到第三预设比例,如果未达到第三预设比例,则继续为所述IP编码器分配相同存储资源中的空闲slice,如果达到第三预设比例,则为所述IP编码器选择所述DataNode的其他存储资源;其中,所述第三预设比例可以根据实际需要进行设定(可以与第二预设比例相同,也可以不同),例如可以设定为90%; [0068] Upon receipt of request allocation of the new IP si ice encoder transmitted, determining whether the primary storage resource DataNode si ice belongs occupancy of the slice (i.e., slice in the storage resource of the data has been written spatial proportion of the total space of the storage resource slice) has reached a third preset threshold, if the third predetermined ratio has not reached, continue allocating the same resources stored in the free slice encoder to the IP, if it reaches a third preset ratio, compared with the other IP encoder selects the DataNode storage resource; and wherein said third predetermined ratio may be set according to actual needs (the second predetermined ratio may be the same or different) , may be set to 90%;

[0069] 如果所述DataNode的所有存储资源的slice占用率都达到所述第三预设比例,则删除所述DataNode中的老化数据(如存储时间最长的数据、或者重要性不高的数据等)并将所删数据对应的slice挂回到空闲slice队列,为所述IP编码器选择剩余slice空间最大的存储资源,并分配所选存储资源的空闲slice给所述IP编码器。 [0069] If all the storage resources of the slice DataNode occupancy rate has reached the third predetermined proportion, DataNode delete the data in aging (e.g., longest data storage, or importance of the data is not high etc.) and the data slice corresponding to an idle deletion hanging slice queue, the remaining slice select maximum storage space resources of the IP encoder, and assign the selected slice of storage resources to the idle IP encoder.

[0070] 本发明实施例中,在所述IP编码器向DataNode发送分配存储空间请求之前,所述IP编码器向Master管理节点请求分配DataNode; Embodiment [0070] In the present invention, prior to transmitting said IP encoder DataNode request to allocate storage space, requesting allocation of the IP encoder to DataNode Master management node;

[0071] 所述Master管理节点根据第三策略为所述IP编码器分配所述DataNode ; [0071] The Master node to assign the management of the IP DataNode encoder according to a third strategy;

[0072] 其中,Master管理节点用于统一管理和分配DataNode。 [0072] wherein, Master node for managing the unified management and distribution DataNode.

[0073] 第三策略包括:根据各DataNode的剩余存储空间、剩余输入输出(1)处理能力,并基于负载均衡、网络直连(即尽量向IP编码器分配同一子网地址的存储资源)因素,为所述IP编码器分配DataNode。 [0073] The third strategy includes: The DataNode remaining storage space of each of the remaining input and output (1) processing capacity, and the load balancing, network attached (i.e., the same shall allocate to the IP subnet address encoder memory resources) based on factors , assigned to the IP DataNode encoder. 其中,各DataNode的剩余存储空间、剩余1处理能力可以由各DataNode分别上报给Master管理节点。 Wherein each DataNode the remaining storage space, a surplus processing capacity can be separately reported to each DataNode Master management node.

[0074] 通过以上本发明的方法实施例,在保持媒体流iSCSI直存的高性能优势的前提下,支持摄像机存储空间在存储设备集群任意一到多台设备上的分配;实现了存储空间的单元切分与弹性动态分配机制,满足告警联动存储、变码率存储等无法预估摄像机存储空间的场景需求,达到最大程度提高存储空间利用率的效果。 [0074] by the method of the above embodiment of the present invention, the media stream while maintaining a straight iSCSI storage performance advantages, the camera supports the storage device storage space assigned to the cluster on any one of multiple devices; to achieve a storage space segmentation and the elastic unit dynamic allocation, alarm linkage to meet storage, memory, etc. can not change rate estimate scene camera storage space requirements, to maximize the utilization of storage space results.

[0075] 优选的,所述Master管理节点可以包括主Master管理节点和辅Master管理节点,本发明实施例的方法进一步包括: [0075] Preferably, the Master node may include a management node managing the primary and secondary Master Master management node, the method further embodiment of the present invention comprises:

[0076] 所述辅Master管理节点用于对所述主Master管理节点进行冗余保护,在所述主Master管理节点故障时,所述辅Master管理节点切换为主Master管理节点。 [0076] The secondary management Master Master node for the master node for redundancy management, when the Master management master node fails, the secondary switch to the master node management Master Master management node.

[0077] 通过主辅Master管理节点的冗余保护,保障了Master管理节点不受故障影响从而提供正常服务。 [0077] Master primary and secondary protection through redundancy management nodes, protect against failure affecting the Master management node to provide normal service.

[0078] 优选的,本发明实施例的方法进一步包括:所述Master管理节点在所述DataNode故障或断网时,为所述故障或断网的DataNode所对应的IP编码器分配新的可用DataNode。 [0078] Preferably, the method of the present invention further embodiment comprises: the Master when the DataNode management node failure or disconnection, allocates a new DataNode is available DataNode failure or disconnection of the corresponding IP encoder .

[0079] 具体的,所述Master管理节点对各DataNode进行主动探测,以判断各DataNode是否工作正常,将探测失败的DataNode设为下线状态,将探测成功、但与IP编码器的通讯失败计数达到预设阈值的DataNode设为失效状态,并向系统管理员触发告警,进行人工干预,将探测成功、且与IP编码器的通讯失败计数未达到预设阈值的DataNode设为在线状态; [0079] Specifically, the management node for each Master DataNode actively probing, to determine whether each of the DataNode working properly, the probe set failed DataNode offline state, the probe was successful, but a communication failure count encoder IP reaches a preset threshold value is set DataNode failed state, a system administrator to trigger an alarm, manual intervention, the successful detection and communication with IP encoder failure count does not reach a preset threshold value is set DataNode online;

[0080] 相应的,所述Master管理节点是从在线状态的DataNode选择所述新的可用DataNode。 [0080] Accordingly, the Master of the new management node is selected from DataNode DataNode available online state.

[0081] 本发明实施例还提供了一种存储资源的管理和分配系统,如图4所示,主要包括:DataNodelO和IP编码器20,其中, [0081] Embodiments of the present invention further provides a system for the management and allocation of storage resources, shown in Figure 4, including: DataNodelO encoder 20 and IP, wherein

[0082] IP编码器20用于,向DataNodelO发送分配存储空间请求; [0082] IP encoder 20 for transmitting a request to allocate storage space DataNodelO;

[0083] DataNodelO用于,按照第一策略为所述IP编码器20选择存储资源,并根据第二策略为所述IP编码器20分配所选存储资源中的存储空间slice; [0083] DataNodelO for 20 selects the storage resources to IP encoder, and said slice encoder assigned IP 20 selected memory storage resource according to a second strategy according to the first policy;

[0084] IP编码器20还用于,根据所选存储资源的iSCSI挂载信息,通过iSCSI挂载存储资源,根据所分配的存储空间slice的起始地址、空间大小进行iSCSI数据写入。 [0084] IP encoder 20 is further configured, the mount information selected according to iSCSI storage resources, storage resources via iSCSI mount performs data writing according to the allocated iSCSI storage slice start address space.

[0085] 优选的,DataNodelO进一步用于,按照以下第一策略为所述IP编码器20选择存储资源: [0085] preferably, DataNodelO further configured to follow the first policy to the storage resource 20 selects the IP encoder:

[0086] 如果当前正在分配slice的存储资源上仍有空闲状态的slice,则继续选择当前存储资源;且, [0086] If the current storage resources are allocated slice on slice still idle, then continue to select the current storage resources; and,

[0087] 如果当前正在分配slice的存储资源的占用slice达到所述存储资源总slice的第一预设比例以上,则选择下一剩余存储空间最大的存储资源。 The first preset ratio [0087] If the current slice of the storage resources are allocated slice to a total occupation of the memory resource of the slice or higher, selects the next largest remaining storage space of the storage resource.

[0088] 优选的,DataNodelO进一步用于,对存储资源以slice为单位进行切分,并通过空闲slice队列和占用slice队列对存储资源中的slice进行管理; [0088] preferably, DataNodelO further configured to store resources be segmented in units of slice, and slice queue storage resources occupied in the free slice managed by a queue and slice;

[0089] DataNodelO根据以下第二策略分配slice:在所述IP编码器20请求的存储资源中选择空闲slice队列中的slice分配给所述IP编码器20,且在应答成功后,将本次分配的slice更新为占用状态并调度到占用队列中。 [0089] DataNodelO second policy assigned according to the following slice: IP assigned to the selected slice encoder slice idle queue 20 in the storage resources requested IP encoder 20, and after the success of the response, the distribution of this the occupancy status and updates the slice scheduling queue occupancy.

[0090] 优选的,IP编码器20进一步用于,在所分配slice的空间被写入比例达到第二预设比例时,向所述DataNode 1请求分配新的s I i ce; [0090] preferably, the IP further configured encoder 20 are written in the ratio reaches a second predetermined proportion of the time slice allocated space, allocate a new s I i ce DataNode 1 to the request;

[0091] 所述DataNodelO进一步用于,根据所述IP编码器20分配新si ice的请求,判断所述原slice所属的存储资源中slice的占用率是否达到第三预设比例,如果未达到第三预设比例,则继续为所述IP编码器20分配相同存储资源中的空闲slice,如果达到第三预设比例,则为所述IP编码器20选择所述DataNodelO的其他存储资源; [0091] The DataNodelO further configured, upon request of the allocation of IP encoder 20 new si ice, it is determined whether the original storage resource occupancy slice belongs slice reaches a third predetermined ratio, if this is not the first three preset ratio, the same allocation of storage resources to continue the free slice encoder 20 to the IP, if it reaches a third predetermined ratio, compared with the IP encoder 20 to select the other DataNodelO storage resource;

[0092] 如果所述DataNodelO的所有存储资源的slice占用率都达到所述第三预设比例,则删除所述DataNodelO中的老化数据并将所删数据对应的slice挂回到空闲slice队列,为所述IP编码器20选择剩余slice空间最大的存储资源,并分配所选存储资源的空闲slice给所述IP编码器20。 [0092] If all slice usage of the storage resources DataNodelO have reached the third predetermined ratio, and to delete the aging data corresponding to the deleted data slice DataNodelO hung back in the free slice queue, is the IP slice encoder 20 selects the remaining storage space of the largest resource, and allocates resources to the selected memory idle slice encoder 20 to the IP.

[0093] 优选的,该系统还包括:Master管理节点30, [0093] Preferably, the system further comprising: Master management node 30,

[0094] IP编码器20进一步用于,在向DataNodelO发送分配存储空间请求之前,向所述Master管理节点30请求分配DataNodelO; [0094] IP encoder 20 is further used for, before sending a request to allocate storage space DataNodelO, DataNodelO assigned to the Master management node 30 request;

[°°95] 所述Master管理节点30用于,在收到所述IP编码器20分配DataNode的请求后,根据第三策略为所述IP编码器20分配所述DataNodelO。 [°° 95] The management node 30 for the Master, after receiving the request for allocation of IP DataNode encoder 20, is assigned the IP DataNodelO the encoder 20 according to the third strategy.

[0096] 优选的,Master管理节点30进一步用于,根据以下第三策略为IP编码器20分配DataNodelO: [0096] preferably, Master management node 30 for a further 20 to assign IP DataNodelO third encoder according to the following strategy:

[0097] 根据各DataNodelO的剩余存储空间、剩余1处理能力,并基于负载均衡、网络直连因素,为所述IP编码器20分配DataNodelO。 [0097] The remaining storage space of each DataNodelO, the remaining processing capacity of 1, based on load balancing, network attached factors, encoder 20 to assign the IP DataNodelO.

[0098] 优选的,Master管理节点30包括主Master管理节点和辅Master管理节点; [0098] preferably, Master management node 30 includes a master node and secondary management Master Master management node;

[0099] 所述辅Master管理节点用于对所述主Master管理节点进行冗余保护,在所述主Master管理节点故障时,所述辅Master管理节点切换为主Master管理节点。 [0099] The secondary management Master Master node for the master node for redundancy management, when the Master management master node fails, the secondary switch to the master node management Master Master management node.

[0100] 优选的,Master管理节点30进一步用于,在所述DataNode 10故障或断网时,为所述故障或断网的DataNodelO所对应的IP编码器20分配新的可用DataNode。 [0100] preferably, Master management node 30 is further configured to, when the DataNode 10 failure or disconnection, the fault is DataNodelO or disconnection corresponding to the encoder 20 allocates a new IP available DataNode.

[0Ί0Ί] 具体的,所述Master管理节点30对各DataNodelO进行主动探测,以判断各DataNode是否工作正常,将探测失败的DataNode设为下线状态,将探测成功、但与IP编码器的通讯失败计数达到预设阈值的DataNode设为失效状态,并向系统管理员触发告警,进行人工干预,将探测成功、且与IP编码器的通讯失败计数未达到预设阈值的DataNode设为在线状态; [0Ί0Ί] Specifically, the Master management node 30 for each actively probing DataNodelO, to determine whether each of the DataNode working properly, the probe set failed DataNode offline state, the probe was successful, but the encoder IP communication failure the count reaches a preset threshold value is set DataNode failed state, a system administrator to trigger an alarm, manual intervention, the successful detection and communication with IP encoder failure count does not reach a preset threshold value is set DataNode online;

[0Ί02] 相应的,所述Master管理节点30是从在线状态的DataNode选择所述新的可用DataNode。 [0Ί02] Accordingly, the Master of the management node 30 is selected from the new available DataNode DataNode online state.

[0103] 下面再结合图4介绍一种包括数据管理节点DataNodelO、主(Master)管理节点30和IP编码器20的存储资源的管理和分配系统,其中, [0103] Next, FIG. 4 describes a combined data management node DataNodelO, a main storage resource (Master) and a management node 30 IP encoder management and distribution system 20, which comprises,

[0104] DataNodelO,用于管理其所属IPSAN存储设备的存储资源,通过发送注册、保活消息,向Master管理节点30上报DataNode 1所属IPSAN存储设备的剩余存储空间、剩余输入输出(10)处理能力; [0104] DataNodelO, (10) for storing the processing power of their respective resource management IPSAN storage device, by sending a registration, the keep-alive message, reports the remaining storage space IPSAN DataNode 1 belongs to the Master storage device management node 30, the remaining input and output ;

[0105] Master管理节点30,用于通过接收DataNodelO的注册、保活消息,获取并管理各IPSAN存储设备的剩余存储空间、剩余10处理能力;还用于在接收到来自IP编码器20的请求分配DataNode报文后,根据其管理的各IPSAN存储设备的剩余存储空间、剩余10处理能力,为IP编码器20选择DataNode ; [0105] Master management node 30, register for receiving by DataNodelO, the keep-alive message, acquire and manage the remaining storage space of each storage device IPSAN, 10 remaining processing capacity; for further 20 receives the request from the IP encoder after dispensing DataNode packet, according to the remaining storage space of each storage device IPSAN it manages, the processing capacity of the remaining 10, 20 selects the IP DataNode encoder;

[0106] IP编码器20,用于向Master管理节点30请求分配DataNode,并获取Master管理节点30为IP编码器20分配的DataNode的信息;还用于根据Master管理节点30分配的DataNode的信息,向相应的DataNodelO请求分配存储资源; [0106] IP encoder 20, 30 for the distribution request to the management node DataNode Master, Master and the management node 30 acquires the information of the DataNode allocated IP encoder 20; DataNode information for further distribution in accordance with the Master management node 30, requesting allocation of storage resources to the respective DataNodelO;

[0107] DataNodelO还用于,收到来自IP编码器20的分配存储资源的请求后,为IP编码器20进行存储资源的选择,并将所选存储资源的信息发送给IP编码器20。 [0107] DataNodelO further configured to receive a request from the IP allocation of storage resources encoder 20, the encoder is selected for the IP resource storage 20, and transmits the selected information to the IP storage resources encoder 20.

[0108] 优选的,DataNodelO进一步用于,对其所属IPSAN存储设备的存储资源以片(slice)为单位进行切分,并记录所有slice的所属存储资源标识、起始地址、空间大小、状态,所述状态包括空闲状态和占用状态;对每个存储资源的所有slice用空闲队列和占用队列进行管理,未被分配的slice为空闲状态并加入到空闲队列中,已被分配的slice为占用状态并调度到占用队列中。 [0108] preferably, DataNodelO further for their storage resources relevant to a memory device to IPSAN sheet (slice) as a unit be segmented, and identifying the storage resource records relevant to all of the slice, start address space, state, said state comprises an idle state and an occupied state; slice with the management of all the idle queue and queue occupancy for each storage resource, the slice is not allocated to the idle state and added to the free list have been allocated for the slice occupied state and scheduling queue occupancy.

[0109] 优选的,DataNodelO进一步用于,当其所属IPSAN存储设备配置了一个或多个存储资源时,按照以下原则为IP编码器20进行存储资源的选择: [0109] preferably, DataNodelO further configured, when it belongs IPSAN storage device configured with one or more storage resources, IP encoder 20 is a storage resource selection according to the following principles:

[0110] 如果当前正在分配s I i ce的存储资源上,仍有空闲状态的s I i ce,则继续选择当前存储资源;且, [0110] If the currently assigned storage resources s s I i ce, there are still idle I i ce, it continues to select the current storage resources; and,

[0111] 如果当前正在分配slice的存储资源的占用slice达到所述存储资源总slice的第一预设比例(如90%)以上,则选择下一剩余空间最大的存储资源。 The first predetermined ratio (e.g., 90%) [0111] If the current slice of the storage resources are allocated slice to a total occupation of the memory resource of the slice or higher, selects the next largest remaining space of storage resources.

[0112] 优选的,DataNodelO所选存储资源的信息包括:所选存储资源的唯一标识和iSCSI挂载信息; [0112] Preferably, the information storage resources selected DataNodelO comprising: a unique identifier and the selected iSCSI mount information storage resource;

[0113] IP编码器20进一步用于,在收到DataNodelO所选存储资源的信息之后,执行相应存储资源的iSCSI挂载,并在挂载成功后,在IP编码器20本地生成逻辑盘符,并向DataNodelO请求在所选存储资源中分配存储空间。 [0113] IP encoder 20 is further configured to, after receiving the information of the selected storage resources DataNodelO, the iSCSI mount respective storage resources, and after the success of the mount, the letter generation logic 20 in the local IP encoder, request to allocate storage space DataNodelO selected memory resource.

[0114] 优选的,DataNodelO进一步用于,在收到IP编码器20分配存储空间的请求后,根据所述请求中携带的存储资源标识,在对应存储资源的空闲slice队列中分配slice,并将所分配slice所属资源标识、所分配slice的起始地址、空间大小应答给IP编码器20;在应答成功之后,将本次分配的si ice更新为占用状态并调度到占用队列中。 [0114] preferably, DataNodelO further configured to, after receiving the request IP encoder 20 allocated in storage, according to the request carries the identifier storage resource, the free slice assignment slice in the corresponding storage queue resource, and slice belongs resource identifier allocated, the allocated slice start address, the response to the IP space encoder 20; after a successful response, this will be allocated to the occupied state si ice and update schedule queue occupancy.

[0115] 优选的,IP编码器20进一步用于,在收到DataNode 10分配si ice的应答后,根据所述应答中的资源标识获取对应的本地盘符,并按照分配sI ice的起始地址、空间大小,对流媒体块进行iSCSI直存写入为IP编码器20分配的存储空间。 [0115] Preferably, the encoder 20 the IP further configured to, after receiving the DataNode 10 si ice dispensing response, acquiring local drive letter corresponding to the identifier of the resource according to the response, and in accordance with the start address of the allocated sI ice , space, streaming media block is written as iSCSI storage linear encoder 20 IP allocated storage space.

[0116] 优选的,上述实施例中的Master管理节点30可以部署于数据管理(DM,DataManagement)服务器中,DataNode 1可以部署于IPSAN存储设备中,且所述系统中的多台IPSAN存储设备组成存储设备集群。 [0116] preferably, Master management node 30 of the above-described embodiments may be deployed in the data management (DM, DataManagement) server, DataNode 1 IPSAN can be deployed on a storage device, and the plurality of storage devices IPSAN system storage cluster.

[0117] 如图5所示,图5为本发明实施例的一种硬件组网示意图,在所述组网方案中,主要包括以下设备: [0117] As shown in FIG. 5, FIG. 5 A hardware embodiment of network schematic diagram of the present invention, the network programs, including the following equipment:

[0118] IP编码器,其中部署Client模块,支持对IP编码器各摄像机通道录像的iSCSI块存储。 [0118] IP encoder, wherein the Client module is deployed, iSCSI support block storage for each camera channel encoder IP video. 在本方案中,在IP编码器上部署Client模块,用于本方案的存储设备、存储空间的动态分配机制;Client模块用以实现上述图4所示系统中提及的IP编码器20的功能。 In the present embodiment, the encoder deployed on IP Client module for storing the apparatus of the present embodiment, the dynamic allocation of memory space; Client module for implementing the system shown in FIG. 4 mentioned above IP encoder function 20 .

[0119] DM服务器,其中部署Master管理节点,即Master管理节点部署在DM等管理服务器上。 [0119] DM server, wherein the management node deployment Master, Master i.e. deployed on the management node DM et management server. 一方面,通过接收IPSAN存储设备的注册、保活消息,获取管理本系统所有IPSAN存储设备的剩余存储空间、剩余10处理能力、IP地址、在线状态等信息;另一方面,在IP编码器的录像iSCSI存储流程中,基于负载均衡、网络直连等因素,给IP编码器分配存储录像的IPSAN设备。 In one aspect, the storage device register receiving IPSAN, keep-alive message, acquires the management of all storage devices IPSAN remaining storage space of the system, the processing capacity of the remaining 10, IP addresses, online status and other information; on the other hand, the encoder in the IP iSCSI storage recording process, based on load balancing, network attached and other factors, to the IP video encoder IPSAN allocate storage device.

[0120]另外,为提高Master管理节点服务的可靠性,支持如下高可用性机制: [0120] In addition, in order to improve the reliability of Master node management services to support high availability mechanisms as follows:

[0121] 在设备级,Master管理节点支持双机热备机制,实现主备冗余保护;在主用Master管理节点宕机之后,备用Master管理节点能够在短时间内切换为主用节点,向外部提供服务; [0121] In the device level, the Master management node supports dual-hot standby mechanism, achieve standby redundancy; after primary Master management node goes down, the backup management node Master node can be switched in a short time mainly used to external provision of services;

[0122] 在网络链路上,支持网络链路冗余机制;在主链路故障断开的情况下,可以通过备用链路,实现Master管理节点与外部设备之间的可靠网络连接。 [0122] on the network link, network link redundancy support; if the primary link fails open, the standby link can be, reliable network management node between the Master and the external devices.

[0123] IPSAN存储设备,其中部署DataNode。 [0123] IPSAN storage device, wherein deploying DataNode. 在本发明实施例的方案中,多台IPSAN存储设备组成了存储设备集群;同时,在本发明实施例的IPSAN存储设备上,部署DataNode。 In the solution of the embodiment of the present invention, a plurality of storage devices IPSAN cluster storage device; Meanwhile, in the storage device IPSAN embodiment of the present invention, deployment DataNode.

[0124] 一方面,DataNode通过向Mas ter管理节点进行注册、保活,上报DataNode所属IPSAN存储设备的剩余存储空间、剩余1处理能力、IP地址、在线状态等信息,支持Master管理节点对存储设备集群的统一调度管理; [0124] In one aspect, the DataNode by Mas ter registration management node, keep alive, reports the remaining storage space belongs DataNode IPSAN storage device, a remaining processing capacity, IP addresses, online status and other information, the management node supports Master storage device the unified management of the cluster;

[0125] 同时,DataNode支持对其所属IPSAN存储设备上所有存储资源,以slice为单位进行存储单元切分、分配、管理。 [0125] Meanwhile, DataNodes supports all its relevant IPSAN storage resources on storage devices in a storage unit in units of slice segmentation, distribution, management.

[0126] slice管理方式如下: [0126] slice managed as follows:

[0127] 在IPSAN存储设备创建任一存储资源后,DataNode自动按照一定空间单元粒度(比如256M字节),将存储资源空间切分为若干个slice JataNode记录本设备所有slice的所属存储资源标识、起始地址、空间大小、状态(空闲或者占用)等信息; [0127] After creating IPSAN storage device according to any one of storage resources, DataNodes automatically in a space unit size (such as 256M bytes), a memory resource space is divided into a plurality of cut slice JataNode storage resource records relevant to the present device identifying all of the slice, start address space, state (idle or occupied) and other information;

[0128] 对每一个存储资源的所有slice ,DataNode使用空闲队列、占用队列等两个队列来进行管理。 [0128] each slice for all storage resources, the idle queue DataNodes use, occupies two queues to manage the like. 新创建存储资源的所有slice,首先挂入空闲队列中;在收到IP编码器的存储空间申请请求后,DataNode从指定存储资源的空闲si ice队列中分配si ice给IP编码器,用于新录像的存储;在分配si ice成功之后,DataNode修改已分配的si ice状态为占用状态,并将已分配的slice从空闲队列调度到占用队列。 All newly created Slice storage resources, linked to the first idle queue; after receiving the application request IP encoders storage space, allocated si ice DataNodes si ice from the free queue specified storage resource to the IP encoder for new video storage; si ice success after dispensing, si ice DataNode modified state assigned to the occupied state, and the slice allocated to the idle queue scheduling queue occupancy.

[0129]由本发明实施例的上述存储资源的管理和分配系统实现的一种存储资源的管理和分配方法,包括: [0129] A method for the management and allocation of resources stored in the storage resources by the embodiment of the present invention to achieve management and distribution system, comprising:

[0130] DataNode管理其所属IPSAN存储设备的存储资源,通过发送注册、保活消息,向Master管理节点上报DataNode所属IPSAN存储设备的剩余存储空间、剩余1处理能力; [0130] DataNode manage the storage device to which it belongs IPSAN storage resources, by sending a registration, the keep-alive message, reports the remaining storage space DataNode IPSAN belongs to the Master storage device management node, the remaining processing capacity of 1;

[0131] Master管理节点通过接收DataNode的注册、保活消息,获取并管理各IPSAN存储设备的剩余存储空间、剩余1处理能力;并在接收到来自IP编码器的请求分配IPSAN存储设备的报文后,根据其管理的各IPSAN存储设备的剩余存储空间、剩余1处理能力,为IP编码器选择IPSAN存储设备; [0131] Master management node registered to receive DataNode, the keep-alive message, acquire and manage the remaining storage space of each IPSAN storage device, the remaining 1 processing capacity; and partitioned IPSAN storage device receiving a request from IP encoder packet after, according to the remaining storage space of each storage device IPSAN it manages, a remaining processing capacity, storage devices selected IPSAN IP encoder;

[0132] IP编码器根据Master管理节点分配的IPSAN存储设备的信息,向相应的DataNode请求分配存储资源; [0132] IP encoder IPSAN The information storage device management node Master assignment, requesting allocation of storage resources to the respective DataNodes;

[0133] DataNode收到来自IP编码器的分配存储资源的请求后,为IP编码器进行存储资源的选择,并将所选存储资源的信息发送给IP编码器。 After [0133] DataNode receives a request from the storage resources allocated IP encoder, storage resources selected for the IP encoder, and transmits the selected information to the IP storage resources encoder.

[0134] 优选的,该方法进一步包括: [0134] Preferably, the method further comprising:

[0135] DataNode对其所属IPSAN存储设备的存储资源以si ice为单位进行切分,并记录所有s I i ce的所属存储资源标识、起始地址、空间大小、状态,所述状态包括空闲状态和占用状态;对每个存储资源的所有slice用空闲队列和占用队列进行管理,未被分配的slice为空闲状态并加入到空闲队列中,已被分配的slice为占用状态并调度到占用队列中。 [0135] DataNode be segmented IPSAN their storage resources belong to si ice storage units, and records all relevant s I i ce of storage resource identifier, start address space, state, said state comprises an idle state and occupancy status; for all the slice with the idle queue occupancy and queue management for each storage resource, the slice is not allocated to the idle state and added to the free list have been allocated for the slice occupied state and an occupancy schedule queue .

[0136] 优选的,该方法进一步包括:当DataNode所属的IPSAN存储设备配置了多个存储资源时,按照以下原则为IP编码器进行存储资源的选择: [0136] Preferably, the method further comprising: when the memory device IPSAN DataNode belongs arranged plurality of storage resources, storage resources selected for the IP encoder according to the following principles:

[0137] 如果当前正在分配slice的存储资源上,仍有空闲状态的slice,则继续选择当前存储资源;且, [0137] If you are currently on the allocation of storage resources slice, slice still idle, then continue to select the current storage resources; and,

[0138] 如果当前正在分配slice的存储资源的占用slice达到存储资源总slice的第一预设比例(如90%)以上,则选择下一剩余空间最大的存储资源。 [0138] If the current slice of the storage resources are allocated occupancy reaches a first predetermined proportion of the total slice slice storage resources (e.g. 90%) or more, then selects the next largest remaining space of storage resources.

[0139] 优选的,DataNode所选存储资源的信息包括:所选存储资源的唯一标识和iSCSI挂载信息; [0139] Preferably, the information storage resources selected DataNode comprising: a unique identifier and the selected iSCSI mount information storage resource;

[0140] 该方法进一步包括:IP编码器在收到DataNode所选存储资源的信息之后,执行相应存储资源的iSCSI挂载,并在挂载成功后,在IP编码器本地生成逻辑盘符,并向DataNode请求在所选存储资源中分配存储空间。 [0140] The method further comprising: IP encoder after receiving the information of the selected storage resources DataNodes, the iSCSI mount respective storage resources, and after the success of the mount, the encoder in the IP local drive letter generation logic, and DataNode request to allocate storage space in the selected storage resources.

[0141] 优选的,该方法进一步包括: [0141] Preferably, the method further comprising:

[0142] DataNode在收到IP编码器分配存储空间的请求后,根据请求中携带的存储资源标识,在对应存储资源的空闲slice队列中分配slice,并将所分配slice所属资源标识、所分配slice的起始地址、空间大小应答给IP编码器;在应答成功之后,将本次分配的slice更新为占用状态并调度到占用队列中。 [0142] DataNode Upon request from the storage space allocated IP encoder, according to a request identifier carried in the storage resource, the free slice assignment queue corresponding to the slice in the storage resource, and the resource identifier belongs slice allocation, the allocated slice start address, the response to the IP space encoder; after answering the success of this time slice allocated and scheduled updates the occupancy state of the queue occupancy.

[0143] 优选的,该方法进一步包括: [0143] Preferably, the method further comprising:

[0144] IP编码器在收到DataNode分配slice的应答后,根据应答中的资源标识获取对应的本地盘符,并按照分配s I i ce的起始地址、空间大小,对流媒体块进行iSCSI直存写入为IP编码器分配的存储空间。 [0144] IP encoder slice allocation after receiving a response DataNode acquiring local drive letter corresponding to the response according to the resource identifier, a start address and I i ce of space, streaming media distribution according to a linear block iSCSI s write memory storage space allocated IP encoder.

[0145]结合图5所示的设备组网图,一种更为具体的存储资源的管理和分配方法实施例如图6所示,该方法主要包括: [0145] the networking apparatus shown in FIG. 5, a more specific method of managing and allocating storage resources embodiment as shown in FIG. 6, the method including:

[ΟΙ46] 步骤601,IP编码器在开始存储录像之前,通过其client模块向Master管理节点发送请求分配DataNode的报文。 [ΟΙ46] Step 601, IP encoder memory before starting the recording, sends a request message to the allocated DataNode Master management node through which the client module.

[0147] 步骤602,Master管理节点收到IP编码器的请求报文后,基于各DataNode在存储空间、存储输入输出(10,Input Output)能力的负载情况,选择合适的DataNode,并将所选DataNode的IP地址、通讯端口等信息,返回给IP编码器。 [0147] Step 602, after receiving the request management node Master IP packet encoder, based on the respective storage space in the DataNode storing input and output (10, Input Output) load capacity, to select the appropriate DataNode, and selected DataNode IP address information, and communications ports, back to the IP encoder.

[0148] 作为一种优选的实施方式,Master管理节点在选择DataNode时,还可同时考虑网络因素,尽量向同一IP编码器分配同一子网地址的IPSAN存储设备。 [0148] As a preferred embodiment, Master DataNodes management node in the choice, taking into account the network element may try to assign the same IPSAN storage device to the same IP subnet address encoder.

[0149] 步骤603,IP编码器的client模块收到应答之后,在本地文件中记录所分配的DataNode信息(包括DataNode的IP地址、通讯端口等),然后向所分配的DataNode发送请求分配存储资源的报文。 [0149] Step 603, after the encoder IP client module receives the response, the allocated DataNode information (including the IP address of the DataNode, communication ports, etc.) recorded in the local file, and then send the request to the allocated storage resources allocated DataNode message.

[0150] 步骤604,DataNode收到请求分配存储资源的报文之后,如果本存储设备配置了多个存储资源,则按照以下原则进行存储资源的选择: [0150] Step 604, DataNode allocation of storage resources after receiving the request packet, if the present storage device is configured with multiple storage resources, storage resources is selected in accordance with the following principles:

[0151] 如果当前正在分配slice的存储资源上,仍有空闲状态的slice,则继续选择当前存储资源;且, [0151] If you are currently on the allocation of storage resources slice, slice still idle state, continue to select the current storage resources; and,

[0Ί52] 如果当前正在分配slice的存储资源的占用slice达到所述存储资源总slice的第一预设比例(如90%)以上,则选择下一剩余空间最大的存储资源。 The first predetermined ratio (e.g., 90%) [0Ί52] If the current slice of the storage resources are allocated slice to a total occupation of the memory resource of the slice or higher, selects the next largest remaining space of storage resources.

[0153] DataNode完成存储资源选择之后,将所选择的存储资源的唯一标识、存储资源的iSCSI挂载信息(包括存储资源绑定的Target、LUN、具有资源写权限的initiator等信息)通过应答报文返回给IP编码器。 [0153] DataNode after the completion of the storage resource selection, the iSCSI storage resources that uniquely identifies the selected storage resource mounting information (including storage resource bundle Target, LUN, has write access to the resources of the initiator and other information) by reply IP packet back to the encoder.

[0154] 步骤605,IP编码器收到所分配的存储资源信息(存储资源的唯一标识、iSCSI挂载信息)之后,对存储资源执行iSCSI挂载,并在挂载成功后,在IP编码器本地生成逻辑盘符,如sda0 After [0154] Step 605, the encoder receives IP stores the allocated resource information (resource store the unique identifier, iSCSI mount information), storage resources for the iSCSI mount, and the mount is successful, the encoder in the IP locally generated logical drive letter, such as sda0

[0155] 步骤606,如果当前日期是第一次写入录像,则IP编码器向DataNode请求创建日期目录。 [0155] Step 606, if the current date is the first write-once recording, the DataNode IP encoder to request directory creation date.

[0156] 步骤607 ,DataNode创建对应的日期目录,并向IP编码器返回创建目录成功应答。 [0156] Step 607, DataNode creation date corresponding directory, and returns to create IP Encoder catalog success reply.

[0157] 步骤608,IP编码器向DataNode请求在存储资源中分配存储空间,请求报文中携带存储资源的唯一标识。 [0157] Step 608, IP encoder DataNode request to allocate storage space in the storage resource, the request carries the unique identification packet storage resources.

[0158] 步骤609,DataNode收到IP编码器的存储空间分配请求之后,通过存储资源的唯一标识,在对应存储资源的空闲slice队列中分配slice,并将该slice所属资源唯一标识、si ice的起始地址、空间大小等信息应答给IP编码器; [0158] Step 609, DataNode after receiving IP encoder memory allocation request, uniquely identifies a storage resource by allocating the slice corresponding to the free slice queue storage resources, and the resource that uniquely identifies the slice belongs, si ice of start address space size information response to the IP encoder;

[0159] 在应答成功之后,DataNode将本次分配的slice状态更新为占用状态,并将相应si ice挂入占用队列。 [0159] After a successful response, the slice DataNodes this allocation status is updated to the occupied state, and the corresponding si ice linked into the holding queue.

[0160] 步骤610,IP编码器收到DataNode的slice分配应答消息之后,通过存储资源唯一标识获取该资源所对应的本地盘符,按照分配的slice的起始地址、空间大小,以自定义的块数据格式进行iSCSI录像直存写入该逻辑盘。 [0160] Step 610, after the encoder receives slice allocated IP DataNode response message, acquires the resource uniquely identifies a corresponding local storage resources by letter in accordance with the start address of the slice allocated space to customize iSCSI recording format block data written to the logical disk direct memory. 数据将通过iSCSI传输通道,最终写入IPSAN存储设备的存储资源空间。 ISCSI data through a transmission channel, and ultimately into the storage space resources IPSAN storage device.

[0161] 步骤611,在所分配slice的空间被写入到第二预设比例(如90%)时,IP编码器向DataNode请求分配slice,请求报文中携带该slice所属存储资源的唯一标识;同时,IP编码器在原s I ice中继续新录像的存储。 When the unique identifier [0161] Step 611, the allocated space in the slice is written to a second predetermined ratio (e.g., 90%), IP encoder to DataNodes slice allocation request, the request packet belongs carried in the storage resource of the slice ; Meanwhile, IP in the original encoder s I ice continue recording a new storage.

[0162] 步骤612 ,DataNode在收到携带存储资源标识的si ice分配请求报文之后,首先执行存储资源的选择流程: [0162] Step 612, DataNode after receiving the carry storage resource identifier si ice allocation request packet, the execution selection process of storage resources:

[0163] 按照该存储资源(请求报文中携带的资源标识所对应的存储资源)的slice占用是否达到90%进行判断: [0163] According to the storage resource (memory resource request message carries resource identifier corresponding to) the slice reaches 90% occupancy judgment:

[0164] 如果占有率小于90 %,则继续分配该存储资源中的空闲si ice;否则,选择本IPSAN存储设备的其他存储资源; [0164] If the occupancy is less than 90%, the continued allocation of the idle si ice storage resources; otherwise, selecting another storage resource of the storage device of the present IPSAN;

[0165]如果本设备的所有存储资源的slice占用率都达到90%,则启动满覆盖策略,即删除一部分最老录像所在slice的内容,将其挂回空闲slice队列,然后选择剩余空间最大的存储资源,并分配该资源的空闲slice。 [0165] If the slice utilization of all storage resources of this equipment has reached 90%, then start full coverage strategy, namely to delete the contents of a part of the oldest slice whose recordings, hang it back to the free slice queue, and then select the maximum free space storage resources, and assign the free slice of the resource.

[0166] 在完成存储资源选择及空闲slice分配之后,DataNode向IP编码器进行应答: [0166] After completion of the selection and storage resources free slice assignment, DataNode respond to IP encoder:

[ΟΙ67] 如果是在同一存储资源中分配新的si ice,则DataNode返回新分配的si ice的起始地址、空间大小、slice所属存储资源标识等信息等给IP编码器; Start address [ΟΙ67] If it is assigned a new si ice in the same storage resource, the DataNode returns the newly allocated si ice, space, slice belongs to stored identification information and other resources to IP encoder;

[0168] 如果是在新的存储资源中分配slice给IP编码器,则在应答上述slice信息的同时,DataNode将存储资源的iSCSI挂载信息(包括存储资源绑定的Target、LUN、具有资源写权限的initiator等信息)也一起返回给IP编码器。 [0168] If the slice is allocated to the new IP encoder in storage resources, while the above-described slice response information, DataNodes the iSCSI mount information storage resource (memory resources bound to Target, LUN, write resources having initiator permission information) to be returned along with the IP encoder.

[0169] IP编码器如果收到新的存储资源的iSCSI挂载信息,则对存储资源执行本地iSCSI挂载,生成本地逻辑盘符。 [0169] IP if the encoder receives a new iSCSI mount information storage resources, perform the iSCSI mount local storage resources, generates local logical drive letter.

[0170] 步骤613,IP编码器在本地缓存中记录下一s I ice的起始地址、空间大小、si ice所属存储资源标识等信息,继续在当前slice中进行录像存储。 [0,170] Step 613, IP address of the next recording start encoder s I ice, the space, si ice storage resources relevant to the identification information in the local cache, to continue the current record is stored in the slice.

[0171] 步骤614,IP编码器在当前slice的全部空间写满之后,将该slice的录像开始时间、录像结束时间、摄像机标识等信息通知到Master管理节点。 [0171] Step 614, IP encoder after the current slice of all the space is full, the recording start time slice of end time of recording, the camera identification information such as notification to the Master management node.

[0172] 步骤615,Mas ter管理节点将s I i ce的录像开始时间、录像结束时间、摄像机标识等信息记录到数据库中,从而支持VM服务器通过数据库实现对录像的快速检索。 [0172] Step 615, Mas ter management node s I i ce of the recording start time, recording end time, camera identification information recorded in the database, the VM server to support video quick retrieval of a database.

[0173] 步骤616,IP编码器从本地缓存中取得下一slice信息、及其存储资源标识,取得该存储资源对应的本地逻辑盘符,在下一s I ice中继续新录像存储。 [0173] Step 616, IP encoder from the local cache to obtain the next slice information storage resource identifier and acquires the local logical drive letter corresponding to the storage resource, continue new video store in the next s I ice.

[0174] 还需要说明的是,本发明实施例中的DataNode宕机或者断网,造成IP编码器的录像存储、或者信令交互失败时,IP编码器可即时向Master管理节点请求分配DataNode,同时携带通讯失败的DataNode标识;Master管理节点将分配新的DataNode给IP编码器,保证IP编码器的录像存储不中断。 [0174] It is further noted that, in the embodiment DataNode down or off the network of the invention, resulting in IP video encoder memory, or when the signaling interaction fails, IP encoder may request immediate distribution to DataNode Master management node, communications failed while carrying DataNode identity; Master management node will assign a new IP DataNode to the encoder, video storage to ensure uninterrupted IP encoder.

[0175] 此时,Master管理节点触发到DataNode的主动探测机制,判断DataNode是否正常工作。 [0175] At this time, Master DataNode management node to trigger the active detection mechanism to determine DataNode is working properly. 同时,累计DataNode与不同IP编码器的通讯失败计数; Meanwhile, the cumulative DataNode different IP encoders correspond fail count;

[0176] 当DataNode探测失败时,Master管理节点将其置为下线状态,不再向IP编码器分配该DataNode。 [0176] When a probe fails DataNode, Master management node which is set to offline state, no longer assigned to the IP DataNode encoder. 后续只有DataNode重新向Mas ter管理节点注册成功,Master管理节点才会修改相应的DataNode状态为在线,并重新向IP编码器分配该DataNode ; Only DataNode subsequent re-registration success to Mas ter management node, Master management node will modify the state DataNode online, and reassign the DataNode to IP encoder;

[0177] 当DataNode探测成功、但是与不同IP编码器的通讯失败计数达到一定阈值时(比如:DataNode与Master管理节点之间的网络通讯正常,但是与IP编码器的网络断开),Master管理节点将其置为失效状态,不再向IP编码器分配该DataNode,并向系统管理员触发设备与IP编码器的通讯失败告警。 When [0177] When the probe DataNode successful, but reaches a certain threshold different IP communication failure count encoders (for example: DataNode between the Master and the communication network management node is normal, but disconnected from the IP network encoder), Master Management node to be set to a failed state, no longer assigned to the DataNode IP encoder, and trigger system administrator telecommunications equipment and IP encoder failure alarm. 这种情况下,只有管理员人工干预之后,通过管理界面将DataNode更新为在线状态,Master管理节点才会向IP编码器重新分配该DataNode。 In this case, only after the administrator manual intervention, through the management interface will DataNode updated online, Master management node will be reassigned to the DataNode IP encoder.

[0178]另外,在本发明实施例中增加IPSAN存储设备、或者IPSAN存储设备增加硬盘等情况下,只需要对新的IPSAN存储设备、新的硬盘空间创建新的存储资源,并加入DataNode的slice分配管理。 [0178] Further, in the case of this embodiment increases IPSAN storage apparatus according to the present invention, or IPSAN storage devices add a hard disk, only new IPSAN storage device, the new disk space to create a new storage resources, and added DataNode the slice distribution management. 由于新的存储资源空间是对所有摄像机共享使用,因此在无任何开销的情况下,自动支持了对所有摄像机的存储空间扩容。 Since the new storage space is a resource shared by all the cameras, and therefore without any overhead, the automatic support of the storage space for all cameras expansion.

[0179] 综上所述,本发明实施例提出的Master与DataNode两级管理、空间弹性分配的分布式iSCSI块直存集群方案,支持摄像机存储空间在存储设备集群中任意一到多台设备上的分配。 [0179] In summary, Master and two DataNode management provided by the embodiments of the present invention, the spatial distribution of the elastic blocks distributed direct deposit iSCSI cluster solution, to support a camera on any storage device to multiple storage devices in the cluster allocation. 同时,本发明实施例实现了iSCSI存储空间的slice单元切分管理、动态弹性分配机制,满足告警联动存储、变码率存储等无法预估摄像机存储空间的场景需求,达到最大程度上提高存储空间利用率的效果。 Meanwhile, embodiments of the present invention to achieve the slice management unit segmentation iSCSI storage space, dynamic allocation mechanism elasticity, alarm linkage to meet storage, memory, etc. can not change rate estimate scene camera storage space requirements, to achieve the maximum degree of improved storage the utilization of results.

[0180] 在Master管理节点故障时,系统自动切换为单存储设备节点存储模式,保障IP编码器的录像存储不停止。 [0180] When the Master management node failure, the system automatically switches to a single storage node mode of the device, to protect the encoder IP video storage does not stop. 同时,通过DataNode对s I i ce录像信息的本地缓存机制,保障s I i ce录像息不丢失。 Meanwhile, local cache mechanism for DataNode s I i ce video information and guarantee s I i ce video information is not lost.

[0181] 在任意IPSAN存储设备故障时,通过Master管理节点即时向IP编码器分配新的存储设备,保障IP编码器的录像存储不停止。 [0181] When any IPSAN storage device failure, instantly assigned a new storage device to the IP encoder by Master node management, protection of IP video encoder memory does not stop. 同时,实现对DataNode不同故障的错误恢复处理。 At the same time, realize the error of DataNode different fault recovery processing.

[0182] 在本发明所提供的几个实施例中,应该理解到,所揭露的方法、装置和电子设备,可以通过其它的方式实现。 [0182] The present invention provides several embodiments, it should be understood that the disclosed methods, apparatus, and an electronic device, may be implemented in other manners. 以上所描述的设备实施例仅仅是示意性的,例如,所述单元的划分,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式,如:多个单元或组件可以结合,或可以集成到另一个系统,或一些特征可以忽略,或不执行。 Example apparatus described above is merely exemplary, for example, the unit division is merely logical functions, can have other division, such as the actual implementation: a plurality of units or components may be combined, or It can be integrated into another system, or some features may be ignored or not performed. 另外,所显示或讨论的各组成部分相互之间的耦合、或直接耦合、或通信连接可以是通过一些接口,设备或单元的间接耦合或通信连接,可以是电性的、机械的或其它形式的。 Further, the various components shown or discussed coupling, or direct coupling between each other, or the communication connection may be indirectly coupled or communicating through some interfaces, devices or units may be electrical, mechanical, or other form of.

[0183]上述作为分离部件说明的单元可以是、或也可以不是物理上分开的,作为单元显示的部件可以是、或也可以不是物理单元,即可以位于一个地方,也可以分布到多个网络单元上;可以根据实际的需要选择其中的部分或全部单元来实现本实施例方案的目的。 [0183] As the above-described means may be separate components, or may not be physically separated, as the unit display member may be, or may not be physical units, i.e. may be located in one place and to be distributed to a plurality of networks unit; can select some or all of the units according to actual needs to achieve the object of the solutions of the embodiments.

[0184]另外,在本发明各实施例中的各功能单元可以全部集成在一个处理单元中,也可以是各单元分别单独作为一个单元,也可以两个或两个以上单元集成在一个单元中;上述集成的单元既可以采用硬件的形式实现,也可以采用硬件加软件功能单元的形式实现。 [0184] Further, in the embodiment of the present invention, each of the functional units may be all integrated in one processing unit, each unit may be separately as a unit, may be two or more units are integrated into one unit ; and the integrated unit may take the form of hardware, software plus hardware functional unit forms may also be implemented.

[0185] 本领域普通技术人员可以理解:实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成,前述的程序可以存储于一计算机可读取存储介质中,该程序在执行时,执行包括上述方法实施例的步骤;而前述的存储介质包括:移动存储设备、只读存储器(ROM,Read-0nly Memory)、随机存取存储器(RAM,Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。 [0185] Those of ordinary skill in the art will be understood: the hardware implementing the above method can be accomplished by a program instructing relevant to all or part of the steps embodiment, the program may be stored in a computer readable storage medium, the program execution when, comprising the step of performing the above-described embodiment of the method; and the storage medium comprising: a mobile storage device, a read only memory (ROM, Read-0nly memory), a random access memory (RAM, random access memory), a magnetic disk or optical disk various other media may store program codes.

[0186]或者,本发明实施例上述集成的单元如果以软件功能模块的形式实现并作为独立的产品销售或使用时,也可以存储在一个计算机可读取存储介质中。 [0186] Alternatively, the present invention is an integrated unit according to the above embodiment, and if implemented as an independent product sold or used in the form of a software functional module, may be stored in a computer-readable storage medium. 基于这样的理解,本发明实施例的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质中,包括若干指令用以使得一台计算机设备(可以是个人计算机、服务器、或者网络设备等)执行本发明各个实施例所述方法的全部或部分。 Based on such understanding, the technical solutions of the embodiments of the present invention essentially, or the part contributing to the prior art may be embodied in a software product, which computer software product is stored in a storage medium and includes several instructions to enable a computer device (may be a personal computer, a server, or network device) to execute all or part of the methods of the various embodiments of the present invention. 而前述的存储介质包括:移动存储设备、R0M、RAM、磁碟或者光盘等各种可以存储程序代码的介质。 The storage medium includes: any medium may store program codes removable storage devices, R0M, RAM, magnetic disk, or optical disk.

[0187]以上所述,仅为本发明的具体实施方式,但本发明的保护范围并不局限于此,任何熟悉本技术领域的技术人员在本发明揭露的技术范围内,可轻易想到变化或替换,都应涵盖在本发明的保护范围之内。 [0187] The above are only specific embodiments of the present invention, but the scope of the present invention is not limited thereto, any skilled in the art in the art within the technical scope of the present invention is disclosed, variations may readily occur or Alternatively, it shall fall within the protection scope of the present invention. 因此,本发明的保护范围应以所述权利要求的保护范围为准。 Accordingly, the scope of the present invention should be defined by the scope of the claims.

Claims (14)

1.一种存储资源的管理和分配方法,其特征在于,该方法包括: 数据管理节点DataNode接收IP编码器的分配存储空间请求; 所述DataNode按照第一策略为所述IP编码器选择存储资源,并根据第二策略为所述IP编码器分配所选存储资源中的存储空间片slice,以供所述IP编码器根据所选存储资源的iSCSI挂载信息,通过i SCSI挂载存储资源,根据所分配的存储空间sI ice的起始地址、空间大小进行iSCSI数据写入; 所述第一策略包括: 如果当前正在分配si ice的存储资源上仍有空闲状态的si ice,则继续选择当前存储资源;且, 如果当前正在分配slice的存储资源的占用slice达到所述存储资源总slice的第一预设比例以上,则选择下一剩余存储空间最大的存储资源。 1. A method of managing and allocating storage resources, wherein, the method comprising: allocating data storage space management node receives the IP encoder DataNode request; DataNode said selected memory resources for the encoder according to the first policy IP and allocating the selected memory slice sheet storage space resources of the IP encoder according to a second policy for the IP encoder mounting information selected according to iSCSI storage resources, storage resources by i SCSI mount, the starting address for the iSCSI data storage space allocated sI ice, the write space; the first policy comprises: if the current is still being dispensed si ice storage resources on the idle state si ice, then continue to select the current storage resource; and, if the currently assigned storage resources occupied slice slice reaches a first predetermined proportion of the total storage resources than the slice, it selects the next largest remaining memory storage resources.
2.根据权利要求1所述存储资源的管理和分配方法,其特征在于,该方法进一步包括:所述DataNode对存储资源以si ice为单位进行切分,并通过空闲si ice队列和占用si ice队列对存储资源中的slice进行管理; 所述第二策略包括:在所述IP编码器请求的存储资源中选择空闲slice队列中的slice分配给所述IP编码器,且在应答成功后,将本次分配的slice更新为占用状态并调度到占用队列中。 The management and storage resource allocation method according to claim 1, characterized in that, the method further comprising: the storage resources DataNode si ice in units of segmentation, and by the idle and the queue occupancy si ice si ice queue storage resources slice management; the second policy comprises: selecting slice after slice allocate idle queue storage resources of the IP encoder IP request to the encoder, and the response is successful, the slice allocation and scheduling updates the occupancy state of the queue occupancy.
3.根据权利要求1所述存储资源的管理和分配方法,其特征在于,该方法进一步包括:在收到所述IP编码器发送的分配新slice的请求时,所述DataNode根据所述IP编码器分配新si ice的请求,判断原si ice所属的存储资源中si ice的占用率是否达到第三预设比例,如果未达到第三预设比例,则继续为所述IP编码器分配相同存储资源中的空闲slice,如果达到第三预设比例,则为所述IP编码器选择所述DataNode的其他存储资源; 如果所述DataNode的所有存储资源的slice占用率都达到所述第三预设比例,则删除所述DataNode中的老化数据并将所删数据对应的si i ce挂回到空闲s I ice队列,为所述IP编码器选择剩余slice空间最大的存储资源,并分配所选存储资源的空闲slice给所述IP编码器; 其中,所述分配新slice的请求,是由所述IP编码器在所分配slice的空间被写入比例达到第二预设比例 The management and storage resource allocation method according to claim 1, characterized in that, the method further comprising: upon receipt of the request allocates a new slice of the transmitted IP encoder, the encoding based on the IP DataNode si ice allocates a new request, it is judged the original si ice storage resource belongs occupancy si ice has reached a third preset threshold, if the third predetermined ratio has not reached, continue to store the same assigned IP encoder Slice idle resources, if it reaches a third predetermined ratio, compared with the other IP encoder selects the storage resources of the DataNode; Slice occupancy rate of all storage resources if the DataNode has reached the third predetermined ratio, the age data and delete data in the DataNode corresponding to the deleted si i ce s I ice linked back to the idle queue, the remaining slice select maximum storage space resources of the IP encoder, and assign the selected storage slice of the idle resources to the IP encoder; wherein said allocation request of the new slice is encoded by the IP is written in the ratio reaches a second predetermined proportion of the space allocated slice ,向所述DataNode发送的。 , Is transmitted to the DataNode.
4.根据权利要求1至3任一项所述存储资源的管理和分配方法,其特征在于,该方法进一步包括: 在所述IP编码器向DataNode发送分配存储空间请求之前,所述IP编码器向Master管理节点请求分配DataNode; 所述Master管理节点根据第三策略为所述IP编码器分配所述DataNode。 The management and allocation methods 1 to 3 according to any one of the storage resources of the preceding claims, characterized in that, the method further comprising: prior to transmitting said IP encoder DataNodes request to allocate storage space, the encoder IP Master assignment request to the management node DataNode; the Master node to assign the management of the IP DataNode encoder according to the third strategy.
5.根据权利要求4所述存储资源的管理和分配方法,其特征在于,所述第三策略包括: 根据各DataNode的剩余存储空间、剩余输入输出1处理能力,并基于负载均衡、网络直连因素,为所述IP编码器分配DataNode。 The management and storage resource allocation method according to claim 4, wherein said third policy comprises: DataNode The remaining storage space of each of the remaining processing capacity of input and output 1, and based on load balancing, network attached factors assigned to said IP DataNode encoder.
6.根据权利要求4所述存储资源的管理和分配方法,其特征在于,所述Master管理节点包括主Master管理节点和辅Master管理节点,该方法进一步包括: 所述辅Master管理节点用于对所述主Master管理节点进行冗余保护,在所述主Master管理节点故障时,所述辅Master管理节点切换为主Master管理节点。 The management and storage resource allocation method according to claim 4, wherein the node includes a master management Master Master Master secondary management node and the management node, the method further comprising: the secondary node configured to manage Master the master node for redundancy management master, master management when the master node fails, the secondary switch to the master node management master master management node.
7.根据权利要求4所述存储资源的管理和分配方法,其特征在于,该方法进一步包括:所述Master管理节点在所述DataNode故障或断网时,为所述故障或断网的DataNode所对应的IP编码器分配新的可用DataNode。 7. The method of managing and allocating the storage resources according to claim 4, characterized in that, the method further comprising: the management node when the Master DataNode disconnection or failure, or failure of the broken network as DataNode corresponding encoder allocates a new IP available DataNode.
8.—种存储资源的管理和分配系统,其特征在于,该系统包括:1P编码器和数据管理节点DataNode,其中, 所述IP编码器用于,向所述DataNode发送分配存储空间请求; 所述DataNode用于,按照第一策略为所述IP编码器选择存储资源,并根据第二策略为所述IP编码器分配所选存储资源中的存储空间片slice; 所述IP编码器还用于,根据所选存储资源的iSCSI挂载信息,通过iSCSI挂载存储资源,根据所分配的存储空间slice的起始地址、空间大小进行iSCSI数据写入; 所述DataNode进一步用于,按照以下第一策略为所述IP编码器选择存储资源: 如果当前正在分配si ice的存储资源上仍有空闲状态的si ice,则继续选择当前存储资源;且, 如果当前正在分配slice的存储资源的占用slice达到所述存储资源总slice的第一预设比例以上,则选择下一剩余存储空间最大的存储资源。 8.- seed storage resource management and distribution system, wherein the system comprises: 1P encoder DataNodes and data management node, wherein the IP encoder for transmitting an allocation request to the storage space DataNodes; the DataNode for selecting resources for storing the IP encoder according to a first policy and the selected slice allocated storage space sheet storage resources of the IP encoder according to a second policy; IP said encoder is further configured to, the iSCSI mount the selected information storage resources, storage resources via iSCSI mount, according to the allocated memory space slice start address space sizes iSCSI data writing; the DataNode further configured first policy following IP is the storage resource selection encoder: if the current is still being dispensed si ice storage resources on the idle state si ice, then continue to select the current storage resources; and, if the allocation of storage resources currently occupied slice of slice reaches a the first slice of said predetermined proportion of the total resources than storage, then select the next largest remaining memory storage resources.
9.根据权利要求8所述存储资源的管理和分配系统,其特征在于,所述DataNode进一步用于,对存储资源以si ice为单位进行切分,并通过空闲si ice队列和占用si ice队列对存储资源中的slice进行管理; 所述DataNode根据以下第二策略分配slice:在所述IP编码器请求的存储资源中选择空闲slice队列中的slice分配给所述IP编码器,且在应答成功后,将本次分配的slice更新为占用状态并调度到占用队列中。 According to claim 8 storage resource management and distribution system, characterized in that, for the DataNode further, si ice storage resources in units of segmentation, and by the free si ice queues and queue occupancy si ice slice of storage resources management; the second policy based on the following assignment DataNode slice: the free slice assignment selected slice to the IP queue storage resources of the encoder in encoder IP request, and in response success after this time the slice allocation and scheduling updates the occupancy state of the queue occupancy.
10.根据权利要求8所述存储资源的管理和分配系统,其特征在于,所述IP编码器进一步用于,在所分配slice的空间被写入比例达到第二预设比例时,向所述DataNode请求分配新的slice; 所述DataNode进一步用于,根据所述IP编码器分配新si ice的请求,判断原si ice所属的存储资源中slice的占用率是否达到第三预设比例,如果未达到第三预设比例,则继续为所述IP编码器分配相同存储资源中的空闲slice,如果达到第三预设比例,则为所述IP编码器选择所述DataNode的其他存储资源; 如果所述DataNode的所有存储资源的slice占用率都达到所述第三预设比例,则删除所述DataNode中的老化数据并将所删数据对应的si i ce挂回到空闲s I ice队列,为所述IP编码器选择剩余slice空间最大的存储资源,并分配所选存储资源的空闲slice给所述IP编码器。 10. The memory according to claim 8 resource management and distribution system, characterized in that said encoder further configured IP is written in the ratio reaches a second predetermined proportion of the space slice allocated to the DataNode requesting allocation of a new slice; the DataNode further configured to request allocation based on the new IP si ice coder determines whether the occupancy rate of the original si ice storage resource belongs slice reaches a third preset proportion, if not reaches the third preset ratio, the same allocation of storage resources to continue the free slice encoder to the IP, if it reaches a third predetermined ratio, compared with the encoder selects the IP DataNode other storage resource; if slice occupancy rate of all of said storage resources DataNode has reached the third predetermined ratio, and to delete the aging data in the DataNode deleted data corresponding to si i ce s I ice linked back to the idle queue, as the select the remaining slice maximum space of said storage resource IP encoder, and assign the selected slice of storage resources to the idle IP encoder.
11.根据权利要求8至10任一项所述存储资源的管理和分配系统,其特征在于,该系统还包括:Master管理节点; 所述IP编码器进一步用于,在向所述DataNode发送分配存储空间请求之前,向所述Master管理节点请求分配DataNode; 所述Master管理节点用于,在收到所述IP编码器分配DataNode的请求后,根据第三策略为所述IP编码器分配所述DataNode。 11. The said one storage resource management and distribution system of any one of claims 8 to 10, characterized in that the system further comprises: Master management node; IP said encoder is further configured to transmit the allocation to the DataNode before storage request, the request to the management node dispensing Master DataNode; management node for the Master, after receiving the request DataNode encoder IP assignment, the assigned IP encoder according to the third policy DataNode.
12.根据权利要求11所述存储资源的管理和分配系统,其特征在于,所述Master管理节点进一步用于,根据以下第三策略为IP编码器分配DataNode: 根据各DataNode的剩余存储空间、剩余输入输出1处理能力,并基于负载均衡、网络直连因素,为所述IP编码器分配DataNode。 11 according to the storage resource management and distribution system of claim wherein said Master management node is further configured to assign IP DataNode third encoder according to the following strategy: The DataNode remaining storage space of each of the remaining an input-output processing capacity, and based on load balancing, network attached factors assigned to said IP DataNode encoder.
13.根据权利要求11所述存储资源的管理和分配系统,其特征在于,所述Master管理节点包括主Master管理节点和辅Master管理节点; 所述辅Master管理节点用于对所述主Master管理节点进行冗余保护,在所述主Master管理节点故障时,所述辅Master管理节点切换为主Master管理节点。 11 13. The storage resource management and distribution system according to claim, characterized in that the node includes a master management Master Master Master secondary management node and management node; the secondary node management Master Master for the primary manager node redundancy, when the master management master node fails, the secondary switch to the master node management master master management node.
14.根据权利要求11所述存储资源的管理和分配系统,其特征在于,所述Master管理节点进一步用于,在所述DataNode故障或断网时,为所述故障或断网的DataNode所对应的IP编码器分配新的可用DataNode。 11 14. The storage resource management and distribution system of claim wherein said Master management node is further configured to, when the DataNode disconnection or failure, or failure of the DataNode broken network corresponding the encoder allocates a new IP available DataNode.
CN201410245074.1A 2014-06-03 2014-06-03 Management and distribution method and system for storage resources CN104023058B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201410245074.1A CN104023058B (en) 2014-06-03 2014-06-03 Management and distribution method and system for storage resources

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201410245074.1A CN104023058B (en) 2014-06-03 2014-06-03 Management and distribution method and system for storage resources

Publications (2)

Publication Number Publication Date
CN104023058A CN104023058A (en) 2014-09-03
CN104023058B true CN104023058B (en) 2017-05-17

Family

ID=51439627

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201410245074.1A CN104023058B (en) 2014-06-03 2014-06-03 Management and distribution method and system for storage resources

Country Status (1)

Country Link
CN (1) CN104023058B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105898197A (en) * 2014-12-03 2016-08-24 北京航天长峰科技工业集团有限公司 Multi-code-stream video data direct storage method
CN104615508A (en) * 2015-03-03 2015-05-13 浪潮电子信息产业股份有限公司 Method for recovering LVM configuration under Linux system
CN104850469A (en) * 2015-05-12 2015-08-19 浪潮电子信息产业股份有限公司 LV mirror image-based method for realizing data backup, recovery and migration in linux system
CN107087031A (en) * 2017-05-10 2017-08-22 浙江宇视科技有限公司 Method and device for storing resources and balancing loads

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101176074B (en) * 2005-12-09 2010-12-15 松下电器产业株式会社 Nonvolatile memory device, method of writing data, and method of reading out data
CN101232521A (en) * 2008-02-28 2008-07-30 杭州华三通信技术有限公司 Video storing method, coding apparatus and storing device
CN101582903B (en) * 2009-06-24 2012-08-08 浙江宇视科技有限公司 Storing method of streaming media resource and device
CN101599978B (en) * 2009-07-17 2012-09-26 杭州华三通信技术有限公司 Method and equipment for allocating resources
CN102255921B (en) * 2011-08-24 2013-11-27 杭州华三通信技术有限公司 Media stream storage method and data management server

Also Published As

Publication number Publication date
CN104023058A (en) 2014-09-03

Similar Documents

Publication Publication Date Title
US7822862B2 (en) Method of satisfying a demand on a network for a network resource
US10057109B2 (en) Defining interdependent virtualized network functions for service level orchestration
CA2697540C (en) Executing programs based on user-specified constraints
US8874749B1 (en) Network fragmentation and virtual machine migration in a scalable cloud computing environment
US20070083723A1 (en) Highly-available blade-based distributed computing system
US9781486B2 (en) RS-DVR systems and methods for unavailable bitrate signaling and edge recording
US8321862B2 (en) System for migrating a virtual machine and resource usage data to a chosen target host based on a migration policy
US7328223B2 (en) Storage management system and method
US8914513B2 (en) Hierarchical defragmentation of resources in data centers
US20120150930A1 (en) Cloud storage and method for managing the same
US20050076336A1 (en) Method and apparatus for scheduling resources on a switched underlay network
KR20040071187A (en) Managing storage resources attached to a data network
JP2009048607A (en) Storage and server provisioning for virtualized and geographically distributed data centers
US7779368B2 (en) Systems and methods for presenting managed data
WO2012152153A1 (en) Method, monitor and system for seamless migration of virtual machine
EP2922238B1 (en) Resource allocation method
US7523273B2 (en) Autonomic storage provisioning to enhance storage virtualization infrastructure availability
JPWO2005001700A1 (en) Storage capacity management method, the server, a recording medium
US20050210137A1 (en) Method and apparatus for storage network management
CN101877783B (en) Network video recorder clustering video monitoring system and method
US20140223014A1 (en) Collaborative management of shared resources
RU2585981C2 (en) Large-scale data storage system
CN1766851A (en) Systems and methods for data storage management
JP2004508616A (en) Control method and apparatus for extensible computing system
KR20080096547A (en) Virtual network storage system, network storage device and virtual method

Legal Events

Date Code Title Description
C06 Publication
C10 Entry into substantive examination
GR01