Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/907—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
  • G06F16/41—Indexing; Data structures therefor; Storage structures
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
  • G06F12/02—Addressing or allocation; Relocation
  • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
  • G06F12/10—Address translation
  • G06F12/1027—Address translation using associative or pseudo-associative address translation means, e.g. translation look-aside buffer [TLB]
  • G06F12/1045—Address translation using associative or pseudo-associative address translation means, e.g. translation look-aside buffer [TLB] associated with a data cache
  • G06F12/1054—Address translation using associative or pseudo-associative address translation means, e.g. translation look-aside buffer [TLB] associated with a data cache the data cache being concurrently physically addressed
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F15/00—Digital computers in general; Data processing equipment in general
  • G06F15/16—Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs
  • G06F15/163—Interprocessor communication
  • G06F15/173—Interprocessor communication using an interconnection network, e.g. matrix, shuffle, pyramid, star, snowflake
  • G06F15/17306—Intercommunication techniques
  • G06F15/17331—Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
  • G06F16/38—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/383—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
  • G06F16/38—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/387—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
  • G06F16/48—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/483—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
  • G06F16/68—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/683—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
  • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/901—Indexing; Data structures therefor; Storage structures
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/903—Querying
  • G06F16/9035—Filtering based on additional data, e.g. user or group profiles
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/90—Details of database functions independent of the retrieved data types
  • G06F16/903—Querying
  • G06F16/9038—Presentation of query results
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
  • G06F3/0604—Improving or facilitating administration, e.g. storage management
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
  • G06F3/0638—Organizing or formatting or addressing of data
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
  • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
  • G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
  • G06F3/0671—In-line storage system
  • G06F3/0673—Single storage device
  • G06F3/068—Hybrid storage device
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
  • G06F3/0671—In-line storage system
  • G06F3/0683—Plurality of storage devices
  • G06F3/0688—Non-volatile semiconductor memory arrays
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/096—Transfer learning
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L49/00—Packet switching elements
  • H04L49/90—Buffering arrangements
  • H04L49/901—Buffering arrangements using storage descriptor, e.g. read or write pointers
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L67/00—Network arrangements or protocols for supporting network services or applications
  • H04L67/01—Protocols
  • H04L67/10—Protocols in which an application is distributed across nodes in the network
  • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
  • G06F2212/25—Using a specific main memory architecture
  • G06F2212/254—Distributed memory

Definitions

• the present disclosure relates generally to processing of media objects, and particularly to methods and systems for generation of metadata for media objects.
• An embodiment that is described herein provides a controller for use in a storage device of a data processing system.
• the controller includes a host interface, a memory interface and one or more processors.
• the host interface is configured to communicate over a computer network with one or more remote hosts of a data processing system.
• the memory interface is configured to communicate locally with a non-volatile memory of the storage device.
• the one or more processors are configured to manage local storage or retrieval of media objects at the non-volatile memory, and to selectively compute metadata that defines content characteristics of media objects that are stored, or that are to be stored, in the non-volatile memory.
• the one or more processors are configured to generate the metadata for a plurality of unstructured media objects that that are stored, or that are to be stored, in the non-volatile memory.
• the one or more processors are configured to receive from the one or more hosts, over the computer network, a model that specifies extraction of the metadata from the media objects, to generate the metadata based on the received model, and to store the generated metadata locally in the non-volatile memory, for use by the one or more hosts.
• the one or more processors are configured to receive, as the model, a pre-trained Artificial Intelligence (AI) model, and to generate the metadata by applying an AI inference engine to the AI model.
• AI Artificial Intelligence
• the one or more processors are configured to assign a first priority to management of the storage or retrieval of the objects at the non-volatile memory, and to assign a second priority, lower than the first priority, to generation of the metadata.
• the one or more processors are configured to generate the metadata during idle periods during which at least some resources of the one or more processors are free from managing storage of the objects.
• the one or more processors are configured to identify the idle periods, to initiate or resume generation of the metadata during the identified idle periods, and to suspend the generation of the metadata outside the identified idle periods.
• the one or more processors are configured to make the metadata accessible to the one or more hosts, the one or more hosts being situated at a location that is remote from the processor, over the computer network.
• the one or more processors are configured to make the metadata accessible by transmitting the metadata, but less than all of the media objects from which the metadata was generated, over the computer network.
• the one or more processors are configured to send one or more of the media objects over the computer network in response to a request from one or more of the hosts.
• the storage device includes a Solid State Drive (SSD), and the non-volatile memory includes one or more Flash memory devices.
• the storage device includes a Hard Disk Drive (HDD), and the non-volatile memory includes one or more magnetic disks.
• SSD Solid State Drive
• HDD Hard Disk Drive
• a method for generating metadata that defines content characteristics of media objects includes, by a host interface coupled to a storage device situated in a data processing center, communicating over a computer network with one or more remote hosts of the data processing center, and by a memory interface of the storage device, communicating locally with a non-volatile memory of the storage device.
• Local storage or retrieval of media objects at the non-volatile memory is managed by one or more processors of storage device.
• Metadata which defines content characteristics of media objects that are stored, or that are to be stored, in the non-volatile memory, is selectively computed by the one or more processors of the storage device.
• a system including multiple storage devices and one or more hosts.
• Ones of the storage devices respectively includes a non-volatile memory and a controller.
• the controller is configured to manage local storage or retrieval of objects at the non-volatile memory, and to selectively compute metadata that defines content characteristics of media objects that are stored, or that are to be stored, in the non-volatile memory.
• the one or more hosts are configured to retrieve the metadata generated by the multiple storage devices, and to perform a data processing operation on the media objects using the retrieved metadata.
• FIG. 1 is a block diagram that schematically illustrates a data center that performs metadata generation at the storage edge, in accordance with an embodiment that is described herein;
• FIG. 2 is a block diagram that schematically illustrates an edge appliance that performs local metadata generation, in accordance with an embodiment that is described herein;
• FIG. 3 is a flow chart that schematically illustrates a method for combined storage/retrieval and metadata generation in a Solid State Drive (SSD) controller, in accordance with an embodiment that is described herein; and
• Fig. 4 is a block diagram that schematically illustrates an SSD, in accordance with an embodiment that is described herein.
• Embodiments that are described herein provide improved methods and systems for generating metadata for media objects, and for storing and using such metadata, in data processing systems.
• a data processing system is used for storing and analyzing a large volume of content data contained in media objects.
• object analysis applications include identification of persons of interest or other objects in video footage of security cameras, customized insertion of advertisements (“ads”) into streamed videos, analytics of data from autonomous vehicles, analytics of call and response quality in a ChatBot Voice calls data base, text documents and/or text messages database analysis, mood detection, scene identification within a video file or Voice call, identification of persons or objects in surveillance footage, identification of types of actions occurring in surveillance footage, identification of voices or types of sounds in recordings, classification of phrases and/or responses used during conversation, analysis of automotive sensor data and driving responses, and many others.
• Examples of media objects include videos, sound recordings, still images, textual objects such as text messages and e-mails, data obtained from various types of sensors such as automotive sensors and Intemet-of-Things (IoT) sensors, database objects, and/or any other suitable objects.
• Media objects are also referred to herein simply as“objects” for brevity.
• the system generates metadata for the media objects and performs the desired analysis, or executes an action, based on the metadata.
• the system would typically analyze each streamed video, divide the video into scenes, estimate a sentiment associated with each scene, people and/or objects in the scene, identify the context of speech in the scene, and store the estimated scene-sentiments and other information as metadata associated with the video. Based on this metadata, the system can then choose where in the video to insert a given ad in order to maximize the ad’s impact.
• a data processing system is used for off-line analysis of data acquired by cameras and other sensors of an autonomous car.
• the system may scan the video acquired by the car cameras and/or outputs of other car sensors, identify events that are pre-classified by the AI model as being of interest, and tag them as metadata. The system can then use the metadata to investigate the identified events.
• Metadata is generated on the storage side of a vehicle that is periodically connected to a network.
• the metadata is used to select portions of relevant sensor data to be at least temporarily stored at a storage device in the vehicle, or discarded.
• Metadata along with selected relevant objects are then periodically uploaded over a network connection to a central processor where the metadata objects are analyzed and can be applied for various purposes such as improving the behavior of autonomous vehicles, or determining targeted advertisements that are to be conveyed to users of the vehicle.
• the media objects are unstructured.
• the term“unstructured object” means that the media content of the object (e.g., textual content, audio content, image content or video content) is provided is raw form and is not organized in advance according to a fixed field format.
• an unstructured object is not tagged a- priori with metadata that defines any aspects of the content per frame or other content portion.
• unstructured data is non-transactional, and its format does not readily conform to a relational database schema.
• Metadata can be generated from media objects, which are almost always unstructured, in various ways.
• An Artificial Intelligence (AI) model e.g., a neural network.
• an AI model is trained using a“training set” - a body of media objects and corresponding metadata that is known to be accurate. The trained model is then applied to generate metadata for other media objects.
• a software or hardware module that receives a pre-trained AI model and uses it to compute metadata of objects is referred to herein as an“AI inference engine.”
• an“AI inference engine” A software or hardware module that receives a pre-trained AI model and uses it to compute metadata of objects.
• several different AI models will be applied to unstructured or partially structured media objects.
• the embodiments described herein dramatically improve the metadata generation process, by performing this task at the storage edge, close to the locations at which the objects are stored, rather than at one centralized location which conventionally requires the transport of a very large body of media object data through a network between storage and processing devices.
• the term“at the storage edge” means at a processor or processors that are located on the storage-device side of the network, as opposed to the host- side of the network.
• a processor at the storage edge needs to send and/or receive information across the network in order to communicate with the hosts, but does not need to send and/or receive information across the network in order to communicate with the storage device.
• Non-limiting examples of storage-edge processors include Solid State Drive (SSD) controllers, processors is storage-array controllers (also referred to as aggregators), and processors in connected storage-edge appliances such as autonomous vehicles.
• SSD Solid State Drive
• the data processing system comprises one or more host servers (“hosts”) that analyze media objects, and one or more storage devices (e.g., Solid State Drives - SSDs) in which the media objects are stored.
• hosts host servers
• storage devices e.g., Solid State Drives - SSDs
• the hosts and the storage devices communicate over a computer network.
• Each storage device comprises a Non-Volatile Memory (NVM) and a controller.
• NVM Non-Volatile Memory
• controller devices that aggregate storage among one or more storage devices also include a controller.
• controllers are configured to both (i) manage local storage or retrieval of objects in the NVM, and (ii) selectively compute metadata that defines content characteristics of the media objects that are stored, or about to be stored, in the NVM, thereby leveraging controllers at the storage edge as a resource for computing metadata on unstructured or partially structured media objects.
• the controller of each storage device comprises its own local AI inference engine.
• the controller receives from the hosts a pre-trained AI model, compiles the AI model, and generates metadata for media objects by running the compiled AI model on the local AI inference engine.
• the controller of each storage device maintains a local metadata database, which holds the metadata generated for the media objects stored locally on that storage device.
• the local metadata databases of the various storage devices are accessible to the hosts over the computer network, for performing any desired object analysis task.
• metadata is stored in association with, but separately from, the media objects such that metadata can be retrieved separately from the media objects.
• storage of the media objects and their respective metadata is in the same storage device, but in other embodiments the metadata is stored in a separate storage device from the media objects, for instance at a remote location that is accessible over a computer network.
• the controller of a storage device gives higher priority to data storage and retrieval tasks, and lower priority to metadata generation.
• the controller identifies idle periods, in which at least some controller resources are free of data storage and retrieval tasks.
• the controller uses such idle periods to perform metadata generation. In this manner, the performance of the storage device is not degraded by the additional metadata generation task of the controller.
• the disclosed techniques eliminate the need to transport media objects across the network for the sake of computing the metadata.
• most if not all of the media objects remain within the boundaries of the storage devices, and it is the metadata that is provided to the hosts for analysis.
• the hosts are able to select which specific media object, or portions thereof, if at all, need to be retrieved over the network. As such, traffic overhead over the computer network is reduced considerably.
• the disclosed techniques also reduce latency, e.g., because they require less data movement and since they enable distributed metadata generation in parallel by multiple storage-device controllers. Since the disclosed solution lends itself to distributed implementation across multiple storage devices, the solution is highly scalable. Since the disclosed technique improves metadata generation speed, it responds rapidly to updates of media objects.
• Fig. l is a block diagram that schematically illustrates a data processing system 20, in the present example a data center, which performs metadata generation at the storage edge, in accordance with an embodiment that is described herein.
• System 20 is used for storing a large number of objects, calculating metadata for the objects, and analyzing the objects based on the metadata, as explained above.
• system 20 receives, stores and analyzes objects of multiple different types.
• data processing system 20 comprises one or more host servers 24 that communicate over a computer network 28.
• Host servers 24 are also referred to herein simply as“hosts” for brevity.
• Computer network 28 may comprise any suitable type of network, e.g., a Local Area Network (LAN), Wide Area Network (WAN), cloud network, data center network or the like.
• system 20 comprises one or more media generators 30 that generate the media objects being stored and analyzed.
• System 20 further comprises a remote storage rack 32, which is used by hosts 24 for storing objects, metadata and other relevant data.
• storage rack 32 is part of a Storage Area Network (SAN) that communicates with network 28.
• Hosts 24 communicate with storage rack 32 over network 28 for storing and retrieving data.
• SAN Storage Area Network
• storage rack 32 comprises multiple storage units referred to as All-Flash Arrays (AFAs) 36.
• AFAs All-Flash Arrays
• a Top-Of-Rack (TOR) switch 40 manages the communication between AFAs 36 and hosts 24 over network 28.
• storage rack 32 including its various components is regarded as located“at the storage edge” of system 20.
• AFA 36 comprises multiple Solid State Drives (SSDs) 44 in which the data (e.g., objects and metadata) is stored.
• AFA 36 comprises a storage controller 50, which is configured to manage storage of data (e.g., media objects). Storage controller 50 is also referred to herein as an aggregation processor or aggregator.
• AFA 36 further comprises a switch 48 that is configured to communicate with TOR switch 40 over a suitable network cable 42.
• switch 48 communicates with SSDs 44 over a common Peripheral Component Interconnect Express (PCIe) bus, e.g., using the Non-Volatile Memory Express (NVMe) protocol.
• PCIe Peripheral Component Interconnect Express
• hosts 24 communicate with SSDs 44 via switch 48 using Ethernet, e.g., using the NVMe-over-fabrics protocol.
• Ethernet e.g., using the NVMe-over-fabrics protocol.
• other suitable protocols can be used.
• Communication between switch 48 and TOR switch 40 is typically conducted using Ethernet.
• AFA 36 comprises a Central Processor ETnit (CPET) and/or Network Interface Controller (NIC), not shown in the figure, for communicating with TOR switch 40.
• CPET Central Processor ETnit
• NIC Network Interface Controller
• each SSD 44 comprises a plurality of Flash memory devices 52, e.g., NAND Flash memories, and an SSD controller 56.
• SSD controller 56 comprises a memory interface 60 for communicating with Flash devices 52, a host interface 68 for communicating with hosts 24 (via switch 48 and TOR switch 40), and a processor 64.
• Host interface 68 may communicate with hosts 24 using any suitable storage protocol, for example Non-Volatile Memory Express (NVMe) or Serial Advanced Technology Attachment (SATA).
• NVMe Non-Volatile Memory Express
• SATA Serial Advanced Technology Attachment
• processor 64 performs both storage/retrieval tasks and metadata computation tasks.
• metadata computation tasks include identification, description and/or tagging of objects, activities, scene characteristics and other features of content within media objects.
• metadata computation tasks are performed by storage controller (aggregator) 50.
• processor 64 comprises a Flash management module 72 and an Artificial Intelligence (AI) inference engine 76.
• Flash management module 72 is configured to store and retrieve data (e.g., objects and metadata) in Flash devices 52.
• the tasks of Flash management module 72 are also referred to as“Flash Translation Layer” (FTL).
• AI inference engine 76 is configured to compute metadata for media objects, as explained below.
• SSD controller 56 stores in Flash devices 52 (i) media objects 80 and (ii) a metadata database 84 that holds the metadata of media objects 80.
• the SSD configuration of Fig. 1 is a non-limiting example configuration, and that any other suitable SSD controller can be used in alternative embodiments.
• the SSD controller comprises multiple Integrated Circuits (ICs) that communicate with one another via a suitable fabric, is described in Fig. 4 below.
• FIG. 2 is a block diagram that schematically illustrates a data processing system 90 in which an edge appliance 94 performs local metadata generation, in accordance with an embodiment that is described herein.
• Edge appliance 94 comprises, for example, an autonomous car, a surveillance box, an IoT device, or any other suitable type of edge device, in an embodiment.
• edge appliance 94 communicates with a cloud-based data center 102 via a computer network, in the present example the Internet.
• Data center 102 comprises one or more hosts (not shown).
• Edge appliance 94 comprises a Central Processing Linit (CPLi) cluster 106, a local memory 110 (typically a Random Access Memory (RAM) or other volatile memory), and an SSD 114.
• CPLi Central Processing Linit
• RAM Random Access Memory
• SSD 114 typically a Random Access Memory
• the internal structure of SSD 114 in an embodiment, is shown in an inset on the right-hand side of the figure.
• the structure of SSD 114 is similar to that of SSD 44 of Fig. 1.
• FIG. 3 is a flow chart that schematically illustrates a method for combined storage/retrieval and metadata generation, carried out by SSD controller 56, in accordance with an embodiment that is described herein.
• a storage/retrieval process 120 carried out by Flash management module 72 in processor 64 of SSD controller 56, is depicted on the left-hand side of the figure.
• a metadata generation process 124 carried out by AI inference engine 76 in processor 64 of SSD controller 56, is depicted on the right-hand side.
• the object storage/retrieval process begins with processor 64 receiving one or more storage/retrieval commands from hosts 24, via host interface 68, at a command reception operation 128.
• Storage/retrieval commands may comprise, for example, commands to read, write, modify or delete one or more media objects.
• Flash management module 72 executes the commands in Flash devices 52 via memory interface 60.
• the metadata generation process begins with processor 64 receiving a pre- trained AI model from hosts 24, via host interface 68, at a model reception operation 136.
• processor 64 compiles the AI model and configures AI inference engine 76 with the compiled model.
• AI inference engine selects a media object that is stored in Flash devices 52 (i.e., from among objects 80), or that is received over network 28 for storage in Flash devices 52.
• metadata can be generated both for media objects that are already stored in Flash devices 52 and for objects that are intercepted on- the-fly before they are stored in Flash devices 52.
• processor 64 checks whether at least some resources of processor 64 are free of handling storage/retrieval commands. If not, no metadata computation is performed. If an idle period is identified (or, alternatively, if the analytics task was defined by the host as a high-priority task) processor 64 computes the metadata for the selected media object using AI inference engine 76, at a metadata computation operation 152. AI inference engine 76 computes the metadata using the AI model that was provided at operation 136 and compiled at operation 140.
• processor 64 adds the metadata of the selected object to local metadata database 84.
• the metadata database is not necessarily local.
• processor 64 sends the metadata to a remote storage location, with the object data being stored locally at least temporarily. The method then loops back to operation 144 above, for selecting the next media obj ect for metadata computation.
• Fig. 3 The method flow of Fig. 3 is an example flow that is depicted solely for the sake of conceptual clarity. In alternative embodiments, any other suitable method can be used for combined object storage/retrieval and metadata computation by SSD controller 56.
• the metadata in local databases 84 of the various SSDs 44 is accessible for use by hosts 24.
• a host 24 reads the metadata, and possibly associated objects or parts thereof, from SSD 44.
• processor 64 of SSD controller 56 receives from a host 24 a request for certain metadata, and sends the requested metadata to the requesting host in response.
• processor 64 also sends to the requesting host one or more related media objects, or parts thereof.
• processor 64 sends some or even all of the metadata to hosts 24 proactively, i.e., regardless of any request from the hosts.
• a host 24 analyzes the metadata and requests selected segments of object media based on analysis of the metadata.
• an analysis task of a host 24 requires the use of metadata from multiple metadata databases 84 of different SSDs 44.
• the host typically obtains the relevant metadata from the multiple SSDs, and consolidates the metadata as needed.
• the AI model and the objects for storage are received from the same host 24. In other embodiments, the AI model and the objects for storage are received from different hosts 24.
• processes 120 and 124 are carried out concurrently.
• processor 64 gives higher priority to storage/retrieval of objects (process 120) than to metadata generation (process 124).
• AI inference engine 76 generates the metadata during idle periods during which at least some resources of processor 64 are free from managing the storage/retrieval of objects.
• processor 64 identifies such idle periods in real time, and initiates or resumes metadata generation during the identified idle periods.
• Processor 64 suspends generation of metadata outside the identified idle periods.
• processor 64 buffers unprocessed media objects in volatile memory until the associated metadata is generated.
• processor 64 may use any other suitable prioritization scheme for giving precedence to storage/retrieval over metadata generation.
• Fig. 4 is a block diagram that schematically illustrates an SSD 160, in accordance with an alternative embodiment that is described herein.
• SSD 160 comprises multiple Flash devices 52, e.g., NAND Flash memories, an SSD controller 164 and (optionally) a Dynamic RAM (DRAM) 168.
• SSD controller 164 comprises a host interface 172, a media controller 176, a Static RAM (SRAM) 180, one or more CPUs 184, a DRAM controller 188, and an AI inference engine 76 that optionally comprises a CPU 192.
• SRAM Static RAM
• Host interface 172 communicates with hosts 24, e.g., in accordance with the NVMe, SATA protocol or other suitable storage protocol.
• Media controller 176 is a processor that manages storage and retrieval of data in Flash devices 52, similarly to Flash management module 72 of Figs. 1 and 2.
• DRAM controller 188 manages storage of data in DRAM 168.
• the various elements of SSD controller 164 communicate with one another via a fabric 196, which typically comprises suitable data and control buses.
• the example of Fig. 4 demonstrates that in some embodiments the SSD controller (or other storage device controller) comprises multiple processors that jointly perform data storage/retrieval (e.g., storage/retrieval of media objects) and metadata computation.
• the multiple processors may reside in one or more Integrated Circuits (ICs), packaged in a single package as a single device, or in multiple separate packages.
• ICs Integrated Circuits
• FIGs. 1, 2 and 4 show example configurations in which a controller of a storage device (e.g., an SSD controller in an SSD) communicates over a computer network with one or more remote hosts, and locally with the NVM of the storage device.
• the controller comprises one or more processors that jointly perform a dual function: (i) management of local storage or retrieval of media objects in the NVM, and (ii) compute and store in the NVM metadata of media objects that are stored, or that are to be stored, in the NVM.
• any other suitable system and/or storage-device configuration can be used.
• HDDs Hard Disk Drives
• the various elements of data processing systems 20 and 90, and of their components such as SSDs 44 and 114, as well as SSD 160 and its components, e.g., AI inference engine 76, may be implemented using dedicated hardware or firmware, such as using hard-wired or programmable logic, e.g., in an Application-Specific Integrated Circuit (ASICs) or Field- Programmable Gate Array (FPGA), using software, or using a combination of hardware and software elements.
• ASICs Application-Specific Integrated Circuit
• FPGA Field- Programmable Gate Array
• processor 64 of SSD controller 56, CPU 184 and/or CPU 192 comprise programmable processors, which are programmed in software to carry out the functions described herein (e.g., Flash management and metadata computation).
• the software may be downloaded to the processor in electronic form, over a network, for example, or it may, alternatively or additionally, be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• Databases & Information Systems (AREA)
• Data Mining & Analysis (AREA)
• Library & Information Science (AREA)
• Human Computer Interaction (AREA)
• Software Systems (AREA)
• Multimedia (AREA)
• Computational Linguistics (AREA)
• Mathematical Physics (AREA)
• Computer Hardware Design (AREA)
• Computer Networks & Wireless Communication (AREA)
• Signal Processing (AREA)
• Computing Systems (AREA)
• Artificial Intelligence (AREA)
• Biomedical Technology (AREA)
• Molecular Biology (AREA)
• Life Sciences & Earth Sciences (AREA)
• Health & Medical Sciences (AREA)
• General Health & Medical Sciences (AREA)
• Evolutionary Computation (AREA)
• Biophysics (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Memory System (AREA)
• Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
• Management Or Editing Of Information On Record Carriers (AREA)

Priority Applications (4)

Applications Claiming Priority (10)

Publications (1)

Family

ID=65520347

Family Applications (4)

Family Applications After (3)

Country Status (6)

Families Citing this family (43)

Citations (4)

Family Cites Families (134)

• 2019
  • 2019-01-31 US US16/264,248 patent/US11748418B2/en active Active
  • 2019-01-31 JP JP2021503769A patent/JP7326667B2/ja active Active
  • 2019-01-31 US US16/262,975 patent/US11036807B2/en active Active
  • 2019-01-31 CN CN201980050454.0A patent/CN112534423B/zh active Active
  • 2019-01-31 EP EP19707121.0A patent/EP3830713A1/en not_active Ceased
  • 2019-01-31 US US16/264,473 patent/US11080337B2/en active Active
  • 2019-01-31 WO PCT/IB2019/050776 patent/WO2020026036A1/en not_active Ceased
  • 2019-01-31 KR KR1020217004754A patent/KR102894244B1/ko active Active
  • 2019-01-31 US US16/263,387 patent/US11068544B2/en active Active
  • 2019-01-31 US US16/262,971 patent/US11294965B2/en active Active
  • 2019-07-29 JP JP2021503760A patent/JP7419621B2/ja active Active
  • 2019-07-29 CN CN201980050665.4A patent/CN112513834B/zh active Active
  • 2019-07-29 WO PCT/IB2019/056442 patent/WO2020026112A1/en not_active Ceased
  • 2019-07-29 EP EP23167414.4A patent/EP4266194B1/en active Active
  • 2019-07-29 EP EP19762480.2A patent/EP3830717B1/en active Active
  • 2019-07-31 JP JP2021504432A patent/JP2021533447A/ja active Pending
  • 2019-07-31 EP EP23174241.2A patent/EP4220437A1/en active Pending
  • 2019-07-31 EP EP19756453.7A patent/EP3830714B1/en active Active
  • 2019-07-31 KR KR1020217004932A patent/KR102876339B1/ko active Active
  • 2019-07-31 WO PCT/US2019/044551 patent/WO2020028594A1/en not_active Ceased
  • 2019-07-31 JP JP2021504383A patent/JP2021533446A/ja active Pending
  • 2019-07-31 EP EP19758853.6A patent/EP3830715B1/en active Active
  • 2019-07-31 EP EP19759088.8A patent/EP3830716B1/en active Active
  • 2019-07-31 CN CN201980058977.XA patent/CN112673368A/zh active Pending
  • 2019-07-31 CN CN201980064192.3A patent/CN112771515A/zh active Pending
  • 2019-07-31 EP EP23157721.4A patent/EP4206951A1/en active Pending
  • 2019-07-31 WO PCT/US2019/044558 patent/WO2020028597A1/en not_active Ceased
  • 2019-07-31 CN CN201980057663.8A patent/CN112639768B/zh active Active
  • 2019-07-31 JP JP2021503799A patent/JP7351057B2/ja active Active
• 2021
  • 2021-05-02 US US17/246,650 patent/US11727064B2/en active Active
  • 2021-07-15 US US17/377,236 patent/US11734363B2/en active Active
• 2023
  • 2023-09-29 JP JP2023169249A patent/JP2023171874A/ja not_active Abandoned
  • 2023-10-12 JP JP2023176582A patent/JP7694997B2/ja active Active
• 2024
  • 2024-01-05 JP JP2024000847A patent/JP7721700B2/ja active Active

Patent Citations (4)

Also Published As

Similar Documents

Legal Events