WO2010147546A1 - Procédé et système de stockage de données - Google Patents

Procédé et système de stockage de données Download PDF

Info

Publication number
WO2010147546A1
WO2010147546A1 PCT/SE2010/050675 SE2010050675W WO2010147546A1 WO 2010147546 A1 WO2010147546 A1 WO 2010147546A1 SE 2010050675 W SE2010050675 W SE 2010050675W WO 2010147546 A1 WO2010147546 A1 WO 2010147546A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
ctrl
initial
ism
file system
Prior art date
Application number
PCT/SE2010/050675
Other languages
English (en)
Inventor
Göran Olsson
Leif BYSTRÖM
Mats MÅRDBERG
Original Assignee
Ikonoskop Ab
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ikonoskop Ab filed Critical Ikonoskop Ab
Priority to EP10789824A priority Critical patent/EP2443555A1/fr
Priority to US13/377,837 priority patent/US8566557B2/en
Publication of WO2010147546A1 publication Critical patent/WO2010147546A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/60—Memory management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/188—Virtual file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1847—File system types specifically adapted to static storage, e.g. adapted to flash memory or SSD
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/51—Indexing; Data structures therefor; Storage structures

Definitions

  • the present invention relates to a method and a system for storing initial data from an image detecting device in a camera system initial storage medium and later accessing the data.
  • the initial data is stored consecutively in blocks, where file specific pointers representing the starting address for each stored file and file sequence, and dynamic memory pointers that points out the next writable address, are managed and stored during real time writing of the data.
  • ISM Ctrl initial storage medium controller
  • ISM Ctrl initial storage medium controller
  • a storage medium is a medium/device for storing/recording information data.
  • ESM external storage medium
  • ESM Ctrl external storage media controller
  • File system types can be classified into several categories such as disk file systems, flash file systems, network file systems and other special purpose file
  • an initial storage medium controller (ISIVI Ctrl) is optimized for as fast writing as possible, i.e.
  • the inventive method and system avoids the described problems by letting a virtual file system controller (VFS Ctrl) dynamically buffer a virtual representation of the data in a virtual file system (VFS) where the VFS format is commonly known by other kinds of storage medium controllers.
  • VFS Ctrl virtual file system controller
  • VFS The meaning of a VFS is that such a file system is not traditionally implemented in any physical storage medium such as the initial storage medium, the file system is implemented internally within the VFS controllers own processing environment and is only momentarily instantiated on-demand in the VFS control- lers own RAM-like memory when a request to access the data is received.
  • the VFS Ctrl presents the data towards the external storage medium controller in such manner that it is recognised as if the data is stored in files on a
  • a data pipeline is implemented from the image detecting device to the initial storage medium, and due to the high data rate requirements the implementation is not done by using traditional CPU software but is instead implemented in an FPGA by using source code in a hard- ware description language (HDL).
  • HDL hard- ware description language
  • the complexity of a file system and image files in this fashion is almost impossible, especially when the performance is considered, So basically the data has to be stored sequentially as it comes. This is not readable by an ordinary computer but can be made available through the VFS. If a flash memory device is used as ISM it is proposed that the virtual file system origins from the flash file system of the flash memory device.
  • the system makes use of a simplified wear leveling technique; Every time when the flash memory is completeiy erased, the ISM Ctrl randomly calculates an initial memory pointer, pointing out the initial memory block address where ISM Ctrl will start to write data. When the dynamic memory pointer exceeds the last physical memory block address fay one memory block address unit, the ISSVI Ctrl will reset the dynamic memory pointer to the first writable physical memory block address and from there continue to increment the dynamic memory pointer. The last writable memory block will then be the initial memory block address decremented by one memory block address unit.
  • an algorithm to detect bad memory blocks is used, that the positions of identifies bad memory blocks are stored, for example in a list, and that when the dynamic memory pointer enters an address of a bad memory block, the dynamic memory pointer is increment the by one memory block address unit without writing any data into the identified bad memory block,
  • the present invention teaches that complex implementations of the file system architecture itself is created once for every new connection to an externa! computing system and stored in an internal RAM memory, that more voluminous and repetitive parts of the file system, such as file/image data, are created on the fly, and that the actual image data is retrieved on demand from the initial storage
  • the virtual representation of the initial data presents the data in a sta ⁇ dar- dised format for digital photography.
  • One such standardised format is the Digital Negative format (DNG). This particular format can be easily implemented since it does not require any further conversion or compression of the data.
  • DNG Digital Negative format
  • JPEG Joint Photographic Experts Group
  • TIFF Tagged Image File Format
  • these formats would also require that the VFS Ctrl performs a conversion of the initial data into the image file format as the initial data is to be presented to the external storage medium controller.
  • the inventive virtual file system controller can be implemented in different ways.
  • One proposed implementation is that the initial computing system comprises the virtual file system controller, thus making the initial computing system compatible with any externa! computing system that can access the file format presented by the virtual file system controller.
  • the externa! computing system comprises the virtual file system controller. This would provide a simple and less expensive camera system but on the other hand it would require an extra computer program component in the externa! computing system in order to realize the virtual file system controller and access the initial data in the camera system.
  • the advantages of a method and a system according to the present invention are that invention is applicable to all forms of data acquisition devices that have real time constraints on the acquisition side and more relaxed real time constraints during "offloading". Another advantage is that the invention is applicable to all forms of data acquisition devices that have real time constraints on the acquisition side and stricter constraints regarding standardized communication interfaces, standardized data format and standardized data file system representation during "offloading". Another advantage is that the creator of data acquisition devices according to the invention doesn't have Io bother with the ECS side software, especially on driver level.
  • Figure 1 is a schematic and simplified iliustration of an inventive method and system
  • Figure 2 is a schematic and highly simplified iliustration of a initial storage medium
  • Figure 3 is a schematic illustration of a list with bad memory blocks
  • Figure 4 is a schematic and simplified Illustration of an alternative embodiment of an inventive system
  • Figure 5 is a schematic and simplified illustration of a first embodimen with an initial storage medium as a standalone unit
  • Figure 6 is a schematic and simplified illustration of a second embodiment with an initial storage medium as a standalone unit.
  • Figure 2 show schematically an initial storage medium 11 where it is illustrated that the data must be stored in files and sequences of files, and that the ISM Ctrl 12 must, momentarily during real-time writing, be able to store a file specific memory pointer 21 , 22, .., 2n, representing the starting address for each stored file and file sequence Seq.1 , Seq. 2, ,,. , Seq. n.
  • the ISM Ctrl 12 will also store and manage a dynamic memory pointer 3 that points out the next writable address in the ISM 11.
  • the ISM Ctrl 12 must also be able to administrate all the memory pointers 21 , 22, ... , 2n, in such a manner that later on, in a situation with no real-time restrictions, when access is required to data belonging to a specific file and file sequence, the ISM Ctrl 12 could find and read this specific file and file sequence data.
  • the present invention teaches that thus stored data is made accessible through a virtual representation of the data In a virtual file system 4 with a format that Is known by an external storage medium controller 51 so that the data can be transferred to an external medium 52, and that the virtual representation is related to the file specific pointers 21 , 22 « ,., 2n.
  • the virtual file system 4 is momentarily instantiated on-demand when a request to access the data is received and the virtual file system 4 is implemented internally within the processing environment of a virtual file system controller 41.
  • the present invention teaches that the virtual file system controller 41 presents the data towards the external storage medium controller 51 in such manner that it is recognised as if the data is stored in files on a file system commonly known by the external storage medium controller 51.
  • a data pipeline 13 is used from the image detecting device 14 of the system 1 to the initial storage medium 11 , and that a Field-Programmable Gate Array (FPGA) 15 is used to control the storing of
  • FPGA Field-Programmable Gate Array
  • Different memory devices can be used as an initial storage medium 11 and one possible memory device is a flash memory device, in which case the virtual file system 4 origins from the flash fife system of the flash memory device. If using a flash memory device ft is proposed that each lime the flash memory device is completely erased, an initial memory pointer, pointing initial memory block address where new data will start to be written, is r ⁇ caicuiated. It is then proposed that when the dynamic memory pointer 3 exceeds the last physical memory block address in the storage medium 11 by one memory block address unit, the dynamic memory pointer 3 is reset to the first writable
  • an algorithm to detect bad memory blocks is used, that the positions 81 , 62, .... 6n, of identified bad memory blocks are stored, for example in a list 8, schematically shown in figure 3, and that when the dynamic memory pointer 3 enters an address 81 , 62, .,., 6n of a bad memory block, the dynamic memory pointer 3 is increment the by one memory block address unit without writing any data into the identified bad memory block,
  • Complex implementations of the file system architecture itself is created once for every new connection to an external computing system and stored in an internal RAM memory 7, while more voluminous and repetitive parts of the file system, such as file/image data, are created on the fly, and the actual image data is retrieved on demand from the initial storage medium 11.
  • the number of takes stored and the number of frames for each take is used as input to create the virtual file system 4 and It is proposed that the virtual file system 4 is created in the format of a FAT32 file system or file system
  • the initial computing system 1 comprises an initial storage medium (ISM) 11 and an ISM 11
  • the ISM Ctrl 12 momentarily stores file specific 21 , 22, .., 2r sequence Seq, 1 , Seq. 2, ..., Seq. n, during real time writing of the data into the ISM 11.
  • the ISM Ctrl 12 is also adapted to store and manage a dynamic memory pointer 3 that points out the next writable address in the ISM 11 , and the ISM Ctrl 12 is adapted to administrate a!! the memory pointers 21 , 22, ... , 2n in such a manner that when access is required to data belonging to a specific file seqi Seq. 1 , Seq. 2, ,., , Seq. R 1 the ISM Ctrl 12 can find and read the specific file
  • ile system controller (VFS Ctrl) 41 is adapted to dynamically buffer a virtual representation of the data in a virtual file system (VFS) 4 by means of the file specific pointers 21 , 22, ... , 2n, where the VFS format is commonly known by an external storage medium controller (ESM Ctrl) 51 belonging to an external computing system (ECS) 5.
  • ESM Ctrl external storage medium controller
  • the VFS Ctrl 41 is adapted to instantiate the VFS 4 momentarily on- demand when a request to access the data is received from the ESIVl Ctrl 51 , in which case the VFS Ctrl 41 is adapted to implement the VFS 4 internally within the
  • the VFS Ctrl 41 is adapted to present the data towards the ESM Ctri 51 in such manner that it is recognised as if the data is stored in files on a file system commonly known by the ESM Ctrl 51 It is also proposed that the inventive system A comprises a data pipeline
  • FPGA Field-Programmable Gate Array
  • the ISM 11 can be a flash memory device, in which case the VFS 4 origins from the flash file system of the flash memory device. It is proposed that the ISM Ctrl 12 is adapted to randomly calculate an initial memory pointer, pointing out the initial memory block address where new data will start to be written each time the flash memory device is completely erased.
  • the ISM Ctrl 12 is adapted to reset the dynamic memory pointer 3 to the first writable physical memory block address when the dynamic memory pointer 3 exceeds the last physical memory block address in the ISM 11 by one memory block address unit, and that the ISM Ctrl 12 is adapted to continue to be increment the dynamic memory pointer 3 from there.
  • the ISM Ctrl 12 is adapted to use an algorithm to detect bad memory blocks, that the ISM Ctrl 12 Is adapted to store the positions 61, 62, ..., 8n, of identified bad memory blocks, for example in a list 8, schematically shown in figure 3, and that the ISM Ctrl 12 is adapted to increment the dynamic memory pointer 3 by one memory block address unit without writing any data into an identified bad memory block when said dynamic memory pointer enters an
  • the VFS Ctrl 41 is adapted to create complex implementations of the file system architecture itself once for every new connection to an external computing system (ECS) 5 and to store it ⁇ n an internal RAM memory 7.
  • the VFS Ctrl 41 is also adapted to create more voluminous and repetitive parts of the file system, such as file/image data, on the fly, and to retrieve the actual image data from the
  • the VFS Ctrl can be adapted to use the number of takes stored and the number of frames for each take as input to create the VFS In the format of a
  • FAT32 file system or file system skeleton FAT32 file system or file system skeleton.
  • VFS Ctrl 41 is adap- ted to create the virtual representation of the initial data so that it presents the data in a standardised format for digital photography, such as Digital Negative format
  • the initial computing system 1 comprises the VFS Ctrl 41.
  • the externa! computing system 5 f comprises the VFS Ctrl 41 ', where the VFS 4' is created in the externa! computing
  • the 'Ie implementation of the present invention is sffustrated rn figure 5, showing that the ISM 11 " with its ISM Ctrl 12', when removed from the initial computing system 1 ! , can be adapted to act as a standalone unit 8, and as such be connected to the externa! computing system 5".
  • the !SM Ctrl 12 ! is also acting as a VFS Ctrl 41 " and the VFS 4" is realised on the ISM 11'.
  • the external computing system 5' comprises the VFS Ctrl 41"" and the VFS 4'" is created in the external computing system 5" J , in which case it is possible to access the standalone unit 8' and the data in the [SM 11 " without having to realise the VFS and the VFS Ctrl in the standalone unit 8 s .
  • the standalone unit 8, 8 ! wit! allow the external computing system 5", 5"' to access the information in the ISM 11', 11" without having to use the initial computing system 1' to access the information.
  • FIG. 1 shows the present invention in a very simplified way where the Initial Computing System (ICS) 1 is represented by a camera system and the External Computer System (ECS) 5 is represented as a hosting computer.
  • the camera system detects that a hosting computer is connected to the USB 8 connector.
  • the camera system presents itself as a USB mass storage device,
  • the hosting computer then starts to read basic Information about the newfound storage device by reading sectors from the storage device.
  • the camera system responds with sectors that correspond with sectors usually found in mass storage devices ⁇ ke partition tables, boot-sectors and so on.
  • the hosting computer When the hosting computer asks for sectors that is a part of the file sys- tern or file headers, the camera system responds with the pre-computed sectors or sectors created on the fly. Irs this embodiment, the hosting computer reads the file system format as FAT32 and the file format as DNG. The hosting computer will then be able to build up and represent the stored image sequences in a file system format (i.e. FAT32) and fiie format (i.e. DNG, CinernaDNG) that the hosting computer fully understands,
  • the camera system retrieves the data from the ISM. ft wii! be understood that the invention is not restricted to the aforede- scribed and Illustrated exemplifying embodiments thereof and that modifications can be made within the scope of the inventive concept as defined by the ig Claims.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

La présente invention porte sur un procédé et un système (A) pour stocker des données initiales provenant d'un dispositif de détection d'image (14) dans un support de stockage initial de système de caméra (11) et accéder ultérieurement aux données. Les données initiales sont stockées consécutivement dans des blocs, des pointeurs spécifiques de fichier représentant l'adresse de départ pour chaque fichier et séquence de fichier stockés, et des pointeurs de mémoire dynamique qui pointent vers l'adresse d'écriture possible suivante, étant gérés et stockés durant une écriture en temps réel des données. Les données sont rendues accessibles par l'intermédiaire d'une représentation virtuelle des données dans un système de fichier virtuel (4) avec un format qui est connu par un contrôleur de support de stockage externe (51), laquelle représentation virtuelle est relative aux pointeurs spécifiques de fichier.
PCT/SE2010/050675 2009-06-16 2010-06-16 Procédé et système de stockage de données WO2010147546A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP10789824A EP2443555A1 (fr) 2009-06-16 2010-06-16 Procédé et système de stockage de données
US13/377,837 US8566557B2 (en) 2009-06-16 2010-06-16 Method and system for storage of data

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
SE0950463A SE534142C2 (sv) 2009-06-16 2009-06-16 Förfarande och system för lagring av data
SE0950463-0 2009-06-16

Publications (1)

Publication Number Publication Date
WO2010147546A1 true WO2010147546A1 (fr) 2010-12-23

Family

ID=43356619

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/SE2010/050675 WO2010147546A1 (fr) 2009-06-16 2010-06-16 Procédé et système de stockage de données

Country Status (4)

Country Link
US (1) US8566557B2 (fr)
EP (1) EP2443555A1 (fr)
SE (1) SE534142C2 (fr)
WO (1) WO2010147546A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10055425B2 (en) 2011-12-01 2018-08-21 Thomson Licensing Method for storing content data of a content receiver in a portable storage device, and associated control device

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2012140249A1 (fr) * 2011-04-14 2012-10-18 Yubico Ab Dispositif à deux interfaces pour contrôle d'accès et procédé correspondant
WO2016192025A1 (fr) 2015-06-01 2016-12-08 SZ DJI Technology Co., Ltd. Systèmes et procédés pour une architecture de mémoire
US11157422B2 (en) * 2017-03-31 2021-10-26 Intel Corporation Shared memory for intelligent network interface cards
US10444722B2 (en) * 2017-09-29 2019-10-15 Intel Corporation Techniques to direct access requests to storage devices
CN109635588A (zh) * 2018-12-20 2019-04-16 天津天地伟业信息系统集成有限公司 一种基于Linux虚拟文件系统的文件保护方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1991014986A1 (fr) * 1990-03-23 1991-10-03 Eastman Kodak Company Gestion de memoire virtuelle et agencement d'affectation pour systeme de traitement de donnees numeriques
WO1999003040A1 (fr) * 1997-07-10 1999-01-21 Cirrus Logic, Inc. Gestionnaire de memoire virtuelle pour moteurs multimedia
US6195734B1 (en) * 1997-07-02 2001-02-27 Micron Technology, Inc. System for implementing a graphic address remapping table as a virtual register file in system memory
US20040068522A1 (en) * 2002-10-03 2004-04-08 Rodger Daniels Virtual storage systems and virtual storage system operational methods
US20060253484A1 (en) * 2005-05-03 2006-11-09 Bangalore Kiran Kumar G Flash memory directory virtualization

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1991014986A1 (fr) * 1990-03-23 1991-10-03 Eastman Kodak Company Gestion de memoire virtuelle et agencement d'affectation pour systeme de traitement de donnees numeriques
US6195734B1 (en) * 1997-07-02 2001-02-27 Micron Technology, Inc. System for implementing a graphic address remapping table as a virtual register file in system memory
WO1999003040A1 (fr) * 1997-07-10 1999-01-21 Cirrus Logic, Inc. Gestionnaire de memoire virtuelle pour moteurs multimedia
US20040068522A1 (en) * 2002-10-03 2004-04-08 Rodger Daniels Virtual storage systems and virtual storage system operational methods
US20060253484A1 (en) * 2005-05-03 2006-11-09 Bangalore Kiran Kumar G Flash memory directory virtualization

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10055425B2 (en) 2011-12-01 2018-08-21 Thomson Licensing Method for storing content data of a content receiver in a portable storage device, and associated control device

Also Published As

Publication number Publication date
EP2443555A1 (fr) 2012-04-25
SE0950463A1 (sv) 2010-12-17
SE534142C2 (sv) 2011-05-10
US8566557B2 (en) 2013-10-22
US20120102266A1 (en) 2012-04-26

Similar Documents

Publication Publication Date Title
US8566557B2 (en) Method and system for storage of data
US20080195833A1 (en) Systems, methods and computer program products for operating a data processing system in which a file system's unit of memory allocation is coordinated with a storage system's read/write operation unit
US20060168362A1 (en) Control method for storage device controller system, and storage device controller system
US7805584B2 (en) Method and system for restoring data
US8667229B2 (en) Data access method of a memory device
US20140189674A1 (en) Information processing apparatus and information processing method
US8996791B2 (en) Flash memory device, memory control device, memory control method, and storage system
EP1376328A2 (fr) Système de stockage
US8832405B1 (en) Method for achieving sequentail I/O performance from a random workload
KR20140026933A (ko) 플래시 메모리를 기반으로 하는 저장 장치 및 플래시 메모리를 제어하는 메모리 컨트롤러의 쓰기 메모리 블록 할당 방법
US8478933B2 (en) Systems and methods for performing deduplicated data processing on tape
US20100161902A1 (en) Method, system, and program for an adaptor to read and write to system memory
JP3324572B2 (ja) 情報処理装置並びにコンピュータに実行させるためのプログラムを記録した記録媒体
JPH11184639A (ja) 計算機システム
US20090141606A1 (en) Method and system for managing data from host to optical disc
US20100174880A1 (en) Method for fast format of a fully allocated volume when copied from a space efficient volume
US20060143313A1 (en) Method for accessing a storage device
KR20060017816A (ko) 주 메모리와 기억 장치 사이에서 데이터를 전송하는 방법및 장치
CN109658985B (zh) 一种基因参考序列的去冗余优化方法及系统
US20030145180A1 (en) Method and system for providing direct access recovery using seekable tape device
JP2023016972A (ja) ストレージシステム及びデータ処理方法
US20090077309A1 (en) Data transfer
US20080022060A1 (en) Data recording apparatus, program product, and data recording method
US8495279B2 (en) Flash file system and driving method thereof
CN111045961A (zh) 数据处理方法及使用所述方法的存储控制器

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10789824

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 13377837

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2010789824

Country of ref document: EP