EP3077953B1 - Gestion de données sensibles de production - Google Patents

Gestion de données sensibles de production Download PDF

Info

Publication number
EP3077953B1
EP3077953B1 EP14819183.6A EP14819183A EP3077953B1 EP 3077953 B1 EP3077953 B1 EP 3077953B1 EP 14819183 A EP14819183 A EP 14819183A EP 3077953 B1 EP3077953 B1 EP 3077953B1
Authority
EP
European Patent Office
Prior art keywords
data
sensitive
production data
production
sensitive data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP14819183.6A
Other languages
German (de)
English (en)
Other versions
EP3077953A1 (fr
Inventor
Stephen J. Cook
Jackson M. Davis
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Publication of EP3077953A1 publication Critical patent/EP3077953A1/fr
Application granted granted Critical
Publication of EP3077953B1 publication Critical patent/EP3077953B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G06F21/6254—Protecting personal data, e.g. for financial or medical purposes by anonymising data, e.g. decorrelating personal data from the owner's identification

Definitions

  • Sensitive data can comprise, among other things, personally identifiable information (PII), which can be used to identify information about individual users. For instance, medical applications will likely contain doctors' notes about patients. Similarly, online store applications likely have credit card numbers of users in memory.
  • PII personally identifiable information
  • production data can include sensitive data, it is undesirable to make such production data accessible outside a production environment. Moreover, this may be required due to legal requirements and privacy agreements.
  • the production data is useful for diagnostic and other purposes. As a result, production data may be available for use within a production environment, if allowed. Otherwise, test data can be utilized in an attempt to replicate production application state.
  • the subject disclosure pertains to managing sensitive production data. More specifically, sensitive data discovered amongst production data can be concealed from view outside a production environment. Sensitive data can be discovered in a number of ways. In accordance with one aspect, class fields or other program constructs in an application can be identified as locations where sensitive data is known to reside. Instances of sensitive data can subsequently be discovered in production data based on the specified locations. Sensitive data discovered by the above or other technique can be masked in various manners so that the sensitive data is not visible. Furthermore, metadata associated with sensitive data can be maintained with respect to the masked sensitive data.
  • Production data which represents the state of an application executing in a production environment, can include sensitive data that should be kept secret.
  • Sensitive data can be discovered utilizing a variety of techniques/mechanisms.
  • sensitive data can be discovered as a function of identified locations in a production application, such as data fields, where sensitive data is known to reside.
  • Discovered sensitive data can be masked to conceal the sensitive data from view outside a production environment. For instance, the sensitive data can be replaced with an arbitrary value, a hash of the sensitive data, or cipher text.
  • management system 100 includes discovery component 110 and cleanse component 120 and is configured to manage production data 130 produced by production application 140 to prevent leakage of sensitive data. Leakage is prevented by removing sensitive data amongst production data or otherwise preventing sensitive data from being introduced in to the production data. As a result, production data can be exposed outside a production environment since sensitive data will not be visible. Additionally, the management system 100 can maintain metadata regarding sensitive data to facilitate analysis.
  • the production application 140 is a software application that executes in a production environment, as opposed to a pre-production development environment, for example.
  • the application is deployed in the real world serving end-users/clients and providing value (e.g., business value) to the end-users/clients running the application.
  • the production application 140 could be a banking application or an online sales application, among other things.
  • the production data 130 includes data generated during execution of a production application that represents a state of execution at a specific moment time when captured.
  • the production data 130 can correspond to one or more memory snapshots or dumps that represent the memory state of the production application 140 at a time when the data was captured, or, in other words, when the snapshot was taken or when the dump occurred.
  • Such memory snapshots or dumps can include information about stacks, heaps, loaded classes, objects, and values of fields, logs/traces, among other things.
  • the production data 130 can be employed in a diagnostic context to debug a production application, in which case the production data 130 can be termed a diagnostic artifact. Additionally or alternatively, the production data can be employed in an analytics context to analyze program performance.
  • the production data 130 can comprise sensitive data, which is production data that needs to be protected or kept secret for one or more reasons including to satisfy regulatory requirements (e.g., federal or state law), organization privacy policies, or user/client configurable privacy settings, among others.
  • sensitive data is personally identifiable information (PII), which can be any data about particular individuals such as a person's name, social security number, driver's license number, account number, credit/debit card number, and security code or password for an account.
  • PII personally identifiable information
  • Other non-limiting examples of sensitive data can include student information and grades as well as health or medical information.
  • the discovery component 110 and the cleanse component 120 cooperate to prevent sensitive data from being leaked or exposed to unauthorized individuals such as those involved in diagnosing a problem with a production application or those tasked with analyzing production application performance.
  • the discovery component 110 is configured to discover or identify sensitive data
  • the cleanse component 120 is configured to remove, or prevent initial injection of, the sensitive data identified by the discovery component 110.
  • the discovery component 110 can be implemented in a variety of manners and operate in different contexts. Turning attention to FIG. 2 , a representative discovery component 110 is illustrated in further detail.
  • the discovery component 110 includes application analysis component 210 and data analysis component 220.
  • the application analysis component 210 is configured to analyze the production application 140 and identify locations where sensitive data is known or likely to reside.
  • a developer can provide metadata about the nature of sensitive data including specific locations such as class fields where sensitive data is known to reside. Data regarding the location of sensitive data can be specified explicitly in terms of attributes associated with particular data fields, for example.
  • a developer can annotate the production application 140 with attributes that indicate which fields or other program constructs may include sensitive data. Based on this information, the location where an instance of sensitive data may reside amongst production data 130 is known.
  • Other techniques can also be utilized to determine or infer locations of potential sensitive data based on information from the production application 140. For example, names of variables, classes, functions, or methods, among other things may be indicative of the presence and location of sensitive data.
  • the data analysis component 220 is configured to discover specific instances of sensitive data.
  • the data analysis component can exploit information regarding locations that may include sensitive data as determined by the application analysis component 210.
  • the production application 140 may not have been annotated with data identifying locations of potential sensitive data and thus the application analysis component 210 may not identify any locations of interest based on annotation or other techniques.
  • the data analysis component 220 can be configured to operate independent of results from the application analysis component 210.
  • FIG. 3 depicts a representative data analysis component 220 in further detail.
  • Search component 310 is configured to search a set of production data for sensitive data.
  • the search component 310 can seek to locate sensitive data based on patterns capturing format or other characteristics of sensitive data, for instance utilizing regular expressions or context free grammars.
  • social security numbers comprise three numbers, a dash, two numbers, a dash, and four numbers. This defines a shape of data that corresponds to sensitive data. Based on this data shape, captured by a regular expression, for example, instances of social security numbers can be located. Similar pattern matching can be utilized to locate telephone and account numbers, among other things.
  • the search component 310 can be employed to identify the same instances of the sensitive data in other locations. For example, if the name of a person is discovered in a particular data field, the search component 310 can seek to locate other instances of the name elsewhere, outside the data field. Consequently, multiple passes over production data may be needed to discover a particular instance of sensitive data and other instances of the discovered sensitive data.
  • Subset component 320 is configured alone or in conjunction with the search component 310 to identity sensitive data that is a subset of previously discovered or identified sensitive data. For example, if it is determined, based on code annotation or other technique, that a name of a person, including a first name and last name, is sensitive data, the subset component 320 can seek to identify subsets, namely first name and last name, and locate instances of the first name and last name independent of each other. Similarly, if a phone number including an area code is identified as sensitive data, the subset component 320 can attempt to locate a substring of the phone number that does not include the area code.
  • Variance component 330 is configured alone or in combination with the search component 310 to identify variations of sensitive data. Variations can include but are not limited to differences in format, abbreviations, and misspellings. For example, if by some mechanism a social security number of the format "123-45-6789" is identified as sensitive data, other variations such as "123 45 6789” can also be searched for an if located labeled as sensitive data. As another example, suppose the name "David Doe" is considered sensitive data. In this case, the variation component 330 can enable identification of instances of data such as "D. Doe” or “Dave Doe” as equivalent and sensitive data as well.
  • the database 340 can be a computer-readable storage medium that stores data locally as shown, or remotely, for use by components of the data analysis component 220 in locating sensitive data.
  • the database 340 can include data known to correspond to sensitive data such as names and addresses for use by the search component 310. Common patterns can also be stored in the database 340 for employment by the search component 310.
  • the database 340 can include common misspellings of words or names as well as abbreviations for used by the variance component 330.
  • the database 340 can include a language dictionary that can be employed to discover proper nouns by searching against the language dictionary and marking anything not found as a proper noun.
  • FIG. 4 illustrates a representative cleanse component 120 that can be employed in conjunction with the management system 100 of FIG. 1 .
  • the cleanse component includes mask component 410 and metadata component 420.
  • the mask component 410 is configured to remove sensitive data amongst production data.
  • the mask component 410 is configured to replace the sensitive data with other non-sensitive data.
  • the mask component 410 is configured to conceal sensitive data by replacing the sensitive data with non-sensitive data.
  • the mask component 410 can employ a hash algorithm to sensitive data with a hash of the sensitive data.
  • an encryption algorithm can be utilized to replace sensitive data with cipher text.
  • the mask component 410 can simply replace sensitive data with arbitrary values.
  • the metadata component 420 is configured to maintain metadata regarding sensitive data. Masking of sensitive data by way of the mask component 410 removes sensitive data. Alone, however, masking can have an adverse effect on the ability to diagnose problems or evaluate performance of a production application. Metadata component 420 can at least mitigate this problem by maintaining data about sensitive data that is concealed with a mask.
  • the metadata can include data that is helpful in diagnosis sources of a bug or analyzing program performance. For example, the length of a data value can indicate the presence of a software error, flaw, or failure. Accordingly, the metadata component 420 can maintain the length of the sensitive data (e.g., string length). As a result, helpful data is preserved without exposing the sensitive data. Another non-limiting example of useful data that can be retained as metadata is value equality. In other words, metadata can indicate whether or not masked sensitive data is the same or different from other masked sensitive data.
  • the metadata component 420 can maintain metadata a number of different ways.
  • metadata can be appended (e.g. prepend, postpend) to masked data.
  • the length of data can be appended to the masked data.
  • metadata can be embedded or encoded within the masked data.
  • the length of the masked data can be the same as the sensitive data (e.g., variable length hash).
  • identical, similar, or equivalent sensitive data can be concealed with the same mask or, stated differently, replaced with a matching value.
  • a mask for a subset of the data can be generated based on the mask for a superset of the data or vice versa, for example where the mask for a full name is composed of a mask for a first name and a mask for a last name.
  • metadata can be encrypted as part of multiple layer encryption process where a key is provided to access data about the sensitive data by way of a partial decrypt but not the sensitive data itself.
  • the metadata component 420 can store metadata separately from the mask in a data source (e.g., database, other memory stream%), and subsequently a lookup against the source can be performed, for instance based on the mask value and/or memory addresses to acquire the metadata.
  • the metadata component 420 can be configured to write metadata to an arbitrary buffer along with identifiers that identify corresponding masked data.
  • the above implementations represent but a few exemplary ways in which metadata can be maintained. Others are possible and are to be deemed within the scope of this disclosure.
  • a diagnostic artifact can be produced by a production diagnostic system comprising production data.
  • the artifact could be a full process dump or a trace file, among other things. Any data field that was marked as sensitive can be masked out of the diagnostic artifact, for example using a cryptographically strong hash. Further, metadata regarding the sensitive data is retained. This ensures significant diagnostic information such as buffer length and equality are preserved, but sensitive data is removed.
  • a fixed length hash or data index plus length could be used.
  • Discovered instances of sensitive data need not be limited to fields marked as sensitive. Rather, all data collected can be checked against discovered sensitive data in multiple passes. This can include the heap, stack, and registers of a process dump or the contests of (or substring of) any fields in the diagnostic trace file. Further, subsets and variations of discovered sensitive data can be identified. Sensitive data can be destructively overwritten in the diagnostic file with the hash so it cannot be obtained by a developer. Later, when the developer consumes the diagnostic file, the sensitive data is masked and thus is not available. For instance, the name "John Doe” is not visible, but the hash and length are visible.
  • various portions of the disclosed systems above and methods below can include or employ of artificial intelligence, machine learning, or knowledge or rule-based components, sub-components, processes, means, methodologies, or mechanisms (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, classifiers).
  • Such components can automate certain mechanisms or processes performed thereby to make portions of the systems and methods more adaptive as well as efficient and intelligent.
  • the discovery component 110 can include such mechanism to determine or infer sensitive data. For instance, a learning mechanism can be used to identified data that otherwise may be been missed.
  • Missed instances can exist for several reasons.
  • a compiler e.g., JIT compiler
  • enregistered values which are values stored in registers instead of on a stack.
  • a second reason is loss of type information by a developer (e.g., casting a buffer to void*), such that the type cannot be readily inferred.
  • Another reason is stale copies of sensitive data sitting in memory that has been freed by the application but not overwritten by the operating system. For example, stack variables for frames that have already popped usually are still in memory under the current stack pointer and freed heap values are often not overwritten when variables are in question or freed.
  • a method of managing production data 500 is illustrated.
  • sensitive data is discovered amongst production data.
  • Production data represents a state of execution at a specific moment time when captured and can include process stacks, process heap, stack frame registers, and logging/tracing, among other things.
  • Sensitive data can be discovered by way of various techniques/mechanisms. For instance, instances of sensitive data can be discovered based on class data fields in an application marked as including sensitive data.
  • the production data can also be searched to locate known sensitive data, subsets of sensitive data, and variations of sensitive data. For example, regular expression or context free grammar searching can be performed to locate instances of data that match a particular form or format (e.g., social security numbers, phone numbers).
  • production data is cleansed of discovered sensitive data.
  • the production data can be cleansed by replacing the sensitive data with non-sensitive data.
  • sensitive data can be masked to conceal the sensitive data.
  • a hash algorithm can be applied to sensitive data to generate a hash value as the mask, and the sensitive data can be replaced with the mask.
  • sensitive data can be masked by encrypting the sensitive data.
  • the sensitive data can be overwritten with an arbitrary value.
  • the metadata can include data about the sensitive data that is useful for some purpose such as diagnostics or analytics.
  • the metadata can include the length (e.g., string length) of the sensitive data and content identity. This can allow instances of the same data to be recognized as the same (identity), and other pieces of data such as length to be used in deducing a cause of a problem, for instance.
  • the metadata can be maintained in numerous ways including appending the metadata to masked sensitive data, embedding the metadata within the masked data, or saving the metadata to a store (e.g., database, memory stream...) separate from the masked data.
  • length can be appended to masked data or the masked data can be of the same length as the sensitive data.
  • the same mask can be used for the same or equivalent instances of sensitive data.
  • Fig. 6 illustrates a method of discovering and masking sensitive production data. 600.
  • instances of sensitive data are located based on a code annotation or attribute that identifies one or more locations, such as data fields, where sensitive data is known to reside.
  • the sensitive data is masked and metadata associated with it determined and retained as part of the masked data or separate from the masked data. Since instances of sensitive data are not confined to identified locations, production data is searched, at numeral 620, for other instances of the previously discovered sensitive data. If other instances are found at reference 640 ("YES"), the method proceeds back to numeral 620 where the instance is masked. If no other instances are found at 640 ("NO”), the method continues at 650.
  • production data is searched for additional sensitive data.
  • the production data can be searched with a with regular expressions or context free grammars that match patterns of data known to be sensitive such as, but not limited to, social security numbers and telephone numbers. Addresses can also be discovered with a like search or by cross correlating with mapping software.
  • the production data can also be searched for subsets and variations of previously identified sensitive data.
  • a determination is made as to whether additional sensitive data was found. If so (“YES"), the method continues at 670, where the additional sensitive data is masked and metadata maintained. Subsequently, the method loops back to reference 650. If a numeral 660, no additional data was found (“NO”), the method terminates.
  • a component may be, but is not limited to being, a process running on a processor, a processor, an object, an instance, an executable, a thread of execution, a program, and/or a computer.
  • an application running on a computer and the computer can be a component.
  • One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.
  • FIG. 7 As well as the following discussion are intended to provide a brief, general description of a suitable environment in which various aspects of the subject matter can be implemented.
  • the suitable environment is only an example and is not intended to suggest any limitation as to scope of use or functionality.
  • aspects can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. However, some, if not all aspects of the claimed subject matter can be practiced on stand-alone computers. In a distributed computing environment, program modules may be located in one or both of local and remote memory storage devices.
  • the computer 702 includes one or more processor(s) 720, memory 730, system bus 740, mass storage 750, and one or more interface components 770.
  • the system bus 740 communicatively couples at least the above system components.
  • the computer 702 can include one or more processors 720 coupled to memory 730 that execute various computer executable actions, instructions, and or components stored in memory 730.
  • the processor(s) 720 can be implemented with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein.
  • a general-purpose processor may be a microprocessor, but in the alternative, the processor may be any processor, controller, microcontroller, or state machine.
  • the processor(s) 720 may also be implemented as a combination of computing devices, for example a combination of a DSP and a microprocessor, a plurality of microprocessors, multi-core processors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
  • the computer 702 can include or otherwise interact with a variety of computer-readable media to facilitate control of the computer 702 to implement one or more aspects of the claimed subject matter.
  • the computer-readable media can be any available media that can be accessed by the computer 702 and includes volatile and nonvolatile media, and removable and non-removable media.
  • Computer-readable media can comprise computer storage media and communication media.
  • Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data.
  • Computer storage media includes memory devices (e.g., random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM)%), magnetic storage devices (e.g., hard disk, floppy disk, cassettes, tape%), optical disks (e.g., compact disk (CD), digital versatile disk (DVD)...), and solid state devices (e.g., solid state drive (SSD), flash memory drive (e.g., card, stick, key drive)...), or any other like mediums that can be used to store, as opposed to transmit, the desired information accessible by the computer 702. Accordingly, computer storage media excludes modulated data signals.
  • RAM random access memory
  • ROM read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • magnetic storage devices e.
  • Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
  • modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
  • communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.
  • Memory 730 and mass storage 750 are examples of computer-readable storage media.
  • memory 730 may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory%) or some combination of the two.
  • the basic input/output system (BIOS) including basic routines to transfer information between elements within the computer 702, such as during start-up, can be stored in nonvolatile memory, while volatile memory can act as external cache memory to facilitate processing by the processor(s) 720, among other things.
  • Mass storage 750 includes removable/non-removable, volatile/non-volatile computer storage media for storage of large amounts of data relative to the memory 730.
  • mass storage 750 includes, but is not limited to, one or more devices such as a magnetic or optical disk drive, floppy disk drive, flash memory, solid-state drive, or memory stick.
  • Memory 730 and mass storage 750 can include, or have stored therein, operating system 760, one or more applications 762, one or more program modules 764, and data 766.
  • the operating system 760 acts to control and allocate resources of the computer 702.
  • Applications 762 include one or both of system and application software and can exploit management of resources by the operating system 760 through program modules 764 and data 766 stored in memory 730 and/or mass storage 750 to perform one or more actions. Accordingly, applications 762 can turn a general-purpose computer 702 into a specialized machine in accordance with the logic provided thereby.
  • management system 100 can be, or form part, of an application 762, and include one or more modules 764 and data 766 stored in memory and/or mass storage 750 whose functionality can be realized when executed by one or more processor(s) 720.
  • the processor(s) 720 can correspond to a system on a chip (SOC) or like architecture including, or in other words integrating, both hardware and software on a single integrated circuit substrate.
  • the processor(s) 720 can include one or more processors as well as memory at least similar to processor(s) 720 and memory 730, among other things.
  • Conventional processors include a minimal amount of hardware and software and rely extensively on external hardware and software.
  • an SOC implementation of processor is more powerful, as it embeds hardware and software therein that enable particular functionality with minimal or no reliance on external hardware and software.
  • the continuous protection component 136 and/or associated functionality can be embedded within hardware in a SOC architecture.
  • the computer 702 also includes one or more interface components 770 that are communicatively coupled to the system bus 740 and facilitate interaction with the computer 702.
  • the interface component 770 can be a port (e.g., serial, parallel, PCMCIA, USB, FireWire%) or an interface card (e.g., sound, video%) or the like.
  • the interface component 770 can be embodied as a user input/output interface to enable a user to enter commands and information into the computer 702, for instance by way of one or more gestures or voice input, through one or more input devices (e.g., pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, camera, other computer).
  • pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, camera, other computer.
  • the interface component 770 can be embodied as an output peripheral interface to supply output to displays (e.g., LCD, LED, plasma%), speakers, printers, and/or other computers, among other things. Still further yet, the interface component 770 can be embodied as a network interface to enable communication with other computing devices (not shown), such as over a wired or wireless communications link.
  • displays e.g., LCD, LED, plasma
  • the interface component 770 can be embodied as a network interface to enable communication with other computing devices (not shown), such as over a wired or wireless communications link.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Storage Device Security (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Claims (9)

  1. Procédé mis en œuvre par ordinateur comprenant :
    la production, par une application de production (140) s'exécutant dans un environnement de production, de données de production (130) correspondant à un ou plusieurs instantanés ou vidages de mémoire qui représentent l'état de mémoire de l'application de production à un instant particulier ;
    l'identification (610) dans les données de production, par un premier composant exécuté, d'un ou plusieurs emplacements d'instances de données de production sensibles sur la base d'une annotation de code dans l'application de production ;
    la recherche (630), par le premier composant exécuté, des données de production produites par l'application de production pour d'autres instances des données de production sensibles ;
    la génération, par le premier composant exécuté, d'un masque pour les données de production sensibles qui dissimule les données de production sensibles ;
    le maintien de métadonnées concernant les données de production sensibles ; et
    le masquage (620), par le premier composant exécuté, des instances des données de production sensibles dont l'emplacement a été identifié et des autres instances des données de production sensibles trouvées par la recherche en remplaçant les données de production sensibles par le masque.
  2. Procédé selon la revendication 1, le maintien de métadonnées comprend l'ajout des métadonnées au masque.
  3. Procédé selon la revendication 1, le maintien de métadonnées comprend le codage des métadonnées dans le masque.
  4. Procédé selon la revendication 1, le maintien de métadonnées comprend le stockage des métadonnées séparément par rapport au masque.
  5. Procédé selon la revendication 1, le maintien de métadonnées comprend la capture d'une longueur des données.
  6. Système comprenant :
    un processeur couplé à une mémoire, le processeur étant configuré pour exécuter le composant exécutable par ordinateur suivant stocké dans la mémoire :
    un premier composant configuré pour découvrir dans des données de production (130) produites par une application de production (104), des instances de données de production sensibles sur la base d'une ou plusieurs annotations de code (610) spécifiées dans l'application de production (140), et configuré pour chercher des données de production (630) produites par l'application de production pour d'autres instances des données de production sensibles découvertes ;
    dans lequel l'application de production s'exécute dans un environnement de production, et les données de production (130) correspondent à un ou plusieurs instantanés ou vidages de mémoire qui représentent l'état de mémoire de l'application de production à un instant particulier ; et
    dans lequel le premier composant est également configuré pour :
    générer un masque pour les données de production sensibles qui dissimule les données de production sensibles,
    maintenir des métadonnées concernant les données de production sensibles ; et
    masquer (620) les instances découvertes des données de production sensibles et les autres instances des données de production sensibles trouvées par la recherche en remplaçant les données de production sensibles par le masque.
  7. Système selon la revendication 6 comprend en outre un second composant configuré pour découvrir les données de production sensibles sur la base d'un modèle qui décrit des données sensibles.
  8. Système selon la revendication 6 comprend en outre un second composant configuré pour découvrir un sous-ensemble de données de production sensibles précédemment découvertes.
  9. Système selon la revendication 6, les données de production sensibles comprennent au moins une partie d'un artefact de diagnostic.
EP14819183.6A 2013-12-08 2014-12-01 Gestion de données sensibles de production Active EP3077953B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US14/100,007 US10325099B2 (en) 2013-12-08 2013-12-08 Managing sensitive production data
PCT/US2014/067842 WO2015084689A1 (fr) 2013-12-08 2014-12-01 Gestion de données de production sensibles

Publications (2)

Publication Number Publication Date
EP3077953A1 EP3077953A1 (fr) 2016-10-12
EP3077953B1 true EP3077953B1 (fr) 2021-06-23

Family

ID=52146722

Family Applications (1)

Application Number Title Priority Date Filing Date
EP14819183.6A Active EP3077953B1 (fr) 2013-12-08 2014-12-01 Gestion de données sensibles de production

Country Status (4)

Country Link
US (1) US10325099B2 (fr)
EP (1) EP3077953B1 (fr)
CN (2) CN114154190A (fr)
WO (1) WO2015084689A1 (fr)

Families Citing this family (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2523759A (en) * 2014-03-04 2015-09-09 Ibm Method for processing of restricted data
EP3420475B1 (fr) * 2016-02-22 2020-11-18 Tata Consultancy Services Limited Systèmes et procédés permettant de calculer un compromis de confidentialité et d'utilité de données
FR3061389B1 (fr) 2016-12-22 2019-05-31 Airbus Defence And Space Sas Systeme et procede de communication unidirectionnel
WO2018161302A1 (fr) * 2017-03-09 2018-09-13 西门子公司 Procédé, dispositif et système de traitement de données
US20180276410A1 (en) * 2017-03-21 2018-09-27 O.C. Tanner Company System and Method for Providing Secure Access to Production Files in a Code Deployment Environment
US20180276398A1 (en) * 2017-03-21 2018-09-27 O.C. Tanner Company System and method for providing restricted access to production files in a code deployment environment
US10552633B2 (en) * 2017-03-23 2020-02-04 International Business Machines Corporation Privacy annotation from differential analysis of snapshots
US11468186B2 (en) * 2017-10-30 2022-10-11 Equifax Inc. Data protection via aggregation-based obfuscation
EP3480821B1 (fr) * 2017-11-01 2022-04-27 Icon Clinical Research Limited Sécurité des données d'un réseau de support d'essai clinique
CN107944283B (zh) * 2017-11-15 2021-01-01 中国农业银行股份有限公司 一种数据敏感性识别方法及装置
US10795801B2 (en) 2018-01-08 2020-10-06 Ozcode Ltd Time travel source code debugger incorporating asynchronous collaboration
JP2020510890A (ja) * 2018-02-12 2020-04-09 イージーサーティ インコーポレイテッド データ順序を利用した原本データ確認方法及びシステム
US10481998B2 (en) 2018-03-15 2019-11-19 Microsoft Technology Licensing, Llc Protecting sensitive information in time travel trace debugging
US10541042B2 (en) 2018-04-23 2020-01-21 Microsoft Technology Licensing, Llc Level-crossing memory trace inspection queries
US10740219B2 (en) 2018-04-27 2020-08-11 Workman Nydegger Selectively tracing portions of computer process execution
US10747645B2 (en) 2018-04-27 2020-08-18 Microsoft Technology Licensing, Llc Selectively tracing portions of computer process execution
US10698792B2 (en) 2018-05-02 2020-06-30 Microsoft Technology Licensing, Llc Execution control with cross-level trace mapping
US11157563B2 (en) * 2018-07-13 2021-10-26 Bank Of America Corporation System for monitoring lower level environment for unsanitized data
US11222132B2 (en) 2018-10-05 2022-01-11 Optum, Inc. Methods, apparatuses, and systems for data rights tracking
US11157645B2 (en) * 2018-11-01 2021-10-26 International Business Machines Corporation Data masking with isomorphic functions
US11741253B2 (en) 2019-01-31 2023-08-29 Hewlett Packard Enterprise Development Lp Operating system service sanitization of data associated with sensitive information
US10521605B1 (en) 2019-03-15 2019-12-31 ZenPayroll, Inc. Tagging and auditing sensitive information in a database environment
US11200338B2 (en) 2019-03-15 2021-12-14 ZenPayroll, Inc. Tagging and auditing sensitive information in a database environment
US11062043B2 (en) 2019-05-01 2021-07-13 Optum, Inc. Database entity sensitivity classification
US10970415B2 (en) * 2019-07-18 2021-04-06 International Business Machines Corporation Sensitive data redaction in memory dump
US10929307B2 (en) 2019-07-18 2021-02-23 International Business Machines Corporation Memory tagging for sensitive data redaction in memory dump
US11288397B2 (en) 2019-09-03 2022-03-29 International Business Machines Corporation Masking text data for secure multiparty computation
US11983094B2 (en) 2019-12-05 2024-05-14 Microsoft Technology Licensing, Llc Software diagnostic context selection and use
US11669571B2 (en) 2020-03-17 2023-06-06 Optum, Inc. Predicted data use obligation match using data differentiators
US20220222372A1 (en) * 2021-01-14 2022-07-14 Capital One Services, Llc Automated data masking with false positive detection and avoidance
US11652721B2 (en) * 2021-06-30 2023-05-16 Capital One Services, Llc Secure and privacy aware monitoring with dynamic resiliency for distributed systems
CN114491609A (zh) * 2022-03-31 2022-05-13 深圳瑞力网科技有限公司 一种数据安全管理方法及系统

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130185803A1 (en) * 2012-01-18 2013-07-18 Vmware, Inc. Marking and obscuring sensitive values in traces

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7228503B1 (en) * 2000-08-25 2007-06-05 Taiwan Semiconductor Manufacturing Co., Ltd. Method for remote mask data jobview through a network
US7757278B2 (en) 2001-01-04 2010-07-13 Safenet, Inc. Method and apparatus for transparent encryption
US6823203B2 (en) 2001-06-07 2004-11-23 Koninklijke Philips Electronics N.V. System and method for removing sensitive data from diagnostic images
DE10253676B4 (de) 2002-11-18 2008-03-27 Siemens Ag Verfahren und Vorrichtung für die Fernübertragung sensibler Daten
US20060005017A1 (en) 2004-06-22 2006-01-05 Black Alistair D Method and apparatus for recognition and real time encryption of sensitive terms in documents
US8639896B2 (en) 2006-08-02 2014-01-28 International Business Machines Corporation Locating and altering sensitive information in core dumps
US8271417B2 (en) 2007-10-19 2012-09-18 Oracle International Corporation Health meter
US20090132419A1 (en) 2007-11-15 2009-05-21 Garland Grammer Obfuscating sensitive data while preserving data usability
US8069053B2 (en) 2008-08-13 2011-11-29 Hartford Fire Insurance Company Systems and methods for de-identification of personal data
US8156159B2 (en) 2009-02-11 2012-04-10 Verizon Patent And Licensing, Inc. Data masking and unmasking of sensitive data
US8375224B2 (en) * 2009-11-10 2013-02-12 Oracle International Corporation Data masking with an encrypted seed
US9401893B2 (en) 2009-12-29 2016-07-26 International Business Machines Corporation System and method for providing data security in a hosted service system
WO2013101723A1 (fr) 2011-12-27 2013-07-04 Wellpoint, Inc. Procédé et système d'appariement de schéma de données, de masquage et d'élimination de données sensibles

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130185803A1 (en) * 2012-01-18 2013-07-18 Vmware, Inc. Marking and obscuring sensitive values in traces

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"Spring Insight Developer", 16 October 2012 (2012-10-16), XP055226483, Retrieved from the Internet <URL:http://pubs.vmware.com/vfabric52/topic/com.vmware.ICbase/PDF/vfabric-tc-server-insight-dev-1.8.3.pdf> [retrieved on 20151106] *

Also Published As

Publication number Publication date
EP3077953A1 (fr) 2016-10-12
US10325099B2 (en) 2019-06-18
WO2015084689A1 (fr) 2015-06-11
US20150161397A1 (en) 2015-06-11
CN105814580A (zh) 2016-07-27
CN114154190A (zh) 2022-03-08

Similar Documents

Publication Publication Date Title
EP3077953B1 (fr) Gestion de données sensibles de production
US11507671B1 (en) Detection and healing of vulnerabilities in computer code
Ji et al. Rain: Refinable attack investigation with on-demand inter-process information flow tracking
US20200074084A1 (en) Privacy-preserving component vulnerability detection and handling
EP3765964B1 (fr) Protection d&#39;informations sensibles lors d&#39;un débogage de trace de déplacement temporel
US20210149788A1 (en) Software diagnosis using transparent decompilation
Carata et al. A primer on provenance
JP2022537300A (ja) 個人のヘルスケアデータを用いて計算するためのシステムおよび方法
Livshits Dynamic taint tracking in managed runtimes
US20210173760A1 (en) Software diagnostic context selection and use
Inam et al. Sok: History is a vast early warning system: Auditing the provenance of system intrusions
Cheney et al. An analytical survey of provenance sanitization
Roussev Digital forensic science: issues, methods, and challenges
Alqahtany et al. ForensicTransMonitor: A Comprehensive Blockchain Approach to Reinvent Digital Forensics and Evidence Management
Lawall et al. WYSIWIB: exploiting fine‐grained program structure in a scriptable API‐usage protocol‐finding process
Stamatogiannakis et al. Prov 2r: practical provenance analysis of unstructured processes
Ritzdorf et al. Assisted deletion of related content
Hallé et al. Decentralized enforcement of artifact lifecycles
Abreu et al. Provenance Segmentation.
Khan et al. Detecting wake lock leaks in android apps using machine learning
Rupprecht et al. POSTER: Identifying dynamic data structures in malware
Shu et al. Burn After Reading: Expunging Execution Footprints of Android Apps
Kavousi et al. SemFlow: Accurate Semantic Identification from Low-Level System Data
Stamatogiannakis High-Fidelity Provenance: Exploring the Intersection of Provenance and Security
Griffis et al. A platform for expressive and secure data sharing with untrusted third parties

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20160531

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20190321

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20210128

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602014078333

Country of ref document: DE

Ref country code: AT

Ref legal event code: REF

Ref document number: 1404926

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210715

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210923

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

RAP4 Party data changed (patent owner data changed or rights of a patent transferred)

Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1404926

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210623

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210923

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210924

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211025

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602014078333

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20220324

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20211231

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20211201

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20211201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20211231

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20211231

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20211231

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20141201

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230505

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20231121

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20231121

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231122

Year of fee payment: 10

Ref country code: DE

Payment date: 20231121

Year of fee payment: 10

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210623