Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
  • G06F16/33—Querying
  • G06F16/3331—Query processing
  • G06F16/334—Query execution
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
  • G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
  • G06F21/55—Detecting local intrusion or implementing counter-measures
  • G06F21/552—Detecting local intrusion or implementing counter-measures involving long-term monitoring or reporting
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
  • G06Q10/00—Administration; Management

Definitions

• the invention relates to detection of confidential information.
• Confidential information may include sensitive financial data or any information that can be used to identify specific individuals and relate them to the contents of a file.
• Some examples of confidential information include names, addresses, telephone numbers, social security numbers, and credit card numbers.
• a method for reading stored data includes: identifying strings within the stored data, where each string includes a sequence of consecutive bytes which all have values that are in a predetermined subset of possible values; for each of at least some of the strings, determining if the string includes bytes representing one or more format matches, wherein a format match includes a set of values that match a predetermined format associated with confidential information; for each format match, testing the values that match the predetermined format with a set of rules associated with the confidential information to determine whether the format match is an invalid format match that includes one or more invalid values; and calculating a score for the stored data, based at least in part upon the ratio of a count of invalid format matches to a count of other format matches.
• a system for detecting confidential information includes a data storage device; and a runtime environment connected to the data storage device.
• the runtime environment is configured to: read stored data from the data storage device; identify strings within the stored data, where each string includes a sequence of consecutive bytes which all have values that are in a predetermined subset of possible values; for each of at least some of the strings, determine if the string includes bytes representing one or more format matches, wherein a format match includes a set of values that match a predetermined format associated with confidential information; for each format match, test the values that match the predetermined format with a set of rules associated with the confidential information to determine whether the format match is an invalid format match that includes one or more invalid values; and calculate a score for the stored data, based at least in part upon the ratio of a count of invalid format matches to a count of other format matches .
• a computer-readable medium stores a computer program for detecting confidential information.
• the computer program includes instructions for causing a computer to: read stored data; identify strings within the stored data, where each string includes a sequence of consecutive bytes which all have values that are in a predetermined subset of possible values; for each of at least some of the strings, determine if the string includes bytes representing one or more format matches, wherein a format match includes a set of values that match a predetermined format associated with confidential information; for each format match, test the values that match the predetermined format with a set of rules associated with the confidential information to determine whether the format match is an invalid format match that includes one or more invalid values; and calculate a score for the stored data, based at least in part upon the ratio of a count of invalid format matches to a count of other format matches.
• the confidential information may be a credit card number.
• a format match may be determined to occur when the number of bytes with values representing digits detected in the string is equal to a number of digits in a standard format for credit card numbers.
• the rules associated with credit card numbers may include specification of a list of valid issuer identification numbers.
• the rules associated with credit card numbers may include specification of a check sum algorithm.
• the confidential information may be a social security number.
• a format match may be determined to occur when the number of bytes with values representing digits detected in the string is equal to nine.
• the rules associated with social security numbers may include specification of a valid subset of values for the number represented by the first five digits of the social security number.
• the confidential information may be a telephone number.
• a format match may be determined to occur when the number of bytes with values representing digits detected in the string is equal to ten or the number of digits detected in the string is equal to eleven digits with the first digit being "1".
• the rules associated with telephone numbers may include specification of a list of valid area codes.
• the rules associated with telephone numbers may include specification that the first digit after the area code must not be a one or a zero.
• the confidential information may be a zip code.
• a format match may be determined to occur when a sequence of bytes is detected consisting of either five bytes with values representing digits or ten bytes with values representing nine digits with a hyphen between the fifth and sixth digits.
• the rules associated with telephone numbers may include specification of a list of valid five digit zip codes.
• determining if the string includes one or more words that match a name wherein a word is sequence of consecutive bytes within a string that all have values representing alpha-numeric characters, and a name is a sequence of characters from a list of such sequences that are commonly used to refer to individual people; and calculating a score for the stored data, based at least in part upon the a count of names detected in the stored data.
• the list of names may be divided into two subsets: first names and last names.
• the names in the list may each have frequency count associated with them and the average frequency count for the names occurring in the stored data may be calculated and the score for the stored data may be calculated based at least in part upon the average frequency count.
• the average frequency count may be disregarded if the number of names detected in the stored data is less than a threshold.
• For each string counting the number of words consisting of two letters, wherein a word is sequence of consecutive bytes within a string that all have values representing alpha-numeric characters. For each two letter word, determining if the two letter word is a valid state abbreviation; and calculating a score for the stored data based at least in part upon the count of valid state abbreviations and the count of two letter words.
• determining if the string includes one or more state/zip pairs wherein state/zip pairs are sequences of characters consisting of a state abbreviation followed by a space which in turn is followed by a zip code; and calculating a score for the stored data, based at least in part upon the a count of state/zip pairs detected.
• the subset of byte values that define strings may represent alphanumeric characters, parentheses, hyphen, and space.
• Fig 1 is a block diagram of a system for detecting confidential information in stored data.
• Fig. 2 is a block diagram of software used to detect confidential information in stored data.
• Fig. 3 is a flow chart of a process for calculating a score indicative of the likelihood that a file contains confidential information. DESCRIPTION
• a system for detecting confidential information can automatically detect potential confidential data, which can then be reviewed in whole or in part by human agents.
• human review of the confidential data might be limited to cleared personnel with scarce time or avoided entirely, thus reducing or eliminating the invasion of privacy caused by the mishandling of confidential information.
• Fig. 1 depicts an exemplary system for detecting confidential information in electronically stored data.
• the data of interest may be stored in one or more data storage devices, such as a parallel "multifile" 110 implemented on multiple devices in a parallel file system (e.g., as described in U.S. 5,897,638, incorporated herein by reference) or a database server 120.
• the confidential information detection (CID) system 100 uses software executed in a runtime environment 150 to analyze stored data in the data storage device or devices. Results of the analysis, including scores for each unit of stored data, such as a file, and possibly flags indicating which units of stored data are likely to contain confidential information, may be written to the same 110, 120 or other data storage devices 160, 170.
• the user interface 180 may be used by an operator to configure and control execution of the CID system as well as to review the results.
• the runtime environment 150 may be hosted on one or more general-purpose computers under the control of a suitable operating system, such as the UNIX operating system.
• the runtime environment 150 can include a multiple-node parallel computing environment including a configuration of computer systems using multiple central processing units (CPUs), either local (e.g., multiprocessor systems such as SMP computers), or locally distributed (e.g., multiple processors coupled as clusters or MPPs), or remotely, or remotely distributed (e.g., multiple processors coupled via LAN or WAN networks), or any combination thereof.
• CPUs central processing units
• the input, output or intermediate data sets that are accessed by the runtime environment 150 can be a parallel "multifile" stored in a parallel file system (e.g., the data store 160, or in an external data storage 170 coupled to the system 100 locally or remotely over a communication link).
• a parallel file system e.g., the data store 160, or in an external data storage 170 coupled to the system 100 locally or remotely over a communication link.
• Fig. 2 depicts a structure for software that may be executed in the runtime environment to implement a system for detecting confidential information in electronically stored files.
• the CID system 100 treats all file formats as unknown and searches for strings of characters that contain confidential data.
• a file is read from a data storage device 201.
• the string extraction module 210 treats the file as a sequence of bytes of data.
• the approach for identifying strings is to remove all bytes except bytes representing characters that are used in data representing confidential information, or the common formatting of the data representing confidential information.
• Bytes are typically eight bits long, but may be defined to be an arbitrary size suited to the character set sought to be detected. For example, a byte might be defined to be sixteen or thirty-two bits in length.
• the example system depicted uses a byte size of eight bits.
• a subset of the possible byte values associated with the characters of interest is used to identify the strings. Bytes with values outside of the subset are treated as string delimiters.
• the byte values in the subset are the ASCII representations of alphanumeric characters, parentheses, hyphen, and space. Parentheses, hyphen, and space are included because these characters are commonly used to format things like telephone numbers, SSNs, credit cards, and between words in an address.
• Strings identified by the string extraction module 210 are passed to the various confidential information detection modules 220, 230, 240, 250, 260, and 270.
• the credit card number detection module 220 searches each string for a number and checks that number against a set of rules associated with credit card numbers. These rules include the specification of one or more allowed credit card number lengths measured in number of digits. The module starts by searching the string for bytes representing digits. If the number of digits detected in the string is equal to the length in digits of a standardized format for credit card numbers, a basic format match is declared. Basic format matches may occur even when bytes representing other characters are interspersed between the digits representing the number, such as spaces between groups of the digits. For each basic format match, the number represented by digits in the string is tested using the full set of rules associated with credit card numbers.
• Other rules in the full set may include, for example, a specification of valid issuer identification numbers or a valid check sum. If the number fails to satisfy any of the rules, it is identified as an invalid, or look-a-like credit card number.
• the credit card number detection module 220 counts the number of basic format matches and the number of these that are determined to be invalid.
• the ratio of the count of invalid numbers to the count of other basic format matches may be related to the likelihood that the other basic format matches are in fact valid credit card numbers. In the example system, this ratio is used to weight the count of complete format matches in calculating scores.
• a basic format match for a credit card number may be declared only when a string includes an uninterrupted sequence of bytes, each with values representing a digit, that has a length equal to the length in digits of a standardized format for credit card numbers.
• the social security number detection module 230 searches each string for a number and checks that number against a set of rules associated with social security numbers. These rules include the specification that social security numbers must be nine digits in length. The module starts by searching the string for bytes representing digits. If the number of digits detected in the string is equal to nine, a basic format match is declared. Basic format matches may occur even when bytes representing other characters are interspersed between the digits representing the number, such as hyphens between groups of the digits. For each basic format match, the number represented by digits in the string is tested using the full set of rules associated with social security numbers.
• Other rules in the full set may include, for example, specification of a valid subset of values for the number represented by the first five digits of the social security number. If the number fails to satisfy any of the rules, it is identified as an invalid, or look-a-like social security number.
• the social security number detection module 230 counts the number of basic format matches and the number of these that are determined to be invalid. The ratio of the count of invalid numbers to the count of other basic format matches may be related to the likelihood that the other basic format matches are in fact valid social security numbers. In the example system, this ratio is used to weight the count of complete format matches in calculating scores.
• the telephone number detection module 240 searches each string for a number and checks that number against a set of rules associated with telephone numbers.
• These rules include the specification that phone numbers be either ten digits in length or eleven digits in length with the first digit equal to a one.
• the module starts by searching the string for bytes representing digits. If the number of digits detected in the string is equal to ten or it is eleven and the first digit is a one, a basic format match is declared. Basic format matches may occur even when bytes representing other characters are interspersed between the digits representing the number, such as parentheses around the area code digits or hyphens between groups of digits. For each basic format match, the number represented by digits in the string is tested using the full set of rules associated with telephone numbers.
• Other rules in the full set may include, for example, a specification of valid area codes or that the first digit after the area code must not be a one or a zero. If the number fails to satisfy any of the rules, it is identified as an invalid, or look-a-like telephone number.
• the telephone number detection module 240 counts the number of basic format matches and the number of these that are determined to be invalid.
• the ratio of the count of invalid numbers to the count of other basic format matches may be related to the likelihood that the other basic format matches are in fact valid telephone numbers. In the example system, this ratio is used to weight the count of complete format matches in calculating scores.
• a basic format match for a telephone number may be declared only when a string includes an uninterrupted sequence of bytes with values representing one of the following sequences:
• the name detection module 250 searches each string for words that match names from a list of common names.
• a word is a contiguous sequence of bytes that all represent letters.
• Such a list of common names may be derived from a government census.
• the list of names may be split into first names and last names. Certain names that are known to alias as commonly used words maybe excluded to lower the chances of false positives.
• the list of names might be customized to exclude the names of the months and days of the week.
• Another method to compensate for false positives is to monitor the average frequency of the names that occur in the file. For example, the United States census provides a frequency count for each name.
• the name detection module 250 searches each string for first names and last names from a list of common names with associated frequencies. It also detects when a first name occurs immediately before a last name in the same string and counts such an occurrence as a full name. The module 250 outputs a count of first names, a count of last names, a count of full names, and an average frequency for all that names occurring in a file.
• the street address detection module 260 searches each string for sequences of words that include a number followed by one or two words consisting of letters which in turn is followed by a recognized street abbreviation.
• the number at the beginning of the street address must start with a contiguous sequence of digits, with the first digit not equal to zero. This number may have an optional letter at the end before the space preceding the one or two words of the street name.
• the street address detection module passes a count of the number of street address sequences detected in the file to the scoring module.
• the state and zip code detection module 270 searches each string for zip codes and recognized two character state abbreviations. It also counts occurrences of two word sequences consisting of a state abbreviation followed by a valid zip code. The module counts all two letter words and checks whether each two letter word is a valid state name abbreviation, as specified by the United States Post Office. The module 270 also searches each string for numbers and checks that number against a set of rules associated with zip codes. These rules include the specification that zip codes be either a sequence of contiguous digits that is five digits in length or sequence that is nine digits in length with a hyphen between the fifth and sixth digits. If a sequence of bytes matching either of these patterns is detected, a basic format match is declared.
• the number represented by digits in the string is tested using the full set of rules associated with zip codes.
• Other rules in the full set may include, for example, a specification of valid five digit zip codes as a subset of all possible five digit numbers. If the number fails to satisfy any of the rules, it is identified as an invalid, or look-a-like zip code.
• the module 270 counts the number of basic format matches and the number of these that are determined to be invalid.
• the ratio of the count of invalid numbers to the count of other basic format matches may be related to the likelihood that the other basic format matches are in fact valid zip codes. In the example system, this ratio is used to weight the count of complete format matches in calculating scores.
• the module 270 finally counts the number of state and zip sequences which consist of a valid state abbreviation, followed by a space, which is followed by a valid zip code. The module 270 then passes the count of two letter words, the count of valid state abbreviations, the count of zip code format matches, the count of valid zip codes, and the count of state and zip code sequences to the scoring module.
• Each file processed by the CID system is assigned a score and, depending on that score may be flagged as potentially containing confidential information.
• the scoring module 280 calculates the score for the file based on the outputs of the confidential information detection modules 220, 230, 240, 250, 260, and 270. The score may be saved or output from the CID system 100. The score is also passed to the thresholding module 290 which compares the score to a threshold and flags the file if its score exceeds the threshold.
• While the confidential information detection modules are depicted in Fig. 2 as operating independently on the strings, it should be understood that greater efficiency might be achieved in some cases by sharing intermediate processing results for strings between some of the detection modules. For example, the results of a routine that counts the bytes in a string that represent digits could be shared by the credit card number detection module 220, the social security number detection module 230, and the telephone number detection module 240. Also, many of the detection modules might be optimized by ignoring strings with less than the minimum number of bytes required to match the format for the confidential information it is searching for.
• the module boundaries illustrated are intended to convey an understanding of the logic being implemented and not to impose rigid constraints on the structure of code implementing the disclosed methods in software.
• An exemplary scoring method 300 that may be implemented in the scoring module 280 for calculating the score is depicted in Fig. 3.
• the scoring module retrieves 310 the data regarding one of the Confidential Information Types (CI Types) that has been output by one of confidential information detection modules 220, 230, 240, 250, 260, or 270.
• the scoring module then extracts 320 a hit count and false positive indicator for the CI Type.
• the hit count is the number of matches to a CI Type format in the file that have not been determined to be invalid.
• the hit count produced by the credit card number detection module is the number of basic format matches not determined to be invalid.
• the hits are complete matches to the full set of rules associated with the CI Type. It is still possible that a hit is a false positive, as random data could include complete matches even though it does not encode information of the CI Type.
• a false positive indicator is a metric used by the scoring module to assess the reliability of the associate hit count. For example, the false positive indicator produced by the credit card number detection module is the count of basic format matches that are determined to be invalid.
• invalid basic format matches are in a sense 'near misses' or 'look-a-like' credit card numbers and their presence may indicate a higher chance of false positives.
• other metrics may be used as a false positive indicator.
• the average frequency count produced by the name detection module may be used a false positive indicator for name counts.
• the scoring modules calculates 330 a false positive weighting factor based in part upon the false positive indicator. In this example, the weighting factor is a inversely proportional to the false positive indicator.
• Some false positive indicators may require other functional forms to calculate the false positive weighting factor.
• the false positive weighting factor is directly proportional to the average frequency count for names. It may also be advantageous to take additional steps to bound the values that a weighting factor can take.
• the weighting factor for names may be discontinuously bounded to take values between 0.5 and 1.
• CI Types considered in the scoring for which no false positive indicator is available.
• the street address detection module produces no false positive indicator.
• CI Types without a false positive indicator may be factored into the scoring by skipping the false positive weighting factor calculation step
• the scoring modules then calculates 340 a sub-score for each CI Type considered in the scoring.
• the sub-score is calculated as a function of the hit count, the false positive weighting factor, and a file size indicator. More precisely, the sub- score takes the form:
• the scoring module After calculating a sub-score 340, the scoring module checks 350 whether more CI Type data remains to be considered. If sub-scores have not been calculated for all the CI Types to be considered, the scoring module loops back to retrieve data from the confidential information detection module for the next CI Type. If all sub-scores have been calculated, it proceeds to calculate 360 a composite score for the file based on the sub-scores. In the example depicted, the composite score is the sum of all the sub-scores. The sub-scores may be bounded before they are added into the composite score. The resulting composite score may then be saved 370 by, for example, writing it to non- volatile memory on a data storage device such as database server 170 or parallel "multifile" system 160. The composite score may also be passed to the threshold module 290 before termination 395 of the scoring module execution.
• the CID system 100 may be configured to process a list of one or more files provided by a user of the system. In other implementations the CID system may be configured to process all files in a file system. The CID system may be configured to run periodically and in some implementations may be configured to check timestamps associated with the files in a file system and process only those files that were recently updated (e.g., last edited after a given time, such as the last time the CID system was run).
• the confidential information detection approach described above can be implemented using software for execution on a computer.
• the software forms procedures in one or more computer programs that execute on one or more programmed or programmable computer systems (which may be of various architectures such as distributed, client/server, or grid) each including at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port.
• the software may form one or more modules of a larger program, for example, that provides other services related to the design and configuration of computation graphs.
• the nodes and elements of the graph can be implemented as data structures stored in a computer readable medium or other organized data conforming to a data model stored in a data repository.
• the software may be provided on a storage medium, such as a CD-ROM, readable by a general or special purpose programmable computer or delivered (encoded in a propagated signal) over a communication medium of a network to the computer where it is executed. All of the functions may be performed on a special purpose computer, or using special-purpose hardware, such as coprocessors.
• the software may be implemented in a distributed manner in which different parts of the computation specified by the software are performed by different computers.
• Each such computer program is preferably stored on or downloaded to a storage media or device (e.g., solid state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer system to perform the procedures described herein.
• a storage media or device e.g., solid state memory or media, or magnetic or optical media
• the inventive system may also be considered to be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer system to operate in a specific and predefined manner to perform the functions described herein.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Software Systems (AREA)
• Computer Security & Cryptography (AREA)
• General Engineering & Computer Science (AREA)
• Business, Economics & Management (AREA)
• Computer Hardware Design (AREA)
• Databases & Information Systems (AREA)
• Data Mining & Analysis (AREA)
• Computational Linguistics (AREA)
• Tourism & Hospitality (AREA)
• Economics (AREA)
• Entrepreneurship & Innovation (AREA)
• General Business, Economics & Management (AREA)
• Human Resources & Organizations (AREA)
• Marketing (AREA)
• Strategic Management (AREA)
• Quality & Reliability (AREA)
• Operations Research (AREA)
• Storage Device Security (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Priority Applications (6)

Applications Claiming Priority (2)

Publications (2)

Family

ID=42076601

Family Applications (1)

Country Status (8)

Cited By (3)

Families Citing this family (29)

Family Cites Families (34)

• 2008
  • 2008-10-03 US US12/245,507 patent/US9569528B2/en active Active
• 2009
  • 2009-10-01 JP JP2011530236A patent/JP2012504920A/ja active Pending
  • 2009-10-01 CA CA2738480A patent/CA2738480C/en active Active
  • 2009-10-01 WO PCT/US2009/059240 patent/WO2010042386A2/en not_active Ceased
  • 2009-10-01 CN CN200980139094.8A patent/CN102171702B/zh active Active
  • 2009-10-01 AU AU2009302657A patent/AU2009302657B2/en active Active
  • 2009-10-01 KR KR1020157007583A patent/KR20150038738A/ko not_active Ceased
  • 2009-10-01 CN CN201510510773.9A patent/CN105426756B/zh active Active
  • 2009-10-01 EP EP09819679.3A patent/EP2344962B1/en active Active
  • 2009-10-01 KR KR1020117007587A patent/KR101627592B1/ko active Active
• 2014
  • 2014-12-08 JP JP2014247864A patent/JP5957064B2/ja active Active

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events