EP4670181A1

EP4670181A1 - DE-IDENTIFICATION OF MEDICAL IMAGES

Info

Publication number: EP4670181A1
Application number: EP24760999.3A
Authority: EP
Inventors: Mustafa Bashir; Jacob MacDonald; Xiang Li; Kyle LAFATA
Original assignee: Duke University
Current assignee: Duke University
Priority date: 2023-02-22
Filing date: 2024-02-22
Publication date: 2025-12-31
Also published as: WO2024178228A1

Abstract

A medical image de-identification model is used to identify protected health information (PHI) on a medical image for removal, for example, by blocking out text on the image. A method of training a medical image de-identification model includes creating a training data set comprising a plurality of modified medical images with corresponding keys and performing a training operation for the model using the training data set. Creating the training data set includes receiving a first plurality of medical images of text-free medical images; adding, to each of the first plurality of medical images, at least one string of text to create a modified medical image; and generating the corresponding keys. A key for a particular modified medical image includes location information of each string of text on the particular modified medical image.

Description

DE-IDENTIFICATION OF MEDICAL IMAGES

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application Serial No. 63/447,446. filed February 22. 2023.

BACKGROUND

[0002] De-identifying medical images is an essential step for repurposing clinical medical images for use in research. The medical images captured for individual patients in a medical or clinical context can be a primary source for images that are used in research. However, when images are captured and processed in a clinical setting, each individual medical image may become heavily linked with the patient, accumulating a variety of instances of the patient’s identifying information (e.g., protected health information (PHI)). While it is desirable to have very thorough and rigid identification for the generated medical images that are used in a medical or clinical context (e g., identifiers for the patient, the scan instance, the instrument identification, etc.), when the same images are being selected for use for research it is required by a number of regulations and laws that all of the patient’s identifying information be removed from the medical images.

[0003] Therefore, there is a need for an accurate and efficient method of deidentifying medical images captured in a clinical setting, which may be heavily laden with identifying information, so that they can be used in research, shared, or otherwise publicly disclosed in a manner that satisfies data privacy requirements and regulations.

BRIEF SUMMARY

[0004] Systems and techniques for image de-identification are described. Application of the described systems and techniques for image de-identification enable patient images to be repurposed for use in research while satisfying data privacy requirements and regulations. The described techniques and systems provide a trained medical image de-identification model that is used to remove protected health information (PHI) from medical images without removing medically relevant objects and text.

[0005] A method of performing image de-identification includes detecting one or more objects on a medical image, where the one or more objects are groupings of pixels that are identified by a medical image de-identification model as having a value above a confidence threshold for representing characters; and removing at least one of the one or more detected objects from the medical image to create a de-identified medical image. The at least one of the one or more objects on the medical image can include PHI.

[0006] A method of training a medical image de-identification model includes creating a training data set comprising a plurality of modified medical images with corresponding keys and performing a training operation for the medical image deidentification model using the training data set of the plurality of modified medical images and corresponding keys. Creating the training data set includes receiving a first plurality of medical images, wherein each of the first plurality of medical images are text-free; adding, to each of the first plurality of medical images, at least one string of text to create a modified medical image; and generating the corresponding keys. A key for a particular modified medical image includes location information of each of the at least one string of text on the particular modified medical image and may include the content of each corresponding string of text added to the particular modified medical image.

[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 illustrates a high-level process flow diagram for the deidentification of electronic medical images.

[0009] Figure 2A illustrates a method for creating a training data set for a medical image de-identification model for the recognition of text on medical images.

[0010] Figure 2B illustrates a representative diagram of the method of Figure 2A.

[0011] Figure 3 illustrates an example training process for generating a medical image de-identification model.

[0012] Figure 4 illustrates an example method of performing an image de- identification process on a medical image.

[0013] Figure 5 illustrates an example process flow for filtering and de-identifying a plurality of image modalities.

[0014] Figure 6 illustrates components of example computing systems that may carry out the described processes. DETAILED DESCRIPTION

[0015] Systems and techniques for image de-identification are described. Application of the described systems and techniques for image de-identification enable patient images to be repurposed for use in research while satisfying data privacy requirements and regulations. The described techniques and systems provide a trained medical image de-identification model that is used to remove protected health information (PHI) from medical images without removing medically relevant objects and text.

[0016] Generally, there are two areas of a medical image where identifying information can be found on a medical image. First, there is structured information, such as metadata. For example, medical imaging data is often stored in the DICOM (Digital Imaging and Communications in Medicine, also known as DCM) format, which includes extensive metadata fields intended to facilitate the identification and organization of those data. Second, there is non-structured information, such as information found on the image itself, where within the pixel array of the image, there can be pixels forming a string of text ' etched'’ directly into the image (e.g., white pixels spelling out the patient name “John Doe” across an ultrasound image). As used herein, “text” refers to alphanumeric and special characters (e.g., symbols such as &, %, etc.).

[0017] In research use, where links between the images and the patient of origin must be destroyed, the process of de-identifying images having non-structured PHI (e.g., removing the non-structured PHI) can be complex and time-consuming. It is particularly challenging to remove non-structured PHI from medical images when the non-structured information is on the medical image itself (e.g., the text on the image is superimposed, overlayed, etched-in, burnt-in, etc.). This limits the number of images that can be processed and used for research.

[0018] Figure 1 illustrates a high-level process flow diagram for the de- identification of electronic medical images. Referring to Figure 1, three examples of medical images are shown. Medical image 102, medical image 110, and medical image 120 each have a string of protected heath information (PHI) text (e.g., PHI text 104 “DOB: 03/20/96”. PHI text 112 “C. Baker”, and PHI text 122 “MRN:BBB222”) directly on the image. PHI is any information in the medical record or designated record set that can be used to identify an individual and that was created, used, or disclosed in the course of providing a heath care service. A patient’s PHI is information that is often heavily protected by various laws and/or regulations (e.g., Health Insurance Portability and Accountability Act (HIPP A), General Data Protection Regulation (GDPR), etc.). Therefore, to prepare the medical images (e.g., medical image 102, medical image 110, and medical image 120) produced in a clinical setting for use in research, any patient PHI must be removed from the medical images.

[0019] Medical image de-identification models (e.g., medical image deidentification model 106, medical image de-identification model 116, and medical image de-identification model 126) created as described herein are used (e.g., as part of process 400 described with respect to Figure 4) to obtain de-identified medical images (e.g., deidentified medical image 108, de-identified medical image 118, and de-identified medical image 128).

[0020] The medical image de-identification models (e.g.. medical image de- identification model 106. medical image de-identification model 116. and medical image de-identification model 126) are applied to identify groupings of pixels in a medical image that have a value above a confidence threshold for representing characters. In this manner objects corresponding to text can be detected in the medical images. In some cases, the medical image de-identification models are individually trained to process a particular imaging modality (e.g.. ultrasound, radiation, magnetic resonance imaging (MRI). computed tomography (CT) scan, etc ). For example, medical image de-identification model 106 may be trained to de-identify ultrasound images while medical image de- identification model 116 may be trained to de-identify MRI images.

[0021] Non-structured identifying information superimposed on medical images can be particularly difficult because the image characteristics (e.g., corresponding to contrast values associated with a particular imaging modality) of any given medical image vary considerably. Medical image characteristics that contribute to the difficulty in training a medical image de-identification model to accurately identify non-structured identifying information on medical images include, but are not limited to, image modality, subject matter (e.g., body part/imaging target), contrast levels, and bodily anomalies.

[0022] The medical image de-identification models are specifically trained as described herein to enable removal of text while minimizing affects on the other objects in the image. For example, as illustrated for the first example medical image 102 having PHI text 104 “DOB: 03/20/96”, medical image de-identification model 106 is sufficiently trained to extract text so as to enable generation of de-identified medical image 108. Here, it can be seen that the PHI text 104 is blocked out on de-identified medical image 108.

[0023] As another example, as illustrated for the second example medical image 110 having PHI text 112 “C. Baker”, medical image de-identification model 116 is sufficiently trained to extract text so as to enable generation of de-identified medical image 118 that removes the PHI text 112, but maintains bodily object 114.

[0024] Indeed, through certain embodiments of the described techniques, it is possible for the PHI text 112 “C. Baker” to be removed while the bodily object 114 is not removed, leaving the bodily object 114 part of the image that is visible and useable for research purposes. Without specific and intentional training of the medical image deidentification model 116 (e.g., as described with respect to Figures 2A and 3), it may be difficult for the model to accurately determine that bodily object 114 is not non-structured identifying information, and should not be removed, while PHI text 112 “C. Baker” is nonstructured identifying information and should be removed. Indeed, with the amount of varying image characteristics in medical images, it can be difficult for a model to identify text when a particular medical image features a unique, difficult-to-identify bodily object. However, these bodily objects are likely essential information for use of the medical image for research purposes.

[0025] As yet another example, as illustrated for the third example medical image 120 having PHI text 122 ”MRN:BBB222“. medical image de-identification model 126 is sufficiently trained to extract text so as to enable generation of de-identified medical image 128 that removes the PHI text 122, but maintains measurement text 124 “32.4mm”. Indeed, as described herein, it is possible for some text to be permitted to remain on the medical images.

[0026] As described below, a specialized medical image training data set can be created that is used to train a medical image de-identification model to identify text on medical images so that PHI can be redacted without redacting important bodily objects, medically relevant non-PHI text, and/or objects on the medical image that are otherwise not PHI.

[0027] Figure 2A illustrates a method of creating a training data set for a medical image de-identification model for the recognition of text on medical images. Figure 2B illustrates a representative diagram of the method of Figure 2A. Referring to Figures 2A and 2B, method 200 can include receiving (202) a plurality of medical images 210; adding (204), to each of the plurality of medical images, at least one string of text 212 (e.g., string of text 212a and string of text 212b) to create a modified medical image (e.g., modified medical image 214a); and generating (206) corresponding keys (e.g., corresponding key 216a, corresponding key 216b. corresponding key 216c, and corresponding key 216d) for each image of the plurality of modified medical images 214. The resulting modified medical images and corresponding keys become part of a training data set 208.

[0028] Receiving (202) the plurality of medical images 210 can include receiving a plurality of medical images where each of the plurality of medical images are text-free or have blocked out text (e.g., manually redacted or otherwise covered). A text-free medical image is an image that does not have any text on the face of the image (e.g.. no superimposed text).

[0029] In some cases, receiving (202) the plurality of medical images 210 includes receiving a plurality of medical images 210 where each image of the plurality of medical images is of the same image modality. For example, the medical images can all be ultrasound images or the medical images can all be CT images.

[0030] Adding (204), to each of the plurality of medical images 210, a string of text 212 to create a modified medical image 214 can include obtaining a string of text 212 (e.g., letters, numbers, symbols, etc.) and adding the string of text 212 onto an image of the plurality of medical images 210 by an editing program or function (e.g., an image generator such as provided as part of keras_ocr, a media labeler available as part of a python library, etc ). In some cases, obtaining the string of text includes generating the string based on a program or rules. In some cases, obtaining the string of text includes accessing and selecting from a set of predetermined strings. In some cases, obtaining the string of text includes synthetically generating text. In some cases, obtaining the string of text includes creating randomly formed strings of alphanumeric text. In some cases, each string of text 212 added to each medical image of the plurality of medical images 210 is unique. In some cases, certain strings of text 212 are reused.

[0031] The string of text 212 added (204) to each of the plurality of medical images 210 can have varying textual features, including, but not limited to character type, character length, color, orientation, font, font size, etc. to improve generalization. In some cases, the particular textual features of a string of text can be programmatically or randomly applied. In addition to the textual features of the string of text, the location of application of the string of text can be programmatically or randomly applied. In some cases, the location of the added text is the same for at least some of the images. In some cases, the location of the added text is random or in a pattern. In some cases, the location is random or in a pattern within predefined regions.

[0032] The corresponding keys 216 generated (206) for each image of the plurality of modified medical images 214 include the location information 218 of each string of text 212 on the particular modified medical image 214. In some cases, the keys 216 further include the content/value of the string of text, which can be useful for application as part of a validation data set (see e.g., Figure 3). For example, the corresponding key 216a for the modified medical image 214a identifies the string of text 212a “Jane Doe,” the location information 218a for the string of text 212a “Jane Doe,” the string of text 212b “03/20/94,” and the location information 218b for the string of text 212b “03/20/94.”

[0033] In some cases, the location information 218 included in the corresponding key 216 are bounding box coordinates of a bounding box created to border a particular string of text 212. A bounding box is a rectangular region that outlines the object (e.g., the string of text 212) in an image (e g., modified medical image 214), defining the spatial location of the object. The bounding box can be the coordinates of a rectangular border that fully encloses a digital object (e.g., text). In some cases, the bounding box representation is the (x,y) coordinates of each of the four comers of the box. In some cases, the bounding box representation is the (x,y) coordinates of the center of the bounding box rectangle, the width of the bounding box rectangle, and the height of the bounding box rectangle. In some cases, another suitable method for object detection in an image may be used as the location information 218 for the corresponding key 216.

[0034] The corresponding keys 216 for each image of the plurality of modified medical images 214 can be stored as metadata of that image and/or stored in a manner that can be retrieved in association with that image.

[0035] Figure 3 illustrates an example training process for generating a medical image de-identification model. Referring to Figure 3, a training process 300 for generating a medical image de-identification model (e.g., model 310) can include receiving a training data set 305 (e.g., training data set 208 described with respect to Figures 2A and 2B) including the plurality of modified medical images and corresponding keys; and performing a training operation 312 for the medical image de-identification model (e.g., model 310) using the training data set 305. In some cases, the medical image de- identification model is an optical character recognition (OCR) model. Any suitable machine learning algorithm and/or neural network architecture capable of performing optical character recognition may be used. For example, the model 310 can be for a convolutional neural network (CNN).

[0036] In addition to training data generated as described with respect to Figures 2A and 2B, the training data set 305 can include training data generated by other methods, including, but not limited to, marking the location of text on images already having text (e.g.. generating a key with location information of existing text locations).

[0037] The training data set 305 used for a particular model can be grouped according to imaging modality, commonality of imaged object, and/or type of text on the image, as examples.

[0038] The medical image de-identification model 310 can be generated in an iterative fashion in order to determine desired weights and/or features for the medical image de-identification model 310. In some cases, confidence thresholds associated with recognizing groupings of pixels as corresponding to a character are adjusted as part of the training process. In addition, adjustments can be made to the training data set to update and improve the models. For example, the medical image de-identification model 310 can undergo an evaluating operation 314 to check the medical image de-identification model 310 for accuracy, potential failings/shortcomings, etc. In some cases, a validation data set 316 can be used by the evaluating operation 314. The validation data set 316 may include a subset of images and keys from the training data set 305 (and/or be separately generated as described with respect to training data set 208 of Figures 2A and 2B).

[0039] The evaluating operation 314 can include checking accuracy of the model 310 trained during the training operation 312 by checking how the medical image de- identification model 310 detected/extracted text in a test image against the keys associated with the image. Results of the evaluating operation 314 can be fed back for use in the training operation 312 (e.g., to adjust weights, confidence thresholds, etc ). The evaluating operation 314 can also include categorizing errors in the results as a “false positive” or a “false negative” for further action with respect to the training data (e g., to create an updated training data set based on the results so that the medical image de-identification model can be updated using the updated training data set).

[0040] A “false positive” can occur when an object that is not text is identified as text. In some cases, a false positive may occur due to a medical image having certain image characteristics (e.g., contrast level, bodily anatomy/anomalies, etc.) or by confidence thresholds for the model that may be too low.

[0041] A “false negative” can occur when an object that is text fails to be identified as text. In some cases, a false negative may occur due to certain textual features (e.g., font, font size, orientation, etc.) or by confidence thresholds for the model that may be too high.

[0042] In some cases, information on false positives and false negatives are used to adjust the thresholds for the model (e.g., used as feedback to the training operation 312). For example, if the evaluating operation 314 results in identifying a certain number of false positives (e.g., a number or percentage above a predetermined threshold), the probability thresholds of the model are adjusted higher; and if the evaluating operation 314 results in identifying a certain number of false negatives (e.g., a number or percentage above a predetermined threshold), the probability thresholds of the model are adjusted lower.

[0043] In some cases, information on false positives and false negatives are used to generate new modified medical images to update the training data set. For example, based on the images identified as having false positives, process 200 can be performed to generate (318) additional images having similar (i.e., satisfying a similarity metric) image characteristics to medical images triggering false positives, where the plurality of medical images that are received in step 202 are images having similar image characteristics as those found in the images identified as having false positives. As another example, based on images identified as having false negatives, process 200 can be performed to generate (320) additional images having similar (i.e., satisfying a similarity metric) textual features to medical images triggering false negatives, where the at least one string of text added in step 204 involves adding a string text with similar textual features as those found in the images identified as having false negatives. A similarity' metric for image characteristics and/or textual features can be based on keypoint matching, a histogram method (on various features, texture, direction, scale, etc.), classification, feature matching, or a combination thereof, as some examples.

[0044] As an illustrative scenario, if a medical image categorized as causing a false positive is a medical image showing a small anomaly in the stomach lining of the patient, generating (318) additional images having similar image characteristics may include generating additional modified medical images based on images of the stomach lining having similar sized/shaped anomalies in similar locations on the image as determined using a histogram method.

[0045] As another illustrative scenario, if a medical image categorized as causing a false negative is a medical image having white text superimposed over an image of bone matter, generating (320) additional images having similar textual features may include generating additional modified medical images where the string of text added to images is white in color (e.g., at different orientations or font sizes). In some cases, both the underlying bodily objects and the textual features are the focus of the similarity metric. [0046] Advantageously, by specifically tailoring an updated training data set to include more images with features that may be challenging, the false positives and/or false negatives may be further reduced.

[0047] Figure 4 illustrates an example method of performing an image deidentification process on a medical image. Method 400 can be carried out by a computing system having access to medical images containing PHI 405.

[0048] Referring to Figure 4, method 400 includes detecting (410) one or more objects on a medical image (e.g., from the medical images containing PHI 405); and removing (420) at least one of the one or more detected objects from the medical image to create a de-identified medical image.

[0049] The detecting (410) of the one or more objects involves using a medical image de-identification model (e.g., model 310). That is, the one or more objects are groupings of pixels that are identified by the medical image de-identification model as having a value above a confidence threshold for representing characters. In some cases, any object detected by the model is removed in operation 420. In some cases, further evaluation is carried out. For example, additional rules may be applied to determine whether detected text is removed or retained within the image, including the nature of the text (e.g., pattern, terms), text location, and other characteristics. Thus, the method can further include applying one or more rules for determining whether detected text is removed or retained within the medical image; and allowing any of the one or more detected objects that satisfy certain criteria of the one or more rules to remain in the medical image. As an illustrative example, method 400 can further include checking (412) the one or more detected objects for permitted character patterns; and allowing (414) any of the one or more detected objects that satisfy a criterion for permitted character patterns to remain in the medical image. In addition to or as an alternative to checking for permitted character patterns, method 400 can include checking (416) the one or more detected objects for allowable terms; and allowing (418) any of the one or more detected objects that satisfy a criterion for allowable terms to remain in the medical image.

[0050] In some cases, checking (412) for permitted character patterns includes comparing the character types to allowed patterns. For example, an allowed pattern may include a one or two-digit number less than ninety, because isolated numbers less than 90 are not identifying information and their inclusion allows scale bars on images to be maintained. As another example, an allowed pattern can be numbers that are followed by a recognized unit of measurement, for example, " Ocm/s" (no space) or “20 cm/s” (with space).

[0051] In some cases, checking (416) for allowable terms includes using a dictionary of permitted terms to match extracted/identified text against. Examples of permitted terms may include, but are not limited to, terms for image and/or patient positioning (e.g., right, left, lateral, superior, etc.), and/or anatomical terms (e.g. liver, kidney, etc.).

[0052] Removing (420) at least one of the one or more detected objects can include removing all objects not marked or otherwise indicated as being allowable/permitted by operations 412, 414, 416, and 418.

[0053] In some implementations, the decision to remove or retain the detected characters is based on location of the detected object. For example, there may be regions on medical images (e.g., top banner or portion thereof) that are entirely removed regardless of whether the detected characters satisfy other criteria that would permit the text to remain.

[0054] In some cases, removing (420) at least one of the one or more detected objects includes creating a bounding box around each detected object and replacing all pixels within the bounding box. A bounding box is a rectangular region that outlines an object (e g., text) in an image defining the spatial location of the object. The bounding box can be the coordinates of a rectangular border that fully encloses a digital object (e.g., text). In some cases, the bounding box representation is the (x,y) coordinates of each of the four corners of the bounding box rectangle. In some cases, the bounding box representation is the (x,y) coordinates of the center of the bounding box rectangle, the width of the bounding box rectangle, and the height of the bounding box rectangle.

[0055] In some cases, all pixels within the bounding box are replaced with pixels having a same value (e.g., each pixel within the bounding box is replaced with pixels having a hex code value of #00000). In some cases, each pixel may be replaced with a different value (e.g., values chosen to match/coordinate with the closest value proximate to that pixel in the medical image).

[0056] Advantageously, large numbers of medical images can be automatically deidentified using the above-described techniques, which can enable more images to be included for research purposes. In addition, the corpuses of medical images do not require manual sorting as it is possible to include filtering operations, for example, as described with respect to Figure 5. [0057] Figure 5 illustrates an example process flow for filtering and de-identifying a plurality of image modalities. Referring to Figure 5, process 500 can be applied to a collection of medical images 502. Medical images are often stored in the DICOM format, which includes extensive metadata fields intended to facilitate identification and organization of the data. At step (1), the metadata of each of the medical images 502 can be cleaned using pre-determined filters created to remove fields that can contain PHI.

[0058] The metadata of the images 502 can also be parsed to identify the image type, which can be used, along with some other information such as scanner manufacturer and date of scan to determine whether the metadata-cleaned images 504 have a high probability of containing non-structured/superimposed identifying text/PHI. For example, certain image modalities (e.g.. ultrasound) may have a higher probability of containing superimposed PHI than others (e.g., MRI). In addition, even for those image modalities that have a lower probability of containing the PHI, it is possible for the text to be added later as part of a “secondary capture”. The collection of medical images 502 may contain both primary’ captures and secondary captures. A primary’ capture refers to the original image saved on the capture device (e.g.. scanner). A secondary capture refers to the image that is created when the primary capture is further processed after collection.

[0059] In some cases, process 500 includes determining (2) whether each image of the metadata-cleaned images 504 is a secondary capture image 506; and if it is determined that an image is a secondary’ capture image 506, that image can be discarded. This optional operation can be advantageous because it can reduce the total images needed to be processed. In some cases, it may be inefficient, impossible, or otherwise difficult to obtain the primary' capture; therefore, in some cases, even if at step (2) it is determined that the metadata-cleaned image 504 is a secondary capture image 506. the secondary capture image 506 may be included for processing 550 to remove the PHI data.

[0060] If, at step (2), it is determined that an image of the metadata-cleaned image 504 is a primary capture (e.g., not a secondary capture), at step (3), it can be determined whether the primary capture image is an ultrasound modality or a non-ultrasound modality (e.g.. any other medical image modality). If, at step (3) it is determined that the modality of the primary image is ultrasound, the image is considered to have a high likelihood of containing PHI and processing 550 can be performed.

[0061] Ultrasound primary capture images (unlike some other modality primary capture images) are known to be of an exceptionally high risk for PHI (e.g., conventional ultrasound images include a banner at the top of the image containing patient name, medical record number (MRN), and other instances of PHI). If, at step (3), it is determined that the modality of the primary capture image is non-ultrasound, in some cases, it can be assumed that the non-ultrasound image (e.g., medical image 508) does not contain PHI and can be considered de-identified (since the metadata has been removed). Of course, it should be understood that all images, regardless of modality may undergo processing 550. Further, when multiple types of image modalities are supported by the system, each image can be evaluated to determine modality for processing (e.g., parsing metadata for modality and directing the image to be de-identified by the appropriate model). Accordingly, step (3) can be omitted or used for directing the image to the appropriate processing/model.

[0062] The de-identification processing 550 can be carried out such as described with respect to process 400 of Figure 4 and further including determining which model of a plurality of available trained models to apply based on image modality and other potential characteristics/features of the medical images (which may be determined based on the remaining metadata of the images). For example, ultrasound image de-identification model 510 is used for generating de-identified medical image 514 at step 4A and another modality-specific image de-identification model 512 is used for generating de-identified medical image 516 at step 4B.

[0063] Figure 6 illustrates components of an example computing system that may carry out the described processes. Referring to Figure 6, system 600 may be implemented within a single computing device or distributed across multiple computing devices or subsystems that cooperate in executing program instructions. Accordingly, more or fewer elements described with respect to system 600 may be incorporated to implement a particular system. The system 600 can include one or more blade server devices, standalone server devices, personal computers, routers, hubs, switches, bridges, firewall devices, intrusion detection devices, mainframe computers, network-attached storage devices, and other types of computing devices.

[0064] In embodiments where the system 600 includes multiple computing devices, the server can include one or more communications networks that facilitate communication among the computing devices. For example, the one or more communications networks can include a local or wide area network that facilitates communication among the computing devices. One or more direct communication links can be included between the computing devices. In addition, in some cases, the computing devices can be installed at geographically distributed locations. In other cases, the multiple computing devices can be installed at a single geographic location, such as a server farm or an office.

[0065] System 600 can include processing systems 605 of one or more processors to transform or manipulate data according to the instructions of software 610 stored on a storage system 615. Examples of processors of the processing systems 605 include general purpose central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs). application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof.

[0066] The software 610 can include an operating system and instructions 620. In some cases, instructions 620 can cause the system 600 to perform some or all of method 200 as described with respect to Figures 2A and 2B. In some cases, instructions 620 can cause the system 600 to perform some or all of method 300 as described with respect to Figure 3. In some cases, instructions 620 can cause system 600 perform some or all of method 400 as described with respect to Figure 4. In some cases, instructions 620 can cause system 600 to perform some or all of process 500 as described with respect to Figure 5. In some cases, instructions 620 can include instructions for performing some or all of the described methods.

[0067] In some cases, system 600 can host a neural network 675 supporting the methods described herein. In some cases, system 600 can communicate with another computing system to execute a neural network.

[0068] In some cases, models (e.g., medical image de-identifi cation model 106, medical image de-identification model 116, medical image de-identification model 126, model 310, ultrasound image de-identification model 510, and/or modality-specific image de-identification model 512) may be stored in storage system 615.

[0069] Storage system 615 may comprise any suitable computer readable storage media. Storage system 615 may include volatile and nonvolatile memories, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of storage media of storage system 615 include random access memory, read only memory, magnetic disks, optical disks, CDs, DVDs, flash memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage media. In no case do storage media consist of transitory, propagating signals. [0070] Storage system 615 may be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems co-located or distributed relative to each other. Storage system 615 may include additional elements, such as a controller, capable of communicating with processing system 605.

[0071] Network interface 640 may include communications connections and devices that allow for communication with other computing systems over one or more communication networks (not shown).

[0072] Alternatively, or in addition, the functionality, methods and processes described herein can be implemented, at least in part, by one or more hardware modules (or logic components). For example, the hardware modules can include, but are not limited to, application-specific integrated circuit (ASIC) chips, field programmable gate arrays (FPGAs), system-on-a-chip (SoC) systems, complex programmable logic devices (CPLDs) and other programmable logic devices now known or later developed. When the hardware modules are activated, the hardware modules perform the functionality, methods and processes included within the hardware modules.

[0073] Certain Embodiments may be implemented as a computer process, a computing system, or as an article of manufacture, such as a computer program product or computer-readable storage medium. Certain methods and processes described herein can be embodied as software, code and/or data, which may be stored on one or more storage media. Certain embodiments of the invention contemplate the use of a machine in the form of a computer system within which a set of instructions, when executed by hardware of the computer system (e.g., a processor or processing system), can cause the system to perform any one or more of the methodologies discussed above. Certain computer program products may be one or more computer-readable storage media readable by a computer system (and executable by a processing system) and encoding a computer program of instructions for executing a computer process. It should be understood that as used herein, in no case do the terms "storage media”, ‘‘computer-readable storage media” or “computer- readable storage medium” consist of transitory carrier waves or propagating signals.

[0074] Although the subject matter has been described in language specific to structural features and/or acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as examples of implementing the claims and other equivalent features and acts are intended to be within the scope of the claims.

Claims

CLAIMS What is claimed is:

1. A method, comprising: creating a training data set comprising a plurality⁷ of modified medical images with corresponding keys, wherein creating the training data set comprises: receiving a first plurality' of medical images, wherein each of the first plurality of medical images are text-free; adding, to each of the first plurality⁷ of medical images, at least one string of text to create a modified medical image; and generating the corresponding keys, wherein a key for a particular modified medical image comprises location information of each of the at least one string of text on the particular modified medical image; and performing a training operation for a medical image de-identification model using the training data set comprising the plurality of modified medical images and corresponding keys.

2. The method of claim 1, wherein each of the first plurality⁷ of medical images are of a same image modality .

3. The method of claim 1, wherein each string of text added to each of the first plurality of medical images is randomly generated.

4. The method of claim 1, wherein the location information of the string of text comprises bounding box coordinates of a bounding box enclosing the string of text.

5. The method of claim 1, wherein performing the training operation for the medical image de-identification model comprises adjusting confidence thresholds of the medical image de-identification model.

6. The method of claim 1, further comprising: creating an updated training data set based on results of an evaluating operation for the medical image de-identification model; and updating the medical image de-identification model using an updated training data set.

7. The method of claim 6, wherein creating the updated training data set comprises: generating additional modified medical images having image characteristics satisfying a similarity metric with respect to image characteristics of test images identified as having false positives in the evaluating operation for the medical image deidentification model.

8. The method of claim 6, wherein creating the updated training data set comprises: generating additional modified medical images having textual features satisfying a similarity metric with respect to textual features of test images identified as having false negatives in the evaluating operation for the medical image de-identification model.

9. The method of claim 1, further comprising performing an image de-identification process on a medical image using the medical image de-identification model.

10. The method of claim 9, wherein performing an image de-identification process on the medical image comprises: detecting one or more objects on the medical image, wherein the one or more objects are groupings of pixels that are identified by the medical image de-identification model as having a value above a confidence threshold for representing characters; and removing at least one of the one or more detected objects from the medical image to create a de-identified medical image.

11. The method of claim 10, wherein removing at least one of the one or more detected objects from the medical image comprises: creating a bounding box around each detected object, wherein the bounding box is identified by coordinates of a rectangular region outlining the detected object; and replacing all pixels within the bounding box.

12. The method of claim 11, wherein replacing all pixels within the bounding box comprises replacing all pixels within the bounding box with pixels having a same value.

13. The method of claim 10, wherein the at least one of the one or more detected objects corresponds to protected health information.

14. The method of claim 10, further comprising: applying one or more rules for determining whether detected text of the one or more detected objects is removed or retained within the medical image; and allowing any of the one or more detected objects that satisfy certain criteria of the one or more rules to remain in the medical image.

15. The method of claim 14, wherein the one or more rules are based on nature of the detected text, location of the detected text, or a combination thereof.

16. The method of claim 10, further comprising: checking the one or more detected objects for permitted character patterns; and allowing any of the one or more detected objects that satisfy a criterion for permitted character patterns to remain in the medical image.

17. The method of claim 10, further comprising: checking the one or more detected objects for allowable terms; and allowing any of the one or more detected objects that satisfy a criterion for allowable terms to remain in the medical image.

18. The method of claim 9, further comprising: receiving a plurality of medical images for de-identification; separating the received plurality of medical images according to w hether an image is a primary capture image or a secondary capture image; discarding any secondary capture images; and performing the image de-identification process on the medical images indicated as primary capture images.

19. A computer-readable storage medium having instructions stored thereon that, when executed by a processing system, perform the method of any preceding claim.

20. A system comprising: a processing system; a storage system; and instructions stored on the storage system that, when executed by the processing system, direct the processing system to perform the method of any of claims 1-18.