KR20230112744A - Weakly supervised image classifier - Google Patents

Abstract

Systems and methods for training and using an image classifier are provided. A plurality of points may be created from each patch of images annotated as being either of the first type or the second type. Points corresponding to the second type of images may be clustered into two clusters. A first of the two clusters can be identified as being closer to points corresponding to images annotated as being of the first type. Points in the first cluster can be assigned class positive, points in the second cluster can be assigned class negative, and points corresponding to images annotated as type 1 can be assigned an anchor class. A plurality of triplets may be generated based on points from various classes. Parameters of the image classifier may be adjusted based on the loss function of the triplets.

Description

Weakly Supervised Image Classifier {WEAKLY SUPERVISED IMAGE CLASSIFIER}

CROSS REFERENCES TO RELATED APPLICATIONS

This application, 35 U.S.C. §119(e) claims priority to U.S. Provisional Patent Application No. 62/532,795 entitled "Weakly Directed Image Classifier" filed on July 14, 2017, the entire contents of which are incorporated herein by reference.

technology field

This disclosure relates generally to systems and methods for training and using image classifiers. More specifically, the present disclosure relates to systems and methods for training and using a weakly supervised image classifier by applying machine learning techniques to a plurality of annotated images.

Image classification can be an expensive and tedious process. For some image classification applications, such as applications related to the military or medical field, accurate and reliable classification of images can be very important and is typically performed by specialized trained personnel. It is difficult to accurately and reliably classify certain types of images in an automated manner.

Methods, systems and apparatus for training an image classifier are provided. Computational image classifier models may have the potential to substantially reduce the work and resources for automated image classification. In general, the term image classification may include determining whether an uncategorized input image is of a first type or a second type, where the first and second types are known. In some implementations, the systems and methods of this disclosure can receive an uncategorized input image having a complex set of features that can be represented in a multi-dimensional space. For example, an image may have multiple pixels (eg, on the order of 10 ⁶ or greater), and each pixel may have an associated value for each of multiple colors, such as red, green, and blue. The systems and methods described further below can learn lower-dimensional embeddings of input images from weak supervision. In some implementations, the images can be divided into multiple patches, and the patches can be automatically partitioned into clusters representing the first type and the second type. This learned embedding makes it possible to solve an overall classification task for an input image by using a simple classifier such as k-nearest neighbors for patches constituting the input image. In general, a weak map may refer to instances of slides in a data set where only the overall slice type (eg, either the first type or the second type) is known. There may be no information about the classification of any individual patch of the image. Thus, the techniques of this disclosure contrast with fully supervised learning, where the label of each patch of an image is known and used to train an image classification model.

In some implementations, an image classification model can be trained based on a set of images of known type (eg, classification). Images can be divided into patches, and libraries of patches can be created. For example, one library may contain patches corresponding to images of a first type, and a second library may contain patches corresponding to images of a second type. Training the image classification model can be accomplished in an iterative training process, and a subset of patches can be sampled from each library every training round. In a given training round, each patch can be transformed into a point in a low-dimensional embedding space using an image classification model, which can be, for example, a convolutional neural network.

In some implementations, points originating from one library can be clustered into two clusters using a clustering algorithm. Among the two clusters, points included in the cluster having the minimum distance to points derived from the second library may be assigned to class positive, and points included in the other cluster may be assigned to class negative. Points derived from the second library can be treated as an anchor class. Points from the Anchor, Positive, and Negative classes can be sampled to generate triplets, each triplet containing one anchor point, one positive point, and one negative point. A loss function can then be calculated, such as the triplet margin loss. In some implementations, gradient descent optimization can be used to tune the parameters of the image classification model to minimize the loss function.

Once the classifier is trained, new images can be received and divided into multiple patches. A classifier can make predictions or classifications of patches. Prediction (eg, classification) for all patches of an image may be aggregated to classify the overall image as either a first type or a second type. In addition, the systems and methods described further below may provide patch-wise segmentation of a whole image to demarcat features responsible for classifying an image as either a first type or a second type.

The techniques described in this disclosure have a variety of useful applications. In some implementations, the image classification system described herein can be used for medical diagnosis. For example, each image can be an image of a pathology slide representing a tissue sample taken from a patient, and each slide can be classified as either benign or tumorous. These slides can be used to train an image classification model that can determine whether an uncategorized input image represents tissue or tumorous tissue. A tumor typically represents only a small portion of a slide categorized as neoplastic. Because of this, it can be difficult for even a trained human to accurately classify these slides. The systems and methods of the present disclosure can help automate this classification, and can also segment the images corresponding to the tumorous slides, such that the neoplastic portion of each slide classified as neoplastic is clearly demarcated. In some other implementations, the techniques of the present disclosure can be applied to other types of images. For example, the systems and methods described herein can be satellite images, for example. Based on a set of training images, it can be used to automatically determine if a particular object or person is present in an input image.

One aspect of the present disclosure relates to a method of training an image classifier. The method may include generating, by a device comprising one or more processors, a plurality of points from respective patches created from images annotated as being either of a first type or a second type. The method may include clustering, by a device using a clustering algorithm, points corresponding to images annotated as being of the second type into two clusters. The method may include identifying, by the device, a first cluster of the two clusters that is closer to points corresponding to images annotated as being of the first type than a second cluster of the two clusters. The method may include assigning, by the device, points in a first cluster to class positive, points in a second cluster to class negative, and points corresponding to images annotated as being of the first type to an anchor class. The method may include generating, by a device, a plurality of triplets. Each triplet may contain each point from the anchor class, each point from the class positive, and each point from the class negative. The method may include calculating, by the device, a loss function of a plurality of triplets. The method may include adjusting, by the device, parameters of the image classifier based on the loss function.

In some implementations, each patch can be represented by image data in multi-dimensional space. The method may further include generating, by the device, a plurality of points by transforming image data for each of the patches into a low-dimensional space using an image classifier. In some implementations, the image classifier can be implemented as a convolutional neural network.

In some implementations, each image can correspond to a respective pathology slide, images annotated as type 1 identified as benign and images annotated as type 2 identified as neoplastic. In some other implementations, each image may correspond to a satellite photograph.

In some implementations, the method can include receiving, by a device, image data corresponding to a plurality of images. Creating a plurality of points from each patch of images may be performed for only a subset of the plurality of images. In some implementations, calculating the loss function of the plurality of triplets can include calculating, by the device, a triplet margin error for the plurality of triplets.

In some implementations, the method can further include determining, by the device, at least one of a blurriness metric or a light level metric for each of the patches. The method may also include discarding, by the device, prior to generating the plurality of points, a subset of patches in the discarded subset of patches based on at least one of an obscurity metric or a light level metric for each patch. In some implementations, the method may further include performing, by the device, data augmentation including at least one of rotation or color normalization of at least one of the patches prior to generating the plurality of points.

In some implementations, identifying a first one of the two clusters that is closer than a second cluster to points corresponding to images annotated as being of the first type comprises determining, by the device, a first average Euclidean distance between points in the first cluster and points corresponding to images annotated as being of the first type, a second average Euclidean distance between points in the second cluster and points corresponding to images annotated as being of the first type. (second average Euclidean distance) may be further included.

In some implementations, the method may further include receiving, by the device, the uncategorized input image following the adjusting parameters of the image classifier. The method may further include dividing, by the device, the input image into a plurality of uncategorized patches. The method may further include determining, by the device, a classification for each uncategorized patch based on the image classifier.

In some implementations, the method may further include aggregating, by the device, a classification for each uncategorized patch to determine an overall classification for the input image. In some implementations, the method may further include providing, by the device, an output comprising a patch-by-patch segmentation of the input image. The patch-by-patch segmentation of the input image may indicate a classification mark for the patch for each patch of the input image.

In some implementations, generating a plurality of points from each patch of images annotated as being either of the first type or the second type may further include identifying a first set of points of the plurality of points. Clustering, identifying a first cluster, assigning points, generating a plurality of triplets, calculating a loss function, and adjusting parameters of an image classifier may be performed based on the first set of points. The method may further include, in response to adjusting parameters of the image classifier based on the first set of points, repeating clustering, identifying the first cluster, assigning points, generating a plurality of triplets, calculating a loss function, and adjusting parameters of the image classifier using a second set of points of the plurality.

Another aspect of the present disclosure relates to a system for training a model for image classification. The system may include a device having one or more processors. The device may be configured to generate a plurality of points from each patch created from images annotated as being either of the first type or the second type. The device may be configured to cluster points corresponding to images annotated as being of the second type into two clusters using a clustering algorithm. The device may be configured to identify a first cluster of the two clusters that is closer to points corresponding to images annotated as being of the first type than a second cluster of the two clusters. The device may be configured to assign points of the first cluster to class positive, points of the second cluster to class negative, and points corresponding to images annotated as being of the first type to an anchor class. The device may be configured to generate a plurality of triplets, each triplet including a respective point from an anchor class, a respective point from a class positive, and a respective point from a class negative. The device may be configured to calculate a loss function of a plurality of triplets. The device may be configured to adjust parameters of the image classifier based on the loss function.

In some implementations, each patch can be represented by image data in a multi-dimensional space. The device may be further configured to generate a plurality of triplets by transforming the image data for each of the patches into a low-dimensional space using an image classifier. In some implementations, the image classifier can be implemented as a convolutional neural network.

In some embodiments, each image can correspond to a respective pathology slide. Images annotated as type 1 may be identified as benign, and images annotated as type 2 may be identified as neoplastic.

In some implementations, the device may be further configured to receive image data corresponding to the plurality of images and to generate a plurality of points from each patch of the images only for a subset of the plurality of images. In some implementations, the device may be further configured to calculate a loss function of the plurality of points by calculating a triplet margin error for the plurality of triplets.

Other aspects of the present disclosure relate to methods and systems for classifying images using the image classification system described herein. Classification of images may be image-wise classification. In some implementations, classification of images can be patch-by-patch classification or segmentation. In some implementations, an image classification system can receive a given image. Images may be unannotated. The image classification system may divide the image or otherwise derive patches from the image. The image classification system can transform at least a subset of the patches into lower embedded points. The image classification system can then classify each subset of patches using the lower embedded points and a trained image classifier. Aggregate classification of patches can be used to classify images.

The above and other objects, aspects, features and advantages of the present disclosure will become more apparent and better understood by referring to the following description taken in conjunction with the accompanying drawings.
1A is a block diagram illustrating an embodiment of a network environment including a client device communicating with a server device.
1B is a block diagram illustrating a cloud computing environment including a client device communicating with cloud service providers.
1C and 1D are block diagrams illustrating embodiments of useful computing devices in connection with the methods and systems described herein.
2A is a block diagram illustrating data flow in a system for image classification.
2B is a block diagram illustrating data flow in a system for image classification for pathology slides.
3 shows part of the architecture of an implementation of an image classification system.
4 is a flow diagram of an example method of training an image classifier.
5 is a flow diagram of an exemplary method for classifying uncategorized input images.

To read the description of the various embodiments below, the following description of the sections of this specification and their respective content may be helpful:

Section A describes network and computing environments that may be useful in practicing the embodiments described herein.

Section B describes embodiments of systems and methods for classifying images.

A. Computing and Network Environment

Before discussing specific embodiments of the present solution, it may be helpful to describe aspects of the operating environment, as well as related system components (e.g., hardware elements), in connection with the methods and systems described herein. Referring to Fig. 1A, an embodiment of a network environment is shown. In a brief overview, a network environment includes one or more clients 102a-102n (generally local machine(s) 102, client(s) 102, client node(s) 102, client machine(s) 102 in communication with one or more servers 106a-106n (also commonly referred to as server(s) 106, nodes 106 or remote machine(s) 106), Also referred to as client computer(s) 102, client device(s) 102, endpoint(s) 102 or endpoint node(s) 102). In some implementations, client 102 has the ability to function as a client node seeking access to resources provided by the server and as a server providing access to hosted resources for other clients 102a-102n.

Although FIG. 1A depicts a network 104 between clients 102 and servers 106 , clients 102 and servers 106 may be on the same network 104 . In some implementations, there are multiple networks 104 between clients 102 and servers 106 . In either of these embodiments, network 104' (not shown) may be a private network, and network 104 may be a public network. In another of these embodiments, network 104 may be a private network and network 104' may be a public network. In yet another of these embodiments, networks 104 and 104' may both be private networks.

Network 104 can be connected via wired links and wireless links. Wired links may include Digital Subscriber Line (DSL), coaxial cable lines or fiber optic lines. Wireless links may include Bluetooth, Wi-Fi, Worldwide Interoperability for Microwave Access (WiMAX), infrared channels or satellite bands. Wireless links may also include any cellular network standard used to communicate between mobile devices including standards qualified as 1G, 2G, 3G or 4G. Network standards may qualify as one or more mobile communication standard generations by meeting specifications or standards, such as those maintained by the International Telecommunication Union (ITU). For example, the 3G standard may correspond to the International Mobile Telecommunications-2000 (IMT-2000) specification, and the 4G standard may correspond to the International Mobile Telecommunications Advanced (IMT-Advanced) specification. Examples of cellular network standards include AMPS, GSM, GPRS, UMTS, LTE, LTE Advanced, Mobile WiMAX and WiMAX-Advanced. Cellular network standards may use various channel access methods, eg FDMA, TDMA, CDMA or SDMA. In some embodiments, different types of data may be transmitted over different links and standards. In other embodiments, the same type of data may be transmitted over different links and standards.

Network 104 may be any type and/or type of network. The geographic scope of network 104 can vary widely and network 104 can be a body area network (BAN), a personal area network (PAN), a local-area network (LAN), such as an intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. The topology of network 104 may be of any shape and may include, for example, any of point-to-point, bus, star, ring, mesh, or tree. Network 104 may be an overlay network that is virtual and sits on top of one or more layers of other networks 104'. Network 104 may be any such network topology known to those skilled in the art that may support the operations described herein. Network 104 may utilize a layer or stack of protocols and different techniques, including, for example, an Ethernet protocol, an Internet Protocol Suite (TCP/IP), an Asynchronous Transfer Mode (ATM) technique, a Synchronous Optical Networking (SONET) protocol, or a Synchronous Digital Hierarchy (SDH) protocol. The TCP/IP Internet Protocol Suite may include an application layer, a transport layer, an Internet layer (including, for example, IPv6), or a link layer. Network 104 may be of the type broadcast network, communication network, data communication network, or computer network.

In some embodiments, a system may include multiple logically grouped servers 106. In either of these embodiments, a logical grouping of servers may be referred to as a server farm 38 (not shown) or a machine farm 38 . In other of these embodiments, servers 106 may be geographically dispersed. In other embodiments, machine farm 38 may be managed as a single entity. In yet other embodiments, machine farm 38 includes a plurality of machine farms 38 . The servers 106 within each machine farm 38 are heterogeneous—one or more of the servers 106 or machines 106 may operate in accordance with one type of operating system platform (e.g., Windows NT manufactured by Microsoft Corporation of Redmond, Wash.), while one or more of the other servers 106 may operate in accordance with a different type of operating system platform (e.g., Unix, Linux, or Mac OS X).

In one embodiment, servers 106 of machine farm 38 may be stored in high-density rack systems along with associated storage systems and may be located in an enterprise data center. In this embodiment, integrating the servers 106 in this manner may improve system manageability, data security, physical security of the system, and system performance by locating the servers 106 and high-performance storage systems on local high-performance networks. By centralizing servers 106 and storage systems and combining them with advanced system management tools, server resources can be used more efficiently.

Servers 106 in each machine farm 38 need not be in physical proximity to other servers 106 within the same machine farm 38 . Accordingly, groups of servers 106 logically grouped into machine farms 38 may be interconnected using wide-area network (WAN) connections or metropolitan-area network (MAN) connections. For example, machine farm 38 may include servers 106 physically located in different continents or continents, countries, states, cities, campuses, or different areas of a room. When the servers 106 are connected using a local area network (LAN) connection or some form of direct connection, data transmission rates between the servers 106 within the machine farm 38 may be increased. Additionally, heterogeneous machine farm 38 may include one or more servers 106 operating according to a type of operating system, while one or more other servers 106 run one or more types of hypervisors rather than operating systems. In these embodiments, hypervisors are used to emulate virtual hardware, separate physical hardware, virtualize physical hardware, and run virtual machines that provide access to a computing environment, allowing multiple operating systems to run concurrently on a host computer. Native hypervisors can run directly on the host computer. Hypervisors include VMware ESX/ESXi manufactured by VMWare, Inc. of Palo Alto, CA; Xen hypervisor, an open source product whose development is overseen by Citrix Systems; HYPER-V hypervisor provided by Microsoft or other vendors. Hosted hypervisors may run within an operating system on a second software level. Examples of hosted hypervisors may include VMware Workstation and VIRTUALBOX.

Management of machine farm 38 may be de-centralized. For example, one or more servers 106 may include components, subsystems, and modules to support one or more management services for machine farm 38 . In either of these embodiments, one or more servers 106 provide functionality for management of dynamic data, including techniques for failover, data replication, and increasing the robustness of machine farm 38. Each server 106 may communicate with a persistent store, and in some embodiments may communicate with a dynamic store.

Server 106 may be a file server, application server, web server, proxy server, appliance, network appliance, gateway, gateway server, virtualization server, deployment server, SSL VPN server, or firewall. In one embodiment, server 106 may be referred to as a remote machine or node. In another embodiment, a plurality of nodes 290 may be in the path between any two communication servers.

Referring to FIG. 1B , a cloud computing environment is illustrated. The cloud computing environment may provide the client 102 with one or more resources provided by the network environment. A cloud computing environment may include one or more clients 102a - 102n that communicate with the cloud 108 over one or more networks 104 . Clients 102 may include, for example, thick clients, thin clients, and zero clients. Each client may provide at least some functionality even when the connection to the cloud 108 or servers 106 is interrupted. A thin client or zero client may rely on a connection to the cloud 108 or server 106 to provide functionality. The zero client may rely on the cloud 108 or other networks 104 or servers 106 to retrieve operating system data for the client device. Cloud 108 may include back end platforms, such as servers 106, storage, server farms, or data centers.

Cloud 108 may be public, private, or hybrid. Public clouds may include public servers 106 maintained by owners of clients or third parties to clients 102 . Servers 106 may be located off-site in remote geographic locations, as described above or otherwise. Public clouds may be connected to servers 106 through a public network. Private clouds may include private servers 106 that are physically maintained by the clients 102 or owners of the clients. Private clouds may be connected to servers 106 via a private network 104 . Hybrid clouds 108 may include private and public networks 104 , and servers 106 .

The cloud 108 may also include cloud based delivery, e.g., Software as a Service (SaaS) 110, Platform as a Service (PaaS) 112, and Infrastructure as a Service (IaaS) 114. IaaS may refer to a user who leases the use of infrastructure resources required for a specified period of time. IaaS providers provide storage, networking, servers or virtualization resources from a large pool, allowing users to scale quickly by accessing more resources as needed. Examples of IaaS include AMAZON WEB SERVICES provided by Amazon.com, Inc. of Seattle, WA, RACKSPACE CLOUD provided by Rackspace US, Inc. of San Antonio, TX, Google Compute Engine provided by Google, Inc. of Mountain View, CA, or RIGHTSCALE provided by RightScale, Inc. of Santa Barbara, CA. PaaS providers may provide functionality provided by IaaS - including, for example, storage, networking, servers or virtualization - as well as additional resources, such as, for example, an operating system, middleware or runtime resources. Examples of PaaS include WINDOWS AZURE provided by Microsoft Corporation of Redmond, WA, Google App Engine provided by Google Corporation, and HEROKU provided by Heroku Corporation of San Francisco, CA. SaaS providers may offer resources provided by PaaS, including storage, networking, servers, virtualization, operating system, middleware or runtime resources. In some embodiments, SaaS providers may offer additional resources, including, for example, data and application resources. Examples of SaaS include GOOGLE APPS provided by Google Inc., SALESFORCE provided by Salesforce.com Inc. of San Francisco, CA, or Office 365 provided by Microsoft Inc. Examples of SaaS may also include, for example, DROPBOX provided by Dropbox, Inc. of San Francisco, California, Microsoft SKYDRIVE, provided by Microsoft, Google Drive, provided by Google, or Apple ICLOUD, provided by Apple, Inc. of Cupertino, Calif.

Clients 102 may access IaaS resources using one or more IaaS standards, including, for example, Amazon Elastic Compute Cloud (EC2), Open Cloud Computing Interface (OCCI), Cloud Infrastructure Management Interface (CIMI), or OpenStack standards. Some IaaS standards allow clients to access resources over HTTP, and may use Representational State Transfer (REST) protocol or Simple Object Access Protocol (SOAP). Clients 102 may access PaaS resources using different PaaS interfaces. Some PaaS interfaces use HTTP packages, standard Java APIs, JavaMail APIs, Java Data Objects (JDO), Java Persistence APIs (JPA), Python APIs, web integration APIs for different programming languages, including, for example, Rack for Ruby, WSGI for Python, or PSGI for Perl, or other APIs that can be built on top of REST, HTTP, XML, or other protocols. Clients 102 can access SaaS resources through use of a web-based user interface provided by a web browser (e.g., GOOGLE CHROME, Microsoft INTERNET EXPLORER, or Mozilla Firefox provided by the Mozilla Foundation of Mountain View, Calif.). Clients 102 can also access SaaS resources through a smartphone or tablet application, including, for example, the Salesforce Sales Cloud or Google Drive app. Clients 102 may also access SaaS resources through a client operating system, including, for example, a Windows file system for DROPBOX.

In some embodiments, access to IaaS, PaaS or SaaS resources may be authenticated. For example, the server or authentication server may authenticate the user via security certificates, HTTPS or API keys. API keys may include various encryption standards, such as, for example, Advanced Encryption Standard (AES). Data resources may be transmitted over Transport Layer Security (TLS) or Secure Sockets Layer (SSL).

Client 102 and server 106 may be deployed on and/or run on any type and form of computing device, e.g., a computer, network device or appliance capable of communicating over any type and form of network and performing the operations described herein. 1C and 1D show block diagrams of a computing device 100 useful for practicing an embodiment of a client 102 or server 106 . As shown in FIGS. 1C and 1D , each computing device 100 includes a central processing unit 121 and a main memory unit 122 . As shown in FIG. 1C , computing device 100 may include storage device 128, installation device 116, network interface 118, I/O controller 123, display devices 124a-124n, keyboard 126, and pointing device 127, such as a mouse. Storage device 128 may include, without limitation, an operating system, software, and software of image classification system 120 . As shown in FIG. 1D , each computing device 100 may also include additional optional elements, such as memory port 103, bridge 170, one or more input/output devices 130a-130n (generally referred to using reference numeral 130), and cache memory 140 in communication with central processing unit 121.

Central processing unit 121 is any logic circuit that responds to and processes instructions fetched from main memory unit 122 . In many embodiments, central processing unit 121 is a microprocessor unit, such as: those manufactured by Intel Corporation of Mountain View, Calif.; those manufactured by Motorola, Schaumburg, Illinois; ARM processors and TEGRA System-on-Chip (SoC) manufactured by Nvidia of Santa Clara, Calif.; POWER7 processors manufactured by International Business Machines of White Plains, New York; or those manufactured by Advanced Micro Devices of Sunnyvale, California. Computing device 100 may be based on any of these processors or any other processor capable of operating as described herein. Central processing unit 121 may utilize instruction level parallelism, thread level parallelism, different levels of cache, and multi-core processors. A multi-core processor may include two or more processing units on a single computing component. Examples of multi-core processors include AMD PHENOM IIX2, INTEL CORE i5 and INTEL CORE i7.

Main memory unit 122 may include one or more memory chips capable of storing data and any storage location of which may be directly accessed by microprocessor 121 . Main memory unit 122 may be volatile and may be faster than storage 128 memory. The main memory unit 122 may be Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Burst SRAM or SynchBurst SRAM (BSRAM), Fast Page Mode DRAM (FPM DRAM), Enhanced DRAM (EDRAM), Extended Data Output RAM (EDO RAM), BEDO Extended Data Output DRAM (Extended Data Output DRAM), Single Data Rate Synchronous DRAM (SDR SDRAM), DDR SD It may be any variants including Double Data Rate SDRAM (RAM), Direct Rambus DRAM (DRDRAM) or Extreme Data Rate DRAM (XDR DRAM). In some embodiments, main memory 122 or storage 128 is non-volatile; For example, non-volatile read access memory (NVRAM), flash memory non-volatile static RAM (nvSRAM), ferroelectric RAM (FeRAM), magnetoresistive RAM (MRAM), phase-change memory (PRAM), conductive-bridging RAM (CBRAM), silicon-oxide-nitride-oxide-silicon (SONOS), resistive RAM (RRAM), Racetrack , NRAM (Nano-RAM) or Millipede memory. Main memory 122 may be based on any one of the foregoing memory chips or any other available memory chips capable of operating as described herein. In the embodiment shown in FIG. 1C, processor 121 communicates with main memory 122 via system bus 150 (described in more detail below). 1D depicts an embodiment of a computing device 100 in which a processor communicates directly with main memory 122 via memory port 103 . For example, in FIG. 1D, main memory 122 may be a DRDRAM.

1D shows an embodiment in which the main processor 121 communicates directly with the cache memory 140 via a secondary bus (sometimes referred to as a backside bus). In other embodiments, main processor 121 communicates with cache memory 140 using system bus 150 . Cache memory 140 typically has a faster response time than main memory 122 and is typically provided by SRAM, BSRAM or EDRAM. In the embodiment shown in FIG. 1D , processor 121 communicates with various I/O devices 130 via local system bus 150 . Various buses may be used to connect central processing unit 121 to any of I/O devices 130, including a PCI bus, a PCI-X bus, or a PCI-Express bus or NuBus. For embodiments where the I/O device is a video display 124, the processor 121 may use an Advanced Graphics Port (AGP) to communicate with the display 124 or with the I/O controller 123 for the display 124. 1D shows an embodiment of computer 100 in which main processor 121 communicates directly with I/O device 130b or other processor 121' via HYPERTRANSPORT, RAPIDIO, or INFINIBAND communication technologies. 1D also illustrates an embodiment where direct communication with local buses is intermixed: processor 121 communicates with I/O device 130a using a local interconnect bus while communicating directly with I/O device 130b.

A wide variety of I/O devices 130a - 130n may exist in computing device 100 . Input devices include keyboard, mouse, track pad, track ball, touch pad, touch mouse, multi-touch touch pad and touch mouse, microphone, multi-array microphone, drawing tablet, camera, single-lens reflex camera (SLR), digital SLR (DSLR), CMOS sensor, accelerometer, infrared optical sensor, pressure sensor, magnetometer sensors, angular rate sensors, depth sensors, and proximity sensors ), ambient light sensors, gyroscopic sensors, or other sensors. Output devices may include video displays, graphic displays, speakers, headphones, inkjet printers, laser printers and 3D printers.

Devices 130a to 130n may include a combination of multiple input or output devices, including, for example, a Microsoft KINECT, a Nintendo Wiimote for WII, a Nintendo WII U GAMEPAD, or an Apple IPHONE. Some devices 130a through 130n allow gesture recognition input through a combination of some of the inputs and outputs. Some devices 130a - 130n provide facial recognition that can be utilized as an input for different purposes including authentication and other commands. Some devices 130a - 130n provide voice recognition and input, including, for example, Microsoft KINECT, Apple's SIRI for IPHONE, Google Now or Google Voice Search.

Additional devices 130a-130n have both input and output capabilities, including, for example, haptic feedback devices, touch screen displays, or multi-touch displays. A touch screen, multi-touch display, touch pad, touch mouse or other touch sensing device may use different technologies to sense touch - e.g. capacitive, surface capacitive, projected capacitive touch (PCT), in-cell capacitive, resistive, infrared, waveguide, dispersive signal touch (DST), in-cell optical, surface elastic including surface acoustic wave (SAW), bending wave touch (BWT) or force-based sensing techniques. Some multi-touch devices may allow two or more points of contact with a surface, allowing advanced functionality including, for example, pinch, spread, rotate, scroll, or other gestures. Some touch screen devices, including for example Microsoft PIXELSENSE or Multi-Touch Collaboration Walls, may have a larger surface, such as on a table-top or on a wall, and may interact with other electronic devices. Some I/O devices 130a-130n, display devices 124a-124n, or groups of devices may be augmented reality devices. I/O devices may be controlled by I/O controller 123 as shown in FIG. 1C. The I/O controller may control one or more I/O devices, such as, for example, a keyboard 126 and a pointing device 127 (such as a mouse or optical pen, for example). I/O devices may also provide storage and/or installation media 116 for computing device 100 . In yet other embodiments, computing device 100 may provide a USB connection (not shown) to receive a hand held USB storage device. In further embodiments, I/O device 130 may be a bridge between system bus 150 and an external communication bus, such as a USB bus, SCSI bus, FireWire bus, Ethernet bus, Gigabit Ethernet bus, Fiber Channel bus, or Thunderbolt bus.

In some embodiments, display devices 124a - 124n may be coupled to I/O controller 123 . Display devices include, for example, liquid crystal displays (LCDs), thin film transistor LCDs (TFT-LCDs), blue phase LCDs, electronic paper (e-ink) displays, flexible displays, light emitting diode displays (LEDs), digital light processing (DLP) displays, liquid crystal on silicon (LCOS) displays, organic light-emitting diode (OLED) displays, active matrix organic It may include an active-matrix organic light-emitting diode (AMOLED) display, a liquid crystal laser display, a time-multiplexed optical shutter (TMOS) display, or a 3D display. Examples of 3D displays may use, for example, stereoscopy, polarization filters, active shutters, or autostereoscopic. The display devices 124a to 124n may be Head-mounted Displays (HMDs). In some embodiments, display devices 124a - 124n or corresponding I/O controller 123 may have or be controlled through hardware support for OPENGL or DIRECTX APIs or other graphics libraries.

In some embodiments, computing device 100 may include or be coupled to multiple display devices 124a - 124n , each of which may be the same or of a different type and/or form. As such, any of the I/O devices 130a-130n and/or I/O controller 123 may include any type and/or form of suitable hardware, software, or a combination of hardware and software to support, enable, or provide for the connection and use of multiple display devices 124a-124n by computing device 100. For example, computing device 100 may include any type and/or form of video adapter, video card, driver, and/or library to interface with, communicate with, connect to, or otherwise use display devices 124a-124n. In one embodiment, a video adapter may include multiple connections for interfacing to multiple display devices 124a-124n. In other embodiments, computing device 100 may include multiple video adapters, each video adapter coupled to one or more of display devices 124a-124n. In some embodiments, any portion of the operating system of computing device 100 may be configured to use multiple displays 124a-124n. In another embodiment, one or more of display devices 124a - 124n may be provided by one or more other computing devices 100a or 100b coupled to computing device 100 over network 104 . In some embodiments, software may be designed and configured to use another computer's display device as a secondary display device 124a for computing device 100 . For example, in one embodiment, an Apple iPad can be connected to computing device 100 and use the display of device 100 as an additional display screen that can be used as an extended desktop. Those skilled in the art will recognize and understand the various ways and embodiments in which computing device 100 can be configured to have multiple display devices 124a-124n.

Referring back to FIG. 1C , computing device 100 may include storage device 128 (e.g., one or more hard disk drives or redundant arrays of independent disks) for storing an operating system or other related software and for storing application software programs, such as any program related to image classification system software 120. Examples of storage device 128 include, for example, a hard disk drive (HDD); an optical drive including a CD drive, DVD drive or Blu-ray drive; solid-state drives (SSDs); USB flash drive; or any other device suitable for storing data. Some storage devices may include multiple volatile and non-volatile memories including, for example, solid state hybrid drives combining hard disks and solid state caches. Some storage devices 128 may be non-volatile, mutable, or read-only. Some storage device 128 may be internal and connected to computing device 100 via bus 150 . Some storage devices 128 may be external and may be connected to computing device 100 via I/O device 130 providing an external bus. Some storage device 128 may be coupled to computing device 100 via network interface 118 , for example via network 104 including a remote disk for MACBOOK AIR by Apple. Some client devices 100 may not require a non-volatile storage device 128 and may be thin clients or zero clients 102 . Some storage device 128 may also be used as an installation device 116 and may be suitable for installing software and programs. Additionally, the operating system and software may be run from a bootable medium (eg a bootable CD, eg KNOPPIX, a bootable CD for GNU/Linux available as a GNU/Linux distribution from knoppix.net).

The client device 100 may also install software or applications from an application distribution platform. Examples of application distribution platforms include App Store for iOS provided by Apple, Mac App Store provided by Apple, GOOGLE PLAY for Android OS provided by Google, Chrome Webstore for CHROME OS provided by Google, and KINDLE FIRE and Amazon Appstore for Android OS provided by Amazon.com. The application distribution platform may facilitate installation of software on the client device 102 . The application distribution platform may include a repository of applications on a server 106 or cloud 108 that clients 102a - 102n can access over the network 104 . The application distribution platform may include applications developed and provided by various developers. A user of the client device 102 may select, purchase, and/or download applications through the application distribution platform.

Computing device 100 also includes a standard telephone line LAN or WAN link (e.g., 802.11, T1, T3, Gigabit Ethernet, InfiniBand), a broadband connection (e.g., ISDN, Frame Relay, ATM, Gigabit Ethernet, Ethernet-over-SONET (EoS), ADSL, VDSL, BPON, GPON, optical fiber including FiOS), a wireless connection, or some combination of any or all of the foregoing. It may include, but is not limited to, a network interface 118 for interfacing to the network 104 via various connections. Connections can be established using a variety of communication protocols (e.g., TCP/IP, Ethernet, ARCNET, SONET, SDH, Fiber Distributed Data Interface (FDDI), IEEE 802.11a/b/g/n/ac CDMA, GSM, WiMax, and direct asynchronous connections). In one embodiment, computing device 100 communicates with other computing devices 100′ through any type and/or form of gateway or tunneling protocol, e.g., Secure Socket Layer (SSL) or Transport Layer Security (TLS), or Citrix Gateway Protocol manufactured by Citrix Systems, Inc. of Fort Lauderdale, Florida. Network interface 118 may include a built-in network adapter, network interface card, PCMCIA network card, EXPRESSCARD network card, card bus network adapter, wireless network adapter, USB network adapter, modem, or any other device suitable for interfacing computing device 100 to any type of network capable of communicating with and capable of performing the operations described herein.

A computing device 100 of the kind shown in FIGS. 1B and 1C may operate under the control of an operating system that controls scheduling of tasks and access to system resources. Computing device 100 may be any operating system, such as any of any version of the MICROSOFT WINDOWS operating system, different releases of Unix and Linux operating systems, any version of MAC OS for Macintosh computers, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, any operating system for mobile computing devices, or any other operating system capable of running on a computing device and performing the operations described herein. can run Typical operating systems include, among others, WINDOWS 2000, WINDOWS Server 2012, WINDOWS CE, WINDOWS Phone, WINDOWS XP, WINDOWS VISTA and WINDOWS 7, WINDOWS RT and WINDOWS 8 manufactured by Microsoft Corporation of Redmond, Washington; MAC OS and iOS manufactured by Apple Inc. of Cupertino, Calif.; and a freely available operating system, Linux, eg, the Linux Mint distribution ("distro") or Ubuntu, distributed by Canonical Limited, London, England; or Unix or other Unix-like derivative operating systems; and Android designed by Google of Mountain View, CA. Some operating systems including, for example, CHROME OS by Google can be used on zero clients or thin clients, including, for example, CHROMEBOOKS.

Computer system 100 may be any workstation, phone, desktop computer, laptop or notebook computer, netbook, ultrabook, tablet, server, handheld computer, cell phone, smart phone or other portable communication device, media playback device, game system, mobile computing device, or any other type and/or form of computing, communication, or media device capable of communicating. Computer system 100 has sufficient processor power and memory capacity to perform the operations described herein. In some embodiments, computing device 100 may have different processors, operating systems, and input devices consistent with the device. For example, Samsung GALAXY smart phones operate under the control of the Android operating system developed by Google. The GALAXY smart phone receives input through a touch interface.

In some embodiments, computing device 100 is a gaming system. For example, computer system 100 may include a PLAYSTATION 3, or Personal PLAYSTATION PORTABLE (PSP), or PLAYSTATION Vita device, manufactured by Sony Corporation of Tokyo, Japan; a NINTENDO DS, NINTENDO 3DS, NINTENDO WII, or NINTENDO WII U device manufactured by Nintendo of Kyoto, Japan; or the XBOX 360 device manufactured by Microsoft Corporation of Redmond, WA.

In some embodiments, computing device 100 is a digital audio player, such as the Apple IPOD, IPOD Touch and IPOD NANO families of devices manufactured by Apple Computer of Cupertino, California. Some digital audio players may have other functions, including, for example, any functions made available by applications from game systems or digital application distribution platforms. For example, the IPOD Touch can access the Apple App Store. In some embodiments, computing device 100 is a portable media player or digital audio player that supports file formats, including but not limited to MP3, WAV, M4A/AAC, WMA Protected AAC, AIFF, Audible audiobook, Apple Lossless audio file formats, and .mov, .m4v, and .mp4 MPEG-4 (H.264/MPEG-4 AVC) video file formats.

In some embodiments, computing device 100 may include a tablet, eg, the IPAD family of devices by Apple; devices of the GALAXY TAB family by Samsung; or KINDLE FIRE by Amazon.com, Inc. of Seattle, WA. In other embodiments, computing device 100 is an e-book reader, for example, devices in the KINDLE family of devices by Amazon.com, or devices in the NOOK family of devices by Barnes & Noble, Inc., New York City, New York.

In some embodiments, communication device 102 comprises a combination of devices, for example a smart phone combined with a digital audio player or portable media player. For example, one of these embodiments includes a smart phone, eg, smart phones of the IPHONE family manufactured by Apple Inc.; Smartphones of the Samsung GALAXY family manufactured by Samsung; Or the smartphones of the Motorola DROID family. In another embodiment, communication device 102 is a laptop or desktop computer equipped with a web browser and a microphone and speaker system, eg, a telephone headset. In these embodiments, communication device 102 is web-enabled and capable of receiving and initiating phone calls. In some embodiments, the laptop or desktop computer is also equipped with a webcam or other video capture device that enables video chat and video calls.

In some embodiments, the health of one or more machines 102, 106 in network 104 is typically monitored as part of network management. In either of these embodiments, the state of the machine may include an identification for load information (e.g., number of processes on the machine, CPU and memory utilization), port information (e.g., number of communication ports available and port address), or session status (e.g., duration and type of processes, whether a process is active or idle). In other of these embodiments, this information may be identified by a plurality of metrics, which may be applied at least in part to decisions in load balancing, network traffic management, and network failure recovery, as well as any aspect of the operations of the present solution described herein. Aspects of the foregoing operating environments and components will be apparent in relation to the systems and methods disclosed herein.

B. Systems and Methods for Classifying Images

Methods, systems and apparatus for training an image classifier are provided. Computer image classifier models have the potential to substantially reduce the work and resources required for automated image classification. In general, the term image classification may include determining whether an uncategorized input image is of a first type or a second type, where the first and second types are known. In some implementations, the systems and methods of this disclosure can receive an uncategorized input image having a complex set of features that can be represented in a multi-dimensional space. For example, an image may have multiple pixels (eg, on the order of 10 ⁶ or greater), and each pixel may have an associated value for each of multiple colors, such as red, green, and blue. The systems and methods described further below can learn a lower-dimensional embedding of input images from a weak map. In some implementations, the images can be divided into multiple patches, and the patches can be automatically partitioned into clusters representing the first type and the second type. This learned embedding makes it possible to solve an overall classification task for an input image by using a simple classifier such as k-nearest neighbors for patches constituting the input image. In general, a weak map may refer to instances of slides in a data set where only the overall slide type (eg, either the first type or the second type) is known. There may be no information about the classification of any individual patch of the image. Thus, the techniques of this disclosure contrast with fully supervised learning, where the label of each patch of an image is used to train an image classification model.

In some implementations, an image classification model can be trained based on a set of images of known type (eg, classification). Images can be divided into patches, and libraries of patches can be created. For example, one library may contain patches corresponding to images of a first type, and a second library may contain patches corresponding to images of a second type. Training the image classification model can be accomplished through an iterative training process, and a subset of patches can be sampled from each library every training round. In a given round of training, each patch can be transformed into a point in a low-dimensional embedding space using an image classification model, which can be, for example, a convolutional neural network.

In some implementations, points originating from one library can be clustered into two clusters using a clustering algorithm. Among the two clusters, the cluster having the minimum distance to points derived from the second library may be assigned to class positive, and the other cluster may be assigned to class negative. Points derived from the second library can be treated as an anchor class. Points from the Anchor, Positive and Negative classes can be sampled to create a triplet, each triplet consisting of one anchor point, one positive point and one negative point. A loss function can then be calculated, such as the triplet margin loss. In some implementations, gradient descent optimization can be used to tune parameters of an image classification model to minimize a loss function. Finally, the prediction (eg classification) for all patches on the image can be aggregated to classify the entire image as either the first type or the second type. Further, the systems and methods described further below may provide patch-by-patch segmentation of the entire image to demarcating features responsible for classifying the image as either the first type or the second type.

The techniques described in this disclosure have a variety of useful applications. In some implementations, the image classification system described herein can be used for medical diagnosis. For example, each image may be an image of a pathology slide representing a tissue sample taken from a patient, and each slide may be classified as either benign or tumorous. These slides can be used to train an image classification model that can determine whether an uncategorized input image represents benign or tumorous tissue. Because tumors typically represent only a small portion of slides classified as neoplastic, it can be difficult for even trained humans to accurately classify these slides. The systems and methods of the present disclosure can help automate this classification, and can also segment images corresponding to tumorous slides such that the tumorous portions of each slide classified as neoplastic are clearly demarcated. In some other implementations, the techniques of this disclosure can be applied to other types of images. For example, the systems and methods described herein can be used to automatically determine whether a particular object or person is present in an input image based on a set of training images, which can be, for example, satellite images.

Referring now to FIG. 2A , a block diagram illustrating data flow in a system 200 for image classification is shown. System 200 may include an image classification system 210 configured to receive various annotated unclassified images, train an image classification model, and use the model to classify and segment the unclassified images. More specifically, image classification system 210 receives unclassified images 208 as well as annotated images 205 . Annotated images 205 can include any images whose classification is already known, and the annotation of annotated images 205 can indicate their respective classification. In a general example, the annotations may indicate that each annotated image 205 is classified as either a first type or a second type, although it should be understood that the first type and the second type may in fact be any number of specific classifications. For example, in some implementations, annotated images 205 are pathology slides each annotated to indicate whether the slide is classified as benign or neoplastic. In some implementations, image classification system 210 may receive multiple annotated images 205 (eg, approximately 10 ⁴ or more). For example, the image classification system 210 may receive thousands or millions of annotated images 205 .

Image classification system 210 may use the received set of annotated images 205 to train an image classification model. For example, the image classification system 210 can determine the characteristics or specific characteristics of each annotated image 205 that give the image a particular classification, and that information can be incorporated into an image classification model. In some implementations, image classification models can be implemented using machine learning algorithms such as convolutional neural networks. After the image classification model is developed and refined, the image classification system 210 may automatically classify the image by applying the image classification model to the unclassified image 208 . In some implementations, the image classification system 210 can also segment the unclassified image 208 to identify specific portions of the unclassified image 208 that cause the unclassified image 208 to be assigned to a given classification. Stated another way, the image classification system 210 can divide the unclassified image 208 into a plurality of patches and use the image classification model to further determine the classification of each patch. Based on the classification of the patches, the image classification system 210 may determine an overall classification for the image and may segment the image in a patch-by-patch manner to generate image classification and segmentation information 220 . Additional implementation details of the image classification system 210 of FIG. 2A are described below with respect to FIGS. 2B and 3 .

Referring now to FIG. 2B , a block diagram illustrating data flow in a system 225 for image classification for pathology slides is shown. System 225 may include the same components as system 200 described herein with respect to FIG. 2A. At system 225 , image classification system 210 may receive annotated images 205 to train an image classification model for pathology. Annotated images 205 may include a benign slide set 230A and a neoplastic slide set 230B. Positive slide set 230A may include images of histology slides labeled as having benign tissue. The neoplastic slide set 230B may include images of histology slides labeled as having neoplastic tissue. Upon receiving the annotated images 205 , the image classification system 210 may select or generate a random sample set 232 of a benign slide set 230A and a neoplastic slide set 230B. Sampling of the set of benign slides 230A and tumor slides 230B may be according to simple random sampling, systematic random sampling and compressed sensing, among others. Random sample set 232 may include k samples from benign slide set 230A and k samples from neoplastic slide set 230B. Set 232 may include 2k, which is the total number of samples between the two sets 230A and 230B.

The image classification system 210 may apply the random sample set 232 to the image classification model 234 . In some implementations (such as the illustrated implementation), image classification model 234 can include a convolutional neural network (CNN) and a norm function (eg, Euclidean norm). A CNN can include one or more layers (e.g., a convolutional layer, a pooling layer, a rectified linear unit layer, and a fully connected layer, among others) and filters in each layer. The CNN's filters may include a multiplicative factor to apply to elements of the input, such as the random sample set 232. By training, the image classification system 210 can adjust or set filters in each layer of the CNN. Applying the set of random samples 232 to the CNN and then to the norm function of the image classification model 234, the image classification system 210 can generate an output in a feature map. The output may include a set of anchor points 236A and a set of unknown points 236B. Anchor value set 236A may correspond to data from either the original benign slide set 230A or the neoplastic slide set 230B. The output may include D x k data points from anchor point set 236A, and another D x k data points from unknown point set 236B. The output may include D x 2k data points between the two sets 236A and 236B. D may depend on the number of layers and filters in the CNN of the image classification model 234.

Once the output from image classification model 234 is generated, image classification system 210 may apply a clustering algorithm (eg, k-nearest neighbors clustering) to form triplets 238 . Triplets 238 may include an anchor data point set, a positive data point set, and a negative data point set. A set of anchor data points may be numbered D x k. A set of positive data points can be numbered D x m. A set of negative data points can be numbered D x n. A set of positive and negative data points can be determined from applying a clustering algorithm. Using the triplets 238, the image classification system 210 can calculate a loss function (eg, a triplet loss margin). Based on the value of the loss function, the image classification system 210 can adjust the image classification model 234 (e.g., the CNN's filters) to reduce or minimize the loss function between the positive data point set and the anchor data set, while maximizing the difference between the positive and negative data point sets. The image classification system 210 may iterate the application of the image classification model 234, the clustering algorithm, and the computation of the loss function until it is determined that the image classification model 234 has converged. Once the image classification model 234 is trained, the unlabeled pathology slides can be input into the image classification system 210 to generate image classification and segmentation information 220 . Further implementation details of the image classification system 210 of FIG. 2B are described below with respect to FIG. 3 .

FIG. 3 illustrates part of the architecture of an implementation of the image classification system 210 shown in FIG. 2A. As described above, system 210 may receive various annotated unclassified images, train an image classification model, and use the model to classify and segment the unclassified images. In some implementations, the components of system 210 of FIG. 3 may include or be implemented using the systems and devices described above with respect to FIGS. 1A-1D. For example, image classification system 210 and any of its components may be implemented using computing devices similar to those shown in FIGS.

The image classification system 210 includes, among other things, a patch generator 305, a low-dimensional embedding engine 310, a clustering engine 315, a class assignment engine 320, a triplet generator 325, a model refinement engine 330, an image classification model 335, a classification request manager 340, and a database 345 storing images 350 and patches 355. Together, the components of image classification system 210 may be configured to implement algorithms and other functionality briefly referred to above with respect to FIG. 2A . In some embodiments, the patch generator 305, the low -dimensional embedding engine 310, the clustering engine 315, the class allocation engine 320, the triple plant generator 325, the model improvement engine 330, the image classification model 335, the classification request manager 340, and the database 345 each As described further, it can be implemented as a software command set, computer code or logic that performs functions of each of these components. In some implementations, these components may instead be implemented by hardware, for example using one or more field programmable gate arrays (FPGAs) and/or one or more application specific integrated circuits (ASICs). In some implementations, these components may be implemented as a combination of hardware and software.

In some implementations, image classification system 210 may receive multiple images. Received images may include annotated images, such as annotated images 205 shown in FIG. 2A , which may include information regarding the classification of each image. In some implementations, images can be electronic images represented by pixel data. Annotations for each image may be included, for example, as metadata associated with a computer file for each image. Image classification system 210 may also receive one or more uncategorized images for which classification information is desired, such as uncategorized image 208 shown in FIG. 2A . In some implementations, image classification system 210 may store each of the received images as images 350 in database 345 .

The patch generator 305 may divide the plurality of images 350 into a plurality of patches. In general, a patch may refer to a portion of an image. For example, each image 350 may be divided into multiple patches. Each patch can be a specific size and shape (eg, square 50 x 50 pixels). In some implementations, patch generator 305 can divide image 350 into multiple patches according to a grid overlay. For example, patch generator 305 may generate 10,000 patches for image 350 by dividing image 350 having a resolution of 1000 pixels by 1000 pixels into patches each representing an adjacent square portion of the image having a resolution of 10 pixels by 10 pixels. In other implementations, patch generator 305 can create patches in a different way. For example, the patch generator 305 can create patches having rectangular or irregular shapes. In some implementations, the patches may not all be of uniform size, and in some implementations, at least some patches may overlap some of the other patches of the same image 350 . Stated another way, some of the images 350 may be included within one or more patches for that image 350 . In some implementations, patch generator 305 can also provide data augmentation and quality control functionality by further processing the generated patches. For example, patch generator 305 may discard patches that are considered low quality. In some implementations, patch generator 305 may determine a patch as low quality if it is too blurry or too dark to process meaningfully. Accordingly, patch generator 305 can determine an obscurity metric or light level metric for each patch, and discard some of the patches based on the respective obscurity metric and light level metric. In some implementations, patch generator 305 can provide data augmentation by applying one or more image processing techniques to some or all of the patches. For example, the patch generator 305 can apply color normalization to some or all of the patches or spatially rotate some or all of the patches. The patch generator 305 may store the generated patches as patches 355 in the database 345 . In some implementations, patch generator 305 may store a subset of generated patches 355 in database 345 and select them for further processing. In some implementations, patch generator 305 may randomly sample or select a subset of generator patches 355 . For example, the patch generator 305 may calculate the number of indices of the patches 355 to be selected using a pseudo-random number generator. In this way, the number of patches 355 analyzed or processed by the image classification system 210 may be reduced.

The low-dimensional embedding engine 310 may generate a plurality of points from patches of annotated images. As described above, each image may be represented by an image data set. Image data for each image can be relatively complex. For example, a single image may be represented by data corresponding to thousands or millions of pixels, and each pixel may be represented by values for several colors such as red, green, and blue. Thus, an image can be thought of as points defined in a multidimensional space, where dimensions correspond to individual pixel locations, color values of each pixel, and the like. The low-dimensional embedding engine 310 may generate points for each patch. In some implementations, the low-dimensional embedding engine 310 can accomplish this by representing each patch at the same level of granularity used to represent the entire image. Thus, a patch can be represented as a point in a multidimensional space having dimensions corresponding to the pixel location within the patch and the color value of each pixel of the patch. In some other implementations, the low-dimensional embedding engine 310 can reduce the dimensionality of each patch by, for example, applying one or more image transformation techniques to the patches.

After patches are created and represented as points, image classification system 210 may select a first subset of patches to be used to train image classification model 335 . As noted above, there may be thousands or millions of images 350, and each image may be divided into dozens, hundreds, thousands, or millions of patches. Because the iterative steps described below to train the image classification model 335 can be computationally intensive, it may be more efficient to process only a subset of the available patches at each iteration. In some implementations, the image classification system 210 may select a predetermined number of patches for each iteration, which may be represented as a fraction of the available patches based on a desired number of iterations. In some implementations, image classification system 210 may randomly select a predetermined number of patches (eg, using simple random or systematic random sampling techniques). For example, if 100 iterations of the iterative training process are performed, the image classification system 210 may select a subset of patches for each iteration equal to 1% of the total number of patches available, such that all of the patches can be used after all iterations have been performed.

As mentioned above, each annotated image can be classified as either a first type or a second type. Among the selected subset of patches, clustering engine 315 may cluster points corresponding to patches of images identified as the second type into two clusters. In some implementations, clustering engine 315 can perform this clustering operation using any type of clustering algorithm (e.g., k-means clustering algorithm, mean-shift clustering algorithm, expectation-maximization algorithm, and hierarchical clustering algorithm). After clustering engine 315 clusters points corresponding to patches of images identified as being of the second type, clustering engine 315 may identify a first of the two clusters that is closer to points in images annotated as being of first type than the second cluster. The proximity of each cluster to points in images annotated as being of type 1 can be measured in a variety of ways. In some implementations, clustering engine 315 may determine a distance (eg, straight line, Euclidean or L ^p norm) from each cluster to points in images annotated as being of the first type. For example, the clustering engine 315 can determine a first point representing the average of all points in a first cluster, a second point representing the average of all points in a second cluster, and a third point representing the average of all points in images annotated as being of the first type. The clustering engine 315 may determine the distance between the first point and the third point, and the distance between the second point and the third point, and finally, the distance between the first point and the third point is smaller than the distance between the second point and the third point.

Class assignment engine 320 may assign points to various classes based on the cluster in which each point resides. For example, the class assignment engine 320 may assign points of images annotated as type 1 to an anchor class. Class assignment engine 320 may also assign points of the first cluster to class positive due to their proximity to points of the anchor class (e.g., points in images annotated as being of type 1) or to the centroid of the anchor class. Similarly, the class assignment engine 320 may also assign points in the second cluster to class negatives because they are (on average) further from points in the anchor class or expected to be farther from the center of the anchor class than points in class positive.

The triplet generator 325 may generate a plurality of triplets of various points. In general, a triplet can be represented as an ordered set of three points. In some implementations, triplet generator 325 can generate triplets such that each triplet includes a point from an anchor class, a point from class positive, and a point from class negative. In some implementations, triplet generator 325 is configured to randomly sample from each class to generate triplets. To generate each triplet, the triplet generator 325 can randomly select a first point from the anchor class, randomly select a second point from the class positive, and randomly select a third point from the class negative. In some implementations, the random selection for the points of each triplet can be independent. As a result, the triplet generator 325 can generate two or more triplets having at least one common point. For example, a particular point from one of the classes may appear more than triplets generated. In some other implementations, triplet generator 325 can sample from the classes in a way that ensures that no points are repeated across the generated triplets. In some implementations, triplet generator 325 is configured to select points from each class using any suitable selection algorithm to generate triplets.

Model improvement engine 330 may use the triplets generated by triplet generator 325 to calculate a loss function (sometimes referred to as an objective function). In some implementations, the loss function can be a triplet margin error, mean squared error, Bayesian expected loss, and quadratic loss function, among others. In some other implementations, model improvement engine 330 may calculate different types of loss functions based on triplets. In some implementations, model improvement engine 330 can be further configured to select parameters for image classification model 335 that minimize or reduce the loss function. For example, in some implementations, model improvement engine 330 can perform gradient descent optimization on the computed loss function to determine a set of parameters that can minimize or reduce the loss function. In some other implementations, model improvement engine 330 may select parameters for minimizing or reducing the loss function using different optimization techniques. Then, the model improvement engine 330 may update the image classification model 335 using the selected parameters.

As noted above, in some implementations, the process of training the image classification model 335 can be repeated and performed on multiple subsets of patches of the annotated images. Image classification models 335 include, among others, artificial neural networks (ANNs), convolutional neural networks (CNNs) (e.g., described in detail herein as image classification model 234), support vector machines (SVMs), regression models (e.g., logistic or linear), Bayesian classifiers, Markov chains, and the like. Thus, in some implementations, image classification system 210 can determine whether all patches have been used to train the model. If unused patches remain, the image classification system 210 may select a new subset of patches, and the clustering engine 315, class assignment engine 320, triplet generator 325, and model refinement engine 330 may repeat the above steps to further refine the image classification model 335. These steps may be repeated using additional subsets of the available patches until all available patches are used to improve the image classification model 335, such that the image classification model 335 is finalized. It should be appreciated that the image classification model 335 may be updated as more and more patches are analyzed and processed. In some implementations, the image classification model 335 may be improved until the loss function generated by the triplets is below a predetermined threshold representing the level of accuracy of the image classification model.

After the image classification model 335 is complete (e.g., after convergence is reached), the image classification system 210 can be used to classify the uncategorized input image using the image classification model 335 to determine whether the uncategorized input image is of the first type or the second type. In some implementations, classification request manager 340 can receive a request to classify an input image. The request may include image data corresponding to the uncategorized input image. In some implementations, image classification model 335 can store the uncategorized input image along with other images 350 in database 345 .

The patch generator 305 may divide the input image into a plurality of patches as described above for the annotated images used to train the image classification model 335 . For example, an input image can be divided into multiple patches, and each patch can be rectangular, square or irregular in shape. Some of the patches may overlap some of the other patches in the input image. In some implementations, patch generator 305 may also discard patches deemed low quality, for example, if the patches are too blurry or too dark to be processed meaningfully. The patch generator 305 may use blur detection techniques to determine if a patch is blurry. For example, patch generator 305 can calculate the variance of a Laplacian for a patch. If the variance is greater than the threshold, the patch generator 305 may decide that the patch is too blurry and discard the patch. On the other hand, if the variance is below the threshold, patch generator 305 may determine that the patch is not too blurry and store the patch for further use. The patch generator 305 can determine if a patch is dark by calculating brightness. Patch generator 305 may calculate the lightness of the patch based on the arithmetic mean of the red, blue and green values on the patch. If the lightness is greater than the threshold, the patch generator 305 may determine that the patch is not dark and store the patch for further use. Conversely, if the brightness is below the threshold, patch generator 305 may determine that the patch is too dark and discard the patch. Patch generator 305 may also enhance the input image by applying one or more image processing techniques to some or all of the patches of the input image, for example, by normalizing the color of the patches or rotating some of the patches of the input image.

The classification request manager 340 may determine a classification for each patch of the input image. The classification for each patch can be formatted in a similar way to the overall image classification. For example, each patch may be classified as being of either a first type or a second type based on similarity of its features or characteristics compared to an annotated image for which the classification is known. In some implementations, classification request manager 340 may apply image classification model 335 to each patch of the input image to determine a classification for the patch. Thus, any information about the annotated images used to train the image classification model 335 can be used to inform the classification of patches of the input image.

Next, the classification request manager 340 may aggregate the classification of each patch of the input image to determine an overall classification for the input image. In some implementations, techniques or parameters associated with image classification model 335 may be used to aggregate the classification of each patch. For example, in some implementations, classification request manager 340 can determine that an input image should be classified as the second type only if it is determined that at least a subset of the patches of the input image satisfy the k-nearest neighbors test, where it is determined that the patch's neighbors classified as the second type will also be classified as the second type. Otherwise, the input image may be classified as the first type. Thus, in some cases, an input image may have one or more patches individually classified as a first type, while the input image as a whole is classified as a second type. Similarly, in some examples, an input image may have one or more patches individually classified as a second type, while the input image as a whole is classified as a first type.

Classification request manager 340 may also provide output that includes a classification of the input image as well as a patch-by-patch segmentation of the input image. The patch-by-patch division may include any type of information indicating the location of each patch of the second type and each patch of the first type. Thus, the viewer can determine the portions of the input image that are primarily responsible for the overall classification of the input image.

In one particular example, the annotated images may be a pathology slide classified as either benign or neoplastic, and the input image may be a non-categorized pathology slide whose classification is determined to diagnose a subject (eg, patient). Using the steps described above, image classification model 335 can be trained to classify images as either benign or neoplastic based on the information contained in the annotated input images. The classification request manager 340 may then apply the image classification model 335 to the uncategorized input image to determine whether the input image represents benign or neoplastic tissue. Classification request manager 340 may also provide a patch-by-patch segmentation of the input image showing specific regions that are neoplastic (eg, locations corresponding to the patches) based on the individually determined patches to be neoplastic.

4 is a flow diagram of an example method 400 of training an image classifier. In a brief overview, the method 400 includes receiving (405) image data for a plurality of annotated images classified as a first type or a second type, respectively, dividing (410) the plurality of images into a plurality of patches, generating (415) a plurality of points from the patches of the annotated images, selecting (420) a new subset of patches to train a model, clustering (425) the points of the annotated images of the second type, identifying (430) a first cluster that is closer to points in the images annotated as being of the first type than the second cluster, assigning (435) the points to class positive, class negative and anchor classes, generating (440) triplets each containing points from each class, computing (445) a loss function of the triplets, and adjusting (450) parameters of the image classifier based on the loss function. In step 455, if all patches have been selected to train the model, the model is complete (460). Otherwise, the method returns to step 420 and a new subset of patches is selected to train the model.

Referring again to FIG. 4 and in more detail, the method 400 includes receiving 405 image data for a plurality of annotated images each classified as a first type or a second type. In some implementations, this may be performed by an image classification system, such as image classification system 210 . In some implementations, images can be electronic images represented by pixel data. Annotations for each image may be included, for example, as metadata associated with a computer file for each image. In some implementations, image classification system 210 may store each of the received images in a database.

Method 400 also includes segmenting 410 the plurality of images into a plurality of patches. In some implementations, this may be performed by a patch generator such as patch generator 305 shown in FIG. 3 . In general, a patch may refer to a portion of an image. For example, each image can be divided into multiple patches. In some implementations, the patch generator may divide the image into multiple patches according to the grid overlay. For example, a patch generator can create 10,000 patches for an image by dividing an image with a resolution of 1000 pixels by 1000 pixels into patches each representing an adjacent square portion of the image with a resolution of 10 pixels by 10 pixels. In other implementations, patch generator 305 can create patches in a different way. For example, the patch generator can create patches with rectangular or irregular shapes. In some implementations, the patches may not all be of uniform size, and in some implementations, at least some patches may overlap some of the other patches of the same image. Stated another way, some of the images 350 may be included within one or more patches for that image.

In some implementations, the patch generator can also provide data augmentation and quality control functionality by further processing the generated patches. For example, a patch generator may discard patches deemed low quality. In some implementations, if a patch is too blurry or too dark to be meaningfully processed, the patch generator may determine that the patch is low quality. Thus, the patch generator can determine an obscurity metric or light level metric for each patch, and discard some of the patches based on the respective obscurity metric and light level metric. In some implementations, the patch generator can provide data augmentation by applying one or more image processing techniques to some or all of the patches. For example, the patch generator can apply color normalization to some or all of the patches or spatially rotate some or all of the patches.

The method 400 also includes generating 415 a plurality of points from the patches of the annotated images. In some implementations, this may be performed by a low-dimensional embedding engine, such as low-dimensional embedding engine 310 shown in FIG. 3 . As described above, each image may be represented by an image data set. For example, image data for each image may be represented by complex data corresponding to thousands or millions of pixels, and each pixel may be represented by a value for several colors such as red, green, and blue. Thus, an image can be represented by points defined in a multidimensional space, where dimensions correspond to individual pixel locations, color values for each pixel, and the like. A low-dimensional embedding engine can generate points for each patch. In some implementations, the low-dimensional embedding engine can accomplish this by representing each patch at the same level of granularity used to represent the entire image. Thus, a patch can be represented as a point in a multidimensional space having dimensions corresponding to the pixel location within the patch and the color value of each pixel of the patch. In some other implementations, the low-dimensional embedding engine can reduce the dimensionality of each patch, for example by applying one or more image transformation techniques to the patches.

The method 400 also includes selecting 420 a new subset of patches for training the model. The image classification system may select a new subset of patches to be used to train the model, which may correspond to the image classification model image classification model 335 shown in FIG. 3 . As mentioned above, there may be thousands or millions of images, and each image may be divided into tens, hundreds, thousands or millions of patches. Because the iterative steps described below to train an image classification model can be computationally intensive, it may be more efficient to process only a subset of the available patches at each iteration. In some implementations, the image classification system can select a predetermined number of patches for each iteration, which can be represented as a fraction of the available patches based on the desired number of iterations.

The method 400 also includes clustering 425 points derived from the images annotated as being of the second type. In some implementations, this may be performed by a clustering engine, such as clustering engine 315 shown in FIG. 3 . The clustering engine can perform this clustering operation using any type of clustering algorithm. After the clustering engine clusters points corresponding to patches of images identified as the second type, the clustering engine may identify a first of the two clusters that is closer to points in the images annotated as being of the first type than the second cluster (430). The proximity of each cluster to points in images annotated as being of type 1 can be measured in a variety of ways. In some implementations, the clustering engine can determine a Euclidean distance from each cluster to points in the images annotated as being of the first type. For example, the clustering engine may determine a first point representing the average of all points in a first cluster, a second point representing the average of all points in a second cluster, and a third point representing the average of all points in images annotated as being of the first type. Then, the clustering engine determines the Euclidean distance between the first point and the third point and the Euclidean distance between the second point and the third point, and determines that the distance between the first point and the third point is smaller than the distance between the second point and the third point.

The method 400 also includes assigning 435 the points to class positive, class negative, and anchor classes. In some implementations, this may be performed by a class assignment engine, such as class assignment engine 320 shown in FIG. 3 . The class assignment engine can assign points to various classes based on the cluster in which each point resides. For example, the class assignment engine may assign points of images annotated as being of type 1 to an anchor class. The class assignment engine may also assign points of the first cluster to class positive due to proximity to points of the anchor class (eg, points in images annotated as being of first type). Similarly, the class assignment engine can assign points in the second cluster to class negative because they are expected to be (on average) farther from points in the anchor class than points in class positive.

Method 400 also includes generating 440 triplets, each containing points from each class. In some implementations, this step can be performed by a triplet generator, such as triplet generator 325 shown in FIG. 3 . In general, a triplet can be represented as an ordered set of three points. In some implementations, the triplet generator can generate triplets such that each triplet contains a point from an anchor class, a point from class positive, and a point from class negative. In some implementations, the triplet generator is configured to randomly sample from each class to generate triplets. That is, to generate each triplet, the triplet generator can randomly select a first point from the anchor class, a second point from the class positive, and a third point from the class negative. In some implementations, the random selection for the points of each triplet can be independent. As a result, the triplet generator can generate two or more triplets that have at least one point in common. For example, a particular point from one of the classes may appear more than triplets generated. In some other implementations, the triplet generator can sample from the classes in a way that ensures that no points are repeated across the generated triplets.

The method 400 also includes calculating 445 the loss function of the triplets. In some implementations, this may be performed by a model refinement engine, such as model refinement engine 330 shown in FIG. 3 . In some implementations, the loss function can be a triplet margin error. In some other implementations, the model improvement engine may compute different types of loss functions based on triplets. In some implementations, the model improvement engine can be further configured to select parameters for the image classification model that minimize or reduce the loss function. For example, in some implementations, the model improvement engine can perform gradient descent optimization on the computed loss function to determine a set of parameters that can minimize or reduce the loss function. In some other implementations, the model improvement engine may select parameters for minimizing or reducing the loss function using different optimization techniques. The model improvement engine may then update 450 the image classification model by adjusting parameters of the image classification model using the selected parameters based on the loss function.

In step 455, if all patches have been selected to train the model, the model is complete (460). Otherwise, the method 400 returns to step 420 and a new subset of patches is selected to train the model. Accordingly, steps 420, 425, 430, 435, 440, 445, 450 and 455 are repeated until all patches have been used to train the image classification model and the model is complete.

5 is a flow diagram of an exemplary method 500 for classifying uncategorized input images. In brief overview, method 500 includes receiving image data corresponding to an uncategorized input image (505), dividing the input image into a plurality of uncategorized patches (510), determining a classification for each uncategorized patch (515), aggregating the classification of each patch to determine an overall classification for the input image (520), and providing an output comprising a patch-by-patch division of the input image (525).

Referring again to FIG. 5 and in more detail, the method 500 includes receiving 505 image data corresponding to the uncategorized input image. In some implementations, this may be performed by a classification request manager, such as classification request manager 340 shown in FIG. 3 . The method 500 also includes a step 510 of dividing the input image into a plurality of uncategorized patches. For example, an input image can be divided into multiple patches, and each patch can be rectangular, square or irregular in shape. Patches may be created by a patch generator such as patch generator 305 shown in FIG. 3 . Some of the patches may overlap some of the other patches of the input image. In some implementations, the patch generator may also discard patches deemed low quality, for example, if the patches are too blurry or too dark to be processed meaningfully. The patch generator may also augment the input image by applying one or more image processing techniques to some or all of the patches of the input image, such as normalizing the color of the patches or rotating some of the patches of the input image.

Method 500 includes a step 515 of determining a classification for each uncategorized patch. The classification for each patch can be formatted in a similar way to the overall image classification. For example, each patch may be classified as being of either a first type or a second type based on similarity of its features or characteristics compared to annotated images for which the classification is known. In some implementations, a classification request manager may apply an image classification model to each patch of an input image to determine a classification for the patch. Thus, any information about the annotated images used to train the image classification model can be used to inform the classification of patches of the input image.

The method 500 includes a step of aggregating 520 the classification of each patch to determine the overall classification of the input image. In some implementations, techniques or parameters associated with the image classification model may be used to aggregate the classification of each patch. For example, in some implementations, a classification request manager can determine that an input image should be classified as the second type only if it is determined that at least a subset of the patches of the input image satisfy the k-nearest neighbors test, where it is determined that the patch's neighbors classified as the second type will also be classified as the second type. Otherwise, the input image may be classified as the first type. Thus, in some cases, an input image may have one or more patches individually classified as a first type, while the input image as a whole is classified as a second type. Similarly, in some examples, an input image may have one or more patches individually classified as a second type, while the input image as a whole is classified as a first type.

Method 500 includes a step 525 of providing output comprising a patch-by-patch segmentation of an input image. In some implementations, this may be performed by a classification request manager. The patch-by-patch division may include any type of information indicating the location of each patch of the second type and each patch of the first type. Thus, the viewer can determine the portions of the input image that are primarily responsible for the overall classification of the input image.

It should be understood that the systems described above may provide any one or multiple of each of these components, and that these components may be provided on stand-alone machines or, in some embodiments, on multiple machines in a distributed system. The systems and methods described above may be implemented as a method, apparatus, or article of manufacture using programming and/or engineering techniques to create software, firmware, hardware, or any combination thereof. Additionally, the systems and methods described above may be provided as one or more computer-readable programs embodied on or in one or more articles of manufacture. As used herein, the term “article of manufacture” refers to one or more computer-readable devices, firmware, programmable logic, memory devices (e.g., EEPROM, ROM, PROM, RAM, SRAM, etc.), hardware (e.g., integrated circuit chip, field programmable gate array (FPGA), application specific integrated circuit (ASIC), etc.), electronic devices, computer-readable non-volatile storage units (e.g., CD-ROM, floppy disk, hard disk drive, etc.) embedded in and is intended to encompass any code or logic accessible therefrom. An article of manufacture may be accessed from a file server that provides access to computer-readable programs via network transmission lines, wireless transmission media, signals propagating through space, radio waves, infrared signals, and the like. The article of manufacture may be a flash memory card or magnetic tape. An article of manufacture includes hardware logic as well as software or programmable code embedded in a computer-readable medium for execution by a processor. Generally, computer-readable programs may be implemented in any programming language such as LISP, PERL, C, C++, C#, PROLOG or any byte code language such as JAVA. Software programs may be stored on one or more articles of manufacture or as object code.

Although various embodiments of methods and systems have been described, these embodiments are illustrative and do not limit the scope of the described methods or systems in any way. Persons skilled in the art may make changes to the form and detail of the described methods and systems without departing from the broadest scope of the described methods and systems. Accordingly, the scope of the methods and systems described herein should not be limited by any exemplary embodiments, but should be defined in accordance with the appended claims and their equivalents.

Claims (1)

As a method of training an image classifier,
generating, by a device comprising one or more processors, a plurality of points from each patch created from images annotated as either a first type or a second type;
clustering, by the device, points corresponding to images annotated as the second type into two clusters using a clustering algorithm;
identifying, by the device, a first cluster of the two clusters that is closer to points corresponding to images annotated as being of the first type than a second cluster of the two clusters;
assigning, by the device, points of the first cluster to class positive, points of the second cluster to class negative, and points corresponding to images annotated as the first type to an anchor class;
generating, by the device, a plurality of triplets, each triplet including a respective point from the anchor class, a respective point from the class positive, and a respective point from the class negative;
calculating, by the device, a loss function of the plurality of triplets; and
adjusting, by the device, parameters of an image classifier based on the loss function.