KR20230087331A - Method and apparatus for bottom-up segmentation of instance - Google Patents

Abstract

A bottom-up instance segmentation method and apparatus are disclosed. A bottom-up instance segmentation method according to an embodiment of the present disclosure includes the steps of acquiring an image, encoding the image into a seed map and a plurality of sigma maps based on a pre-learned bottom-up segmentation model, and sharing boundaries of each instance and between instances. The method may include identifying pixels and outputting a segmented image of an object in the image based on a boundary of each instance and a shared pixel between instances.

Description

Bottom-up instance segmentation method and apparatus {METHOD AND APPARATUS FOR BOTTOM-UP SEGMENTATION OF INSTANCE}

The present disclosure provides a bottom-up instance segmentation method for performing amodal instance segmentation of cells or nuclei with translucent overlap and occlusion by clustering grouped pixels with spatial embedding to maintain morphological features of the instance in a histopathology image. and devices.

Segmenting the nuclei facilitates further studies of the spatial distribution performed for quantitative assessment of disease or prognostic indicators for cancer grading.

In general, automated nuclear segmentation methods mainly use Haematoxylin and Eosin (H&E) stained histology slides due to their ability to reveal tissue architecture and nuclear morphology, and have been greatly innovated with recent advances in deep learning.

However, in the case of nuclear segmentation, detailed structured annotations are often required when performing deep learning due to the lack of data and especially the high cost of labeling and collection. At the same time, manual delineation may suffer from intra- and inter-observer variability. Therefore reproducible and scalable alternatives are preferred.

Conventionally, segmentation is performed in a modal manner by assigning each pixel to a unique prediction instance to detect nuclei. However, accurate delineation and separation of instances in overlapping clusters can pose challenges inherent to most conventional methods. In addition, overlapping and occluding complicate spatial dispersion due to the diverse distribution of nuclei in size and shape.

On the other hand, general amodal segmentation is performed in a top-down fashion in which segmentation is performed after finding an object to be an instance. Although these top-down methods have shown impressive performance on modal data, they are not effective due to slow inference time in segmentation of data with a large number of instances, such as a single histopathology image.

In addition, the proposed bounding box method is suitable for segmentation in general images, but may not be suitable for histopathology images composed of small-sized nuclei.

Therefore, a bottom-up method is preferred for nuclear segmentation. Following this bottom-up strategy, we redefine the spatial embedding commonly used to predict a single centroid of an object to perform amodal segmentation, and extend it to predict as many attraction centroids as the shared boundaries an instance possesses.

This bottom-up strategy allows us to model primitive data that tend to have round cell shapes and can be separated by elliptical margins. However, center target loss causes the pixel's offset and margin to reduce the distance to a single centroid, preventing instances from sharing pixels.

That is, the bottom-up method described above may also be undesirable for cells with translucent overlapping and occlusion or nuclear instance clusters with many amodal instances of nuclei.

Therefore, there is a need for a way to share pixels between segments regardless of size or shape through a subdivision mechanism to propose a generalized loss target and obtain a more complete instance.

The foregoing background art is technical information that the inventor possessed for derivation of the present invention or acquired during the derivation process of the present invention, and cannot necessarily be said to be known art disclosed to the general public prior to filing the present invention.

Prior Art 1: Davy Neven, Bert De Brabandere, Marc Proesmans, and Luc Van Gool. Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth. In CVPR, pages 88378845, 2019. 2, 3, 4 Prior Art 2: Jiaqian Yu and Matthew Blaschko. Learning submodular losses with the lovasz hinge. In International Conference on Machine Learning, pages 16231631. PMLR, 2015. 4 Prior Art 3: Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot. Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis, 58:101563, 2019. 1, 2, 3, 5, 7, 8

An object of an embodiment of the present disclosure is to cluster pixels grouped by spatial embedding to maintain the morphological characteristics of the instance in a histopathology image, thereby improving the accuracy of segmentation of submodal instances of cells or nuclei with translucent overlap and occlusion. I want to do it.

An object of an embodiment of the present disclosure is to construct a complete instance mask by clustering pixels grouped by spatial embedding by utilizing a flexible Gaussian margin capable of adapting to various sizes and patterns.

An object of an embodiment of the present disclosure is to accurately evaluate amodal prediction and avoid excessive penalty through a unique matching algorithm that updates a common metric used for nuclear segmentation.

The purpose of the embodiments of the present disclosure is not limited to the above-mentioned tasks, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the embodiments of the present invention. will be. It will also be seen that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

A bottom-up instance segmentation method according to an embodiment of the present disclosure includes obtaining a histopathology image, encoding the histopathology image into a seed map and a plurality of sigma maps based on a pre-learned bottom-up segmentation model, and encoding the boundaries of each instance. and identifying shared pixels between instances, and outputting a segmented image of cell nuclei in the histopathology image based on boundaries of each instance and shared pixels between instances.

In addition to this, another method for implementing the present disclosure, another system, and a computer readable recording medium storing a computer program for executing the method may be further provided.

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to an embodiment of the present disclosure, by clustering pixels grouped by spatial embedding to maintain morphological characteristics of instances in a histopathology image, accuracy of segmentation of amodal instances of cells or nuclei with translucent overlapping and occlusion can be improved. can

In addition, by utilizing a flexible Gaussian margin that can adapt to various sizes and patterns, it is possible to cluster pixels grouped by spatial embedding and construct a complete instance mask.

In addition, by applying a unique matching algorithm that updates the common metric used for nuclear segmentation, it is possible to accurately evaluate amodal predictions and avoid excessive penalties, improving the identification accuracy of ambiguous boundaries/closures of complex nuclear clusters.

Effects of the present disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram schematically illustrating a bottom-up instance subdivision system according to an exemplary embodiment.
2 is a diagram for explaining nuclear cluster segmentation according to an exemplary embodiment.
3 is a block diagram schematically illustrating a bottom-up instance segmentation apparatus according to an exemplary embodiment.
4 is a diagram schematically illustrating a network structure of a bottom-up segmentation model according to an embodiment.
5 is an exemplary diagram for explaining a cluster margin according to an exemplary embodiment.
6 is an exemplary diagram for explaining a metric for evaluating amodal prediction matching according to an embodiment.
7 is an exemplary diagram for explaining spatial embedding of a generated data set according to an exemplary embodiment.
8 is an exemplary diagram illustrating qualitative results of a bottom-up segmentation model according to an embodiment.
9 is a flowchart illustrating a bottom-up instance segmentation method according to an exemplary embodiment.

Advantages and features of the present disclosure, and methods of achieving them, will become clear with reference to the detailed description of embodiments in conjunction with the accompanying drawings.

However, it should be understood that the present disclosure is not limited to the embodiments presented below, but may be implemented in a variety of different forms, and includes all conversions, equivalents, and substitutes included in the spirit and technical scope of the present disclosure. . The embodiments presented below are provided to complete the present disclosure and to fully inform those skilled in the art of the scope of the disclosure. In describing the present disclosure, if it is determined that a detailed description of related known technologies may obscure the gist of the present disclosure, the detailed description will be omitted.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present disclosure. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded. Terms such as first and second may be used to describe various components, but components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

Hereinafter, embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same or corresponding components are assigned the same reference numerals, and overlapping descriptions thereof are omitted. I'm going to do it.

FIG. 1 is a diagram schematically illustrating a bottom-up instance subdivision system according to an embodiment, and FIG. 2 is a diagram for explaining nuclear cluster subdivision according to an embodiment.

Referring to FIG. 1 , a bottom-up instance segmentation system 1 may include a bottom-up instance segmentation device 100, a user terminal 200, a server 300, and a network 400.

Histopathology images have a basic three-dimensional structure and it is common to find nuclei that are in contact with each other and overlap. However, current nuclei segmentation strategies are limited to modal segmentation, whose prediction produces unnatural segmentation masks and often does not correctly delineate the boundaries of clustered nuclei. On the other hand, submodal segmentation has the potential to retain true boundary information, but is generally limited to top-down segmentation networks that are not suitable for histopathology data.

To solve this problem, the bottom-up instance segmentation system 1 of an embodiment may perform amodal segmentation using spatial embedding that can obtain complete instances together with an elliptical clustering margin around the instance center. As shown in Figure 2, the features learned to construct the amodal space embedding are key components to resolve ambiguous boundaries and occlusions in cluster nuclei.

In other words, the bottom-up instance segmentation system 1 can use a bottom-up model that uses spatial embedding to retain most of the morphological features of segmented nuclei for more accurate quantitative studies.

These bottom-up models utilize flexible Gaussian margins that can adapt to different sizes and patterns to cluster pixels grouped into spatial embeddings and construct complete instance masks.

In addition, the bottom-up model of an embodiment may update a common metric used for nuclear segmentation to accurately evaluate amodal predictions and apply a unique matching algorithm so that there is no excessive penalty.

This may allow a more realistic representation of the underlying cell distribution in histopathology samples and allow for more accurate quantitative evaluation of morphological features such as shape and size related to the grade of cancer tissue and other studies. In addition, learning of truncated and unnatural features of the nucleus can be prevented by utilizing an amodal mask.

Meanwhile, in an embodiment, users may access an application or website implemented in the user terminal 200 and perform processes such as creating and learning a network of the bottom-up instance segmentation device 100 .

The user terminal 200 includes a desktop computer, a smart phone, a laptop computer, a tablet PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop computer, a media player, a micro server, and a global positioning system (GPS) device operated by a user. , e-book readers, digital broadcast terminals, navigation devices, kiosks, MP3 players, digital cameras, home appliances, and other mobile or non-mobile computing devices, but are not limited thereto.

In addition, the user terminal 200 may be a wearable terminal such as a watch, glasses, hair band, and ring having a communication function and a data processing function. The user terminal 200 is not limited to the above, and a terminal capable of web browsing may be borrowed without limitation.

In one embodiment, the bottom-up instance segmentation system 1 may be implemented by the bottom-up instance segmentation device 100 and/or the server 300 .

In one embodiment, the bottom-up instance segmentation device 100 may be implemented in the server 300, where the server 300 is configured to operate the bottom-up instance segmentation system 1 including the bottom-up instance segmentation device 100. It may be a server or a server implementing all or part of the bottom-up instance segmentation device 100 .

In one embodiment, the server 300 clusters with elliptical margins through Amodal Spatial Embeddings (ASense) for nuclear segmentation and creates pixel groups across multiple centers so that instances can share pixels, resulting in bottom-up A bottom-up instance segmentation device (100 ) may be a server that controls the operation of

Also, the server 300 may be a database server that provides data for operating the bottom-up instance segmentation device 100 . In addition, the server 300 may include a web server, an application server, or a deep learning network providing server.

In addition, the server 300 may include a big data server and an AI server required to apply various artificial intelligence algorithms, a calculation server that performs calculations of various algorithms, and the like.

Also, in this embodiment, the server 300 may include the aforementioned servers or network with these servers. That is, in this embodiment, the server 300 may include or network with the above web server and AI server.

In the bottom-up instance segmentation system 1 , the bottom-up instance segmentation device 100 and the server 300 may be connected by a network 400 . Such a network 400 may be wired networks such as LANs (local area networks), WANs (wide area networks), MANs (metropolitan area networks), ISDNs (integrated service digital networks), wireless LANs, CDMA, Bluetooth, satellite communication However, the scope of the present disclosure is not limited thereto. In addition, the network 400 may transmit and receive information using short-range communication and/or long-distance communication.

Also, the network 400 may include connections of network elements such as hubs, bridges, routers, switches, and gateways. Network 400 may include one or more connected networks, such as a multiple network environment, including a public network such as the Internet and a private network such as a secure enterprise private network. Access to network 400 may be provided through one or more wired or wireless access networks. Furthermore, the network 400 may support an Internet of Things (IoT) network and/or 5G communication in which information is exchanged and processed between distributed components such as things.

3 is a block diagram schematically illustrating a bottom-up instance segmentation apparatus according to an exemplary embodiment.

Referring to FIG. 3 , the bottom-up instance segmentation apparatus 100 may include a communication unit 110, a user interface 120, a memory 130, and a processor 140.

The communication unit 110 may provide a communication interface required to provide a transmission/reception signal between external devices in the form of packet data in conjunction with the network 400 . In addition, the communication unit 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals to and from other network devices through wired or wireless connections.

That is, the processor 140 may receive various data or information from an external device connected through the communication unit 110 and may transmit various data or information to the external device.

In one embodiment, the user interface 120 includes an input interface into which user requests and commands for controlling the operation of the bottom-up instance segmentation device 100 (eg, network parameter change, network learning condition change, etc.) are input. can do.

In one embodiment, the user interface 120 may include an output interface outputting a result of instance segmentation. That is, the user interface 120 may output results according to user requests and commands. An input interface and an output interface of the user interface 120 may be implemented in the same interface.

The memory 130 may store various types of information required for operation control (operation) of the bottom-up instance segmentation apparatus 100 and may store control software, and may include a volatile or non-volatile recording medium.

The memory 130 is connected to one or more processors 140 by an electrical or internal communication interface and, when executed by the processor 140, causes the processor 140 to control the image restoration device 100 (cause) Codes can be saved.

Here, the memory 130 may include non-temporary storage media such as magnetic storage media or flash storage media, or temporary storage media such as RAM, but the scope of the present invention is not limited thereto. The memory 130 may include built-in memory and/or external memory, and may include volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, Non-volatile memory such as NAND flash memory, or NOR flash memory, SSD. It may include a compact flash (CF) card, a flash drive such as an SD card, a Micro-SD card, a Mini-SD card, an Xd card, or a memory stick, or a storage device such as an HDD.

Also, information related to an algorithm for performing learning according to the present disclosure may be stored in the memory 130 . In addition, various information necessary within the scope of achieving the object of the present disclosure may be stored in the memory 130, and the information stored in the memory 130 may be updated as received from a server or an external device or input by a user. may be

The processor 140 may control overall operations of the bottom-up instance segmentation device 100 . Specifically, the processor 140 is connected to the configuration of the bottom-up instance segmentation apparatus 100 including the memory 130, and executes at least one command stored in the memory 130 to operate the bottom-up instance segmentation apparatus 100. can be controlled overall.

Processor 140 can be implemented in a variety of ways. For example, the processor 140 may include an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor Processor, DSP) may be implemented as at least one.

The processor 140, as a kind of central processing unit, may control the operation of the image restoration device 100 by driving control software loaded in the memory 130. The processor 140 may include any type of device capable of processing data. Here, a 'processor' may refer to a data processing device embedded in hardware having a physically structured circuit to perform functions expressed by codes or instructions included in a program, for example.

Hereinafter, a bottom-up instance segmentation process in the processor 140 will be described with reference to FIGS. 4 and 5 and Equations 1 to 7.

4 is a diagram schematically illustrating a network structure of a bottom-up segmentation model according to an embodiment.

In one embodiment, processor 140 may perform submodal nuclear segmentation that may retain the natural shape of the instances in the clustered nucleus region.

As shown in Figure 4, the bottom-up segmentation model of one embodiment is configured to encode an image into a series of maps via an encoder (eg, CNN) to determine the boundaries of instances and also identify shared pixels for submodal segmentation. It can be.

For a bottom-up approach to performing amodal segmentation, such as the bottom-up segmentation model, we need to account for shared pixels in terms of the center of attraction and the cluster margin.

A simple way to construct such a spatial embedding is to divide each amodal ground truth (GT) mask into subcomponents and directly regress the unique attraction centroids for each segment individually. However, it separates the spatial embedding task from the cluster margin estimation and requires additional post-processing. This post-processing may not be optimal for different instance sizes.

Accordingly, the processor 140 may learn an optimal feature for clustering large and small instances by integrating spatial embedding and margin estimation into a single task.

Thus, processor 140 can construct spatial embeddings and margins that always take full instance masks into account, learn features that describe the basic properties of nuclei, and avoid the use of truncated samples. This is a key property to accurately identify and segment nuclei in dense regions.

That is, in one embodiment, you can treat instance segmentation as a pixel assignment problem, and try to associate a pixel with the correct object. To this end, the processor 140 may learn an offset vector of each pixel indicating the center of the object. Here, the object may be a cell nucleus.

Therefore, in one embodiment, unlike the standard regression approach, loss for pixels far from the center can be mitigated by learning an optimal clustering area for each object. In an embodiment, a seed map for each semantic class may be learned to find the center of an object. A detailed description of this will be described later.

에 대한 최적 오프셋

와 주변 마진

을 간접적으로 학습하여, 픽셀

를 인스턴스

로 클러스터링 하는 프레임워크를 수행할 수 있다. 여기서,

는 각 픽셀

에 대한 오프셋 벡터이다. 여기서

는 (

,

)와 같이 x 좌표 및 y 좌표를 가지는 2차원 좌표일 수 있다.The processor 140 is the center of the object

Optimal offset for

and margin around

By learning indirectly, the pixel

instance

You can perform a framework that clusters with . here,

is each pixel

is the offset vector for here

Is (

,

), it may be a two-dimensional coordinate having an x coordinate and a y coordinate.

The processor 140 according to an embodiment may cluster large and small adjacent objects without adversely affecting each other through the bottom-up instance segmentation approach.

를 사용하여 예측 픽셀의 오프셋과 인스턴스 중심 사이의 거리를 해당 인스턴스에 속하는 확률로 변환할 수 있다. 가우스 함수

는 다음 수학식 1과 같이 나타낼 수 있다.In particular, the processor 140 is a Gaussian function

can be used to convert the distance between the offset of the predicted pixel and the center of the instance into a probability of belonging to that instance. Gaussian function

can be expressed as in Equation 1 below.

In this case, a high probability value may mean closer to the center of the instance, and a low probability value may be applied to the background pixel.

The margin of the allowed distance of a pixel from the instance center can be controlled by the sigma value of the Gaussian function. Since this margin is unique to each instance, larger objects can have a larger distance between the center of the instance and the pixel.

> 0.5이면 위치

에서 해당 픽셀보다 인스턴스 k에 할당될 수 있다.For example, the Gaussian function

Position if > 0.5

may be assigned to instance k rather than the corresponding pixel in .

Accordingly, the margin as shown in Equation 2 can be controlled by modifying the sigma parameter of the Gaussian function.

를 출력해야 한다. 다음 수학식 3과 같이,

를 인스턴스 k에 속하는 모든

의 평균으로 정의할 수 있다.The larger the sigma, the larger the margin, and the smaller the sigma, the smaller the margin. To do this, a bottom-up segmentation model is used at each pixel location.

should output As shown in Equation 3 below,

all belonging to instance k

can be defined as the average of

For each instance k, the Gaussian function outputs a foreground/background probability map, so it can be optimized using binary classification loss, using the binary foreground/background map of each instance as ground truth.

In one embodiment, instead of using the standard cross entropy loss function, a hinge loss may be used instead.

Hinge loss is a loss function used to train classifiers, especially SVMs, where the x-axis represents the distance from the boundary of a single instance and the y-axis represents the loss magnitude or penalty the function suffers over distance. Correctly classified points have small or no loss magnitude, whereas incorrectly classified instances have high loss magnitude. A negative distance from the boundary results in high hinge loss, which essentially means that the instance is on the wrong side of the boundary and the instance is misclassified. A positive distance results in little or no hinge loss, and the further away from the boundary in the positive direction, the lower the hinge loss.

Since this hinge loss is a (piecewise linear) convex surrogate for the Jaccard loss, we can directly optimize the intersection of each instance. Therefore, there is no need to account for the class imbalance between foreground and background. Here, the Jaccard loss is converted into a loss function to directly improve the performance of the Jaccard index used as a metric.

The bottom-up segmentation model of one embodiment has no direct oversight of the sigma and offset vector outputs like the standard regression loss. Instead, it can be jointly optimized to maximize the intersection of each instance mask by receiving the gradient with backpropagation through a hinge loss and a Gaussian function.

In the bottom-up segmentation model of an embodiment, sigma is not fixed and is a learnable parameter. Therefore, the bottom-up segmentation model can more efficiently minimize the loss by modifying the sigma.

That is, a bottom-up segmentation model, in addition to pulling instance pixels within a (typically small) region around the center of an instance and pushing background pixels outside this region, can modify the sigma to make the size of the region better suited to that particular instance.

Intuitively, this means that for large objects, apply sigma to make the area around the center larger so that more instance pixels can point inside the area, and for small objects to select a smaller area and apply it to the background more easily so that the pixels are outside the area. It can mean that it can point to .

5 is an exemplary diagram for explaining a cluster margin according to an exemplary embodiment.

Equation 1 above with a scalar value for sigma can produce a circular margin. In addition, Equation 1 can be extended to use 2-dimensional sigma, and as shown in Equation 4 below, an elliptical margin more suitable for elongated objects in natural images can be learned.

In one embodiment, an elliptical margin may be desirable as it has a shape very similar to the regular shape of a cell or nucleus. Also, due to the shapes and patterns commonly seen in histopathology images, circular margins may not be suitable for clustering amodal masks in dense areas.

Referring to the 1 sigma map in Fig. 5, small margins can discard shared pixels, while larger margins can cluster pixels across instance boundaries due to the tilt and elongation of instances.

Equation 4 above can explain an object such as a pedestrian or a train, but cannot explain various shapes of the nucleus. As a result, in one embodiment, a 2D Gaussian function can be extended to use three “sigma maps” to describe the rotation.

That is, in an embodiment, a bottom-up segmentation model may be configured to capture all possible shape deformations. This can express a Gaussian function as shown in Equation 5 below.

,

및

(도 4의 시그마 맵)는 시그마 값

와

, 그리고 회전 각도

로 정의될 수 있다. 즉,

,

및

와 같이 나타낼 수 있으며, 행렬은 수학식 6과 같으며, 이 행렬은 양의 정 부호(positive-definite) 행렬이다.

및

는 객체 또는 대응 인스턴스 중심의 좌표 중 x 좌표 및 y 좌표를 의미할 수 있다.where the coefficient

,

and

(Sigma map in Fig. 4) is the sigma value

and

, and the angle of rotation

can be defined as in other words,

,

and

It can be expressed as, and the matrix is as in Equation 6, and this matrix is a positive-definite matrix.

and

may mean the x coordinate and the y coordinate among the coordinates of the center of the object or corresponding instance.

In one embodiment, a network of bottom-up segmentation models may require several training steps as an initialization without optimizing the rotation factor to ensure convergence.

The main purpose of the optimization function is to reduce the distance from the spatial embedding to the instance centroid as close to zero as possible. It can also apply margins to reduce size.

)으로 힌지 손실을 완화할 수 있다. However, in order to reconstruct the amodal mask, the bottom-up segmentation model must have the flexibility for pixels in shared regions to have smaller offset values, and margins large enough to cover pixels near the center and in overlapping regions. This allows a probability threshold (

) to mitigate the hinge loss.

가 상기 확률 임계값보다 크면, 가우스 함수

를 1로 간주할 수 있다. 따라서 공유 영역의 픽셀은 터치(touching) 인스턴스 간에 강제로 분할되지 않고, 대신 전체 그룹을 한번에 획득하여 클러스터링 프로세스를 단순화하는 가까운 다른 중심으로 그룹화하는 경향이 있다. 예를 들어, 확률 임계값(

)은 0.8로 설정될 수 있다. Specifically, the Gaussian function of Equation 5

is greater than the probability threshold, the Gaussian function

can be considered as 1. Therefore, the pixels in the shared area are not forced to be split between touching instances, but instead tend to group into different nearby centers which simplifies the clustering process by obtaining the entire group at once. For example, the probability threshold (

) may be set to 0.8.

In inference, each object can be clustered using spatial embedding and estimated margins. At this stage, the processor 140 may sample each object using the seed map. Referring to FIG. 5 , the effect of clustering on the size and shape of the margin and the segmentation mask can be confirmed.

Processor 140 must cluster around the center of each object when inferring. Since the above loss function ensures that the spatial embedding pixels lie close to the center of the object, we can sample a good spatial embedding pixel and use that location as the instance center.

So, for each spatial embedding pixel, we can learn how far it was placed from the center of the instance. Spatial embedding pixels placed very close to the instance center score high in the seed map, and spatial embedding pixels far from the instance center score low in the seed map. This allows inference to select spatial embedding pixels with high seed scores. This indicates that the embedding is very close to the center of the object.

In practice, the seediness score of the spatial embedding pixel should equal the output of the Gaussian function since it converts the distance between the embedding and the instance centroid into a proximity score. That is, the closer the embedding is to the center, the closer the output is to 1.

So we can train the seed map using the regression loss function. Background pixels can regress to 0 and foreground pixels can regress to a Gaussian output.

In an embodiment, a seed map for each semantic class may be trained using a loss function as shown in Equation 7 below.

는 상향식 세분화 모델의 픽셀 i의 시드 출력이다. 이때 가우스 함수

를 스칼라로 간주할 수 있으며, 기울기는

에 대해서만 계산될 수 있다.

is the seed output of pixel i of the bottom-up segmentation model. In this case, the Gaussian function

can be considered as a scalar, and the slope is

can be calculated only for

Hereinafter, with reference to FIGS. 6 to 8 , experimental results on implementation performance of the bottom-up segmentation model will be described.

6 is an exemplary view for explaining a metric for evaluating amodal prediction matching according to an embodiment, FIG. 7 is an example view for explaining spatial embedding of a generated data set according to an embodiment, and FIG. It is an exemplary view showing qualitative results of the bottom-up segmentation model according to the embodiment.

Actual histopathology data may be subject to low acquisition quality, noise, cells of different life stages, and other variations that may lead to differences in expert opinion.

Therefore, in one embodiment, as shown in FIG. 7, a Toy data set (hereinafter referred to as a synthetic data set) including an image with an ellipse representing a cellular instance independent of color and texture is generated by You can do an experiment.

In one embodiment, a synthetic data set can be used to evaluate how bottom-up segmentation models perform in different patterns, sizes, and shared domains with well-defined boundaries.

The synthetic data set contains multiple instances with random colors for both background and foreground, and the shared regions can be blended to simulate the presence of translucent objects. Since the synthetic data set does not contain solid occlusion, hallucination images are not required for bottom-up segmentation models.

In addition, in one embodiment, an experiment may be performed using a publicly available benchmark data set (MoNuSeg Dataset) for nuclear segmentation research. The raw annotations of the benchmark data set (hereinafter referred to as the public data set) are amodal, and the training data may include 30 tissue images with single-class nuclear boundary annotations.

Images can be extracted from H&E whole slide images (WSI) captured at 40x, i.e. a single image per slide. Each WSI is a sample taken from seven organs of a single patient: breast, liver, kidney, prostate, bladder, colon, and stomach, and may contain both benign and diseased tissue.

Test data from the public data set may include 14 images from 7 organs: kidney, lung, colon, breast, bladder, prostate and brain. Lung and brain tissues are only available in the test set.

On the other hand, nuclear instance segmentation performance can use various metrics to target reward for good mask prediction and penalty for false positives and negatives.

The algorithm for optimizing the performance of this nuclear instance segmentation has an initial pair matching step, after which the matching and non-matching scores can be calculated for both GT and prediction.

Metrics for calculating scores in this way may include an Ensemble Dice coefficient (DICE_2), an Adjusted Jaccard Index (AJI) method, a Panoptic Quality (PQ) method, and the like.

All of these metrics define a set of pairs and evaluate the degree of agreement between predictions and labels so that pixels outside of an instance, missing predictions, and segments without matching pairs can each be penalized.

More specifically, the ensemble dice counting method calculates the average Dice coefficient by matching all possible prediction segments that overlap with a GT segment and accumulating both the overlapping area and the sum of the areas. In the ensemble dice counting method, as shown in prediction A of FIG. 6, segments that do not match may be ignored and an excessive penalty may be imposed.

The adjusted Jacquard exponential method, a metric similar to the Ensemble dice counting method, uses Intersections Over Union (IoU) instead and accumulates the Intersections Union before calculating the average IoU. The adjusted Jacquard exponential method can potentially use predicted segments of more than one GT segment, and may result in excessive penalties for partially overlapping regions where there is no better match, such as prediction A in Fig. 6.

The panoptic quality method, introduced for evaluation of nuclear instance segmentation to enable more accurate quantification and interpretation, can be decomposed into detection quality (DQ) and segmentation quality (SQ) to provide insight for analysis. . It has been mathematically proven that the panoptic quality method solves the matching problem by accepting only pairs with IoU > 0.5 and produces unique matches.

However, in order to correctly evaluate the quality of amodal prediction, the above metric alone has limited metrics and needs to be extended to account for shared pixels in overlapping regions.

For the amodal mask, the condition IoU > 0.5 indicates no uniqueness in the presence of occlusion. Thus, in one embodiment, we can extend the previous metrics for amodal segmentation based on unique matches (aDICE_2, aAJI and aPQ) inspired by PQ.

That is, in one embodiment, it is possible to first identify the prediction reporting the highest IoU for each GT segment and at the same time identify the label with the highest IoU for each prediction segment.

Thus, the extended metric of one embodiment can be considered a unique match for labels and predictions that have the same pair in all IoU conditions (see predictions A and B in FIG. 6).

That is, the extended metric of one embodiment can solve the excessive penalty problem present in previous implementations and yield more consistent measures when evaluating similar predictions such as A, B, C and D in FIG. 6 . Predictions C and D can indicate consistency between modal and submodal cases.

The extended metric of one embodiment is applicable to both modal and amodal tasks (see predictions C and D in FIG. 6 ), but can remove excessive penalties from DICE_2 and AJI (see prediction A in FIG. 6 ).

In one embodiment, it is still possible to apply a penalty to segments, GTs and predictions that do not match in the denominator of the aforementioned metric.

Thus, the overridden aDICE_2 and aAJI scores cannot be directly compared to previous implementations, but using unique matches may add to the already existing benefits of PQ.

On the other hand, common data augmentation techniques applied to histopathology images include random image flipping, rotation, Gaussian blur or sharpening, contrast and brightness. change, an addition of Gaussian noise may be included.

However, these conversions alone may not be sufficient to account for differences between laboratories. Unlike commonly used color normalization methods, staining variation may be a key component in increasing a CNN's ability to generalize to unseen data.

In one embodiment, random channel-wise mixing of the original images may be performed during training, which may be a normalized, grayscale version. This allows the creation of larger sets of images with different staining combinations.

Also, in one embodiment, the number of quantization levels may be modified during training to make the bottom-up segmentation model robust to various levels of image quality. Images often have 256 quantization levels due to the 8-bit encoding, so by randomly changing the available quantization levels from 10 to 256, the bottom-up segmentation model can learn to identify nuclei and their boundaries, even in extremely low-quality images. can make it possible

In one embodiment, random cropping of 256 X 256 may be performed for training, cropped images may be enlarged, and labels of imperfect instances at image edges may be ignored. These instances can be ignored as they will be completed in other crops.

This allows the bottom-up segmentation model to focus only on full-size instances and potentially avoids creating artefacts in the learning process.

During inference, we can use a sliding window of the same size with a stride of 128 to make valid predictions only for complete instances. The merging process for multiple crops can be performed using only the predicted instances of the 128 X 128 area centered on the center point. Overlapping regions are detected during the inferencing and merging stages so that all instances can remain intact throughout the entire process.

까지 감소하는

의 초기 학습률로 적용될 수 있다.In one embodiment, for example, a lightweight ERFNet can be used as a CNN backbone network. All experiments can be performed using the PyTorch deep learning library with GPU acceleration on a machine with an RTX 2080 Ti. The Adam optimizer has an exponential decay over 100 epochs for synthetic data and 2000 epochs for pathology data.

decreasing to

can be applied as an initial learning rate of

In one embodiment, the artificial data allowed us to evaluate whether the bottom-up segmentation model could handle common instance patterns seen in histopathology images, but with precise labels.

Referring to Figure 7, we can see how clustering with periodic margins cannot perform amodal partitioning, and as a result instances can be truncated at shared boundaries. Here, the cluster can reduce the size of the margin. This is due to the lossy goal of bringing the spatial embedding to a single centroid per instance.

That is, referring to FIG. 7 , spatial embedding has a great influence on the segmentation result at the intersection. Models with extended margins create offsets that allow clusters to share pixels in areas of overlap (yellow arrows). However, only elliptical margins (3 sigma maps) with freedom of rotation can be exploited for successful amodal segmentation. It can be seen that Mask R-CNN generally performed well on instance segmentation in the images of the synthetic data set, but failed to detect small instances in congested areas (blue arrows).

Results with periodic margins show a single centroid per instance in the spatial embedding and sharp boundaries can be identified in areas overlapping with other instances based on color differences. Therefore, using a periodic margin is not suitable for amodal segmentation and may lead to an unnatural segmentation mask.

On the other hand, the extended margin modifies the spatial embedding at the intersection, allowing pixel sharing and allowing clusters to capture multiple centroids to construct complete instance masks. The arrows also show that all types of margin shapes produce similar embeddings, but the expanded elliptical margins with freedom of rotation are more robust (see yellow arrows in Fig. 7).

는 시그마 맵, (ext-m)은 확장 마진을 나타낸다.Table 1 shows the bottom-up segmentation model training results using the synthetic data set. The segmentation quality metric shows the importance of a 3-sigma map with extended margins to correctly segment intersections. Mask R-CNN has a lower score due to bounding box misalignment and lower resolution and mask. here,

is the sigma map, and (ext-m) represents the expansion margin.

Referring to Table 1, it can be seen that the proposed three sigma maps and the proposed ellipse margin (Ours) with extended margins are significantly improved over the previous method. Mask RCNN can be used for amodal segmentation in natural images and shows reasonable performance, but it can be seen that the model has limitations in segmenting small instances in congested regions (see blue arrow in Fig. 7). For this reason, we obtained lower results compared to the bottom-up technique using only one sigma map.

Using an amodal mask during training allows the model to better learn how to segment nuclei in clustered regions (using public data sets). Referring to FIG. 8 , it can be seen that the bottom-up segmentation model of an embodiment can successfully segment various tissue types having nuclei of various sizes and shapes.

For example, a bottom-up segmentation model (Ours) outputs successful results even for the tips of very small cluster nuclei, whereas other methods fail to isolate or miss instances entirely. This is because top-down methods such as Mask R-CNN do not detect small instances or perform accurate delineation through bounding boxes. The HoVer-Net can reasonably detect small nuclei, but the results do not accurately reflect the real nucleus properties. That is, the shape of some nuclei is not elliptical or irregular.

Table 2 shows the bottom-up segmentation model training results using a public data set (eg, MoNuSeg). That is, Table 2 shows the competition scores for the conventional method. A bottom-up segmentation model of an embodiment uses Amodal Spatial Embedding, and will be described below as a bottom-up segmentation model (ASense).

The reference results at the top of Table 2 (marked with diamonds) are not directly comparable as they do not use updated metrics and the actual foreground mask may differ by a few pixels at the border. That is, we extracted the amodal mask directly from the raw annotation using our own algorithm.

) 및 (

)는 각각 다른 데이터가 사용되었음을 나타낸다. For this reason, in one embodiment, we re-evaluated the official HoVer-Net pretrained model using metrics and labels on the same data. (*), ( shown in the table

) and (

) indicates that different data were used.

) 데이터 세트에서 상향식 세분화 모델(ASense)은 더 정확하게 분할할 수 있었던 추가 장기 조직의 존재로 인해 우수한 성능을 보여준다. 마지막으로 (

) 데이터 세트에서 상향식 세분화 모델(ASense)은 다른 비교 방법들 대비 더 많은 개선 사항을 보여준다.In most cases, Mask R-CNN scores were lower than bottom-up segmentation models (ASense). (*) On our dataset, HoVer-Net achieves better aDICE_2 and aAJI, but reports lower aSQ due to the difference in performance for larger and smaller instances. That is, the results were better in small nuclei. however (

) in the data set, a bottom-up segmentation model (ASense) shows superior performance due to the presence of additional organ tissues that could have been segmented more precisely. finally (

) in the data set, the bottom-up segmentation model (ASense) shows more improvement over other comparison methods.

Table 3 shows the inference time. All 30 H&E images in the 1000 X 1000 pixel public dataset can be inferred on a workstation with a single GPU.

Referring to Table 3, the bottom-up segmentation model (ASense) of an embodiment performs inference much faster than HoVer-Net and is similar to Mask R-CNN, which is a mature and optimized method.

Further optimization of the clustering and amodal aggregation mask can potentially speed up the process as the CNN forward pass of bottom-up segmentation model (ASense) requires significantly shorter running time.

That is, in one embodiment, by using a bottom-up strategy of clustering objects with a Gaussian function that directly optimizes spatial embedding for amodal segmentation, the quality of nuclear instance segmentation in a histopathology image can be improved.

As can be seen from the above experimental results, the bottom-up instance segmentation apparatus 100 according to an embodiment can capture natural positions of nuclei and amodal masks for high-quality rendering in the case of difficult regions with overlapping portions.

In addition, the bottom-up instance segmentation apparatus 100 extends the existing nuclear segmentation metric to process amodal labels and predictions, as well as to make them more stable, thereby eliminating the conventional excessive penalty problem.

In addition, in one embodiment, competitive performance and applicability can be verified by performing extensive evaluation of the bottom-up instance segmentation apparatus 100 on both the synthetic data set and the public data set.

9 is a flowchart illustrating a bottom-up instance segmentation method according to an exemplary embodiment.

Referring to FIG. 9 , in step S100, the processor 140 acquires a histopathology image.

In step S200, the processor 140 encodes the histopathology image into a seed map and a plurality of sigma maps based on the pre-learned bottom-up segmentation model to identify boundaries of each instance and shared pixels between instances.

In this case, the pre-learned bottom-up segmentation model is a learning model learned to cluster each pixel into an instance based on an optimal offset with respect to the center of the object. The optimal offset is the sum of each pixel's coordinates and the offset vector for that pixel.

This pre-learned bottom-up segmentation model unsupervisedly learns the spatial embedding and margin of the histopathology image based on the Gaussian function (see Equation 5), and performs seedmap-based sampling to cluster around the center of each instance. , can be configured to infer the boundaries and shared pixels of each pixel of the histopathology image.

The processor 140 may perform spatial embedding for each pixel on the histopathology image in order to identify boundaries of each instance and shared pixels between instances, and group each pixel and shared pixels between instances based on the spatial embedding result. .

Further, the processor 140 may estimate a margin, which is a distance allowed to an instance corresponding to a spatial embedding pixel from the center of each instance, based on the plurality of sigma maps.

Here, the plurality of sigma maps include three sigma maps for estimating margins in the x-axis direction, the y-axis direction, and the rotation direction of each instance.

의 각 계수

,

및

로 도출될 수 있다(수학식 5 참조).And the plurality of sigma maps, based on the distance between the spatial embedding pixel and the center of the corresponding instance, a Gaussian function that converts the spatial embedding pixel into a probability that the corresponding instance belongs

Each coefficient of

,

and

It can be derived as (see Equation 5).

,

및

는, x 시그마 값(

), y 시그마 값(

) 및 회전 각도에 기반하여 도출될 수 있다. At this time, each coefficient

,

and

is the x sigma value (

), y sigma value (

) and the rotation angle.

,

및

의 x 시그마 값(

), y 시그마 값(

) 및 회전 각도를 포함할 수 있다.Also, the processor 140 may estimate the margin of each instance based on the sigma parameter of the Gaussian function. At this time, the sigma parameter is each coefficient

,

and

x sigma value of (

), y sigma value (

) and rotation angle.

That is, the processor 140 may estimate the size and shape of the margin based on the sigma parameter according to the size and shape of each instance. That is, in one embodiment, all possible shape deformations can be captured.

In this case, the processor 140 may learn the sigma parameter so that the value of the Gaussian function becomes 1 when the value of the Gaussian function is greater than or equal to a preset probability threshold. This is because pixels in the shared area must have the flexibility to have smaller offset values and a margin large enough to cover pixels in the near-center and overlapping areas. Therefore, the pixels of the shared area may not be forcibly divided between overlapping instances.

Meanwhile, the processor 140 may determine a clustering area based on the estimated margin, and determine whether each instance includes a shared pixel based on the determined clustering area.

Further, the processor 140 may sample spatial embedding pixels for determining the center of each instance based on the seed map. The seedmap can control the sampling order until the available foreground area is exhausted.

In step S300, the processor 140 outputs a segmented image of cell nuclei in the histopathology image based on the boundaries of each instance and shared pixels between instances.

Embodiments according to the present disclosure described above may be implemented in the form of a computer program that can be executed on a computer through various components, and such a computer program may be recorded on a computer-readable medium. At this time, the medium is a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a ROM hardware devices specially configured to store and execute program instructions, such as RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the purpose of the present disclosure, or may be known and available to those skilled in the art in the field of computer software. An example of a computer program may include not only machine language code generated by a compiler but also high-level language code that can be executed by a computer using an interpreter or the like.

In the specification of the present disclosure (particularly in the claims), the use of the term "above" and similar indicating terms may correspond to both singular and plural. In addition, when a range is described in the present disclosure, as including the invention to which individual values belonging to the range are applied (unless otherwise stated), each individual value constituting the range is described in the detailed description of the invention Same as

Unless an order is explicitly stated or stated to the contrary for steps comprising a method according to the present disclosure, the steps may be performed in any suitable order. The present disclosure is not necessarily limited to the order of description of the steps. The use of all examples or exemplary terms (eg, etc.) in this disclosure is simply to explain the present disclosure in detail, and the scope of the present disclosure is limited due to the examples or exemplary terms unless limited by the claims. it is not going to be In addition, those skilled in the art can appreciate that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or equivalents thereof.

Therefore, the spirit of the present disclosure should not be limited to the above-described embodiments, and not only the claims to be described later, but also all ranges equivalent to or equivalent to these claims are within the scope of the spirit of the present disclosure. will be said to belong to

1: Bottom-Up Instance Subdivision System
100: bottom-up instance segmentation device
110: Communication Department
120: user interface
130: memory
140: processor
200: user terminal
300: server
400: network

Claims (20)

A bottom-up instance segmentation method, wherein at least a portion of each step is performed by a processor, comprising:
acquiring an image;
encoding the image into a seed map and a plurality of sigma maps based on a pre-learned bottom-up segmentation model to identify boundaries of each instance and shared pixels between instances; and
Outputting a segmented image of an object in the image based on a boundary of each instance and a shared pixel between the instances,
A bottom-up instance segmentation method. According to claim 1,
The pre-learned bottom-up segmentation model,
A learning model learned to cluster each pixel into instances based on the optimal offset to the center of the object,
A bottom-up instance segmentation method. According to claim 1,
Identifying a boundary of each instance and a shared pixel between the instances,
performing pixel-by-pixel spatial embedding on the image; and
Grouping each pixel and shared pixels between the instances based on the spatial embedding result,
A bottom-up instance segmentation method. According to claim 3,
Identifying a boundary of each instance and a shared pixel between the instances,
estimating a margin, which is a distance allowed from the center of each instance to an instance corresponding to the spatial embedding pixel, based on the plurality of sigma maps;
The plurality of sigma maps include three sigma maps for estimating margins in the x-axis direction, the y-axis direction, and the rotation direction of each instance,
A bottom-up instance segmentation method. According to claim 4,
The plurality of sigma maps,
Based on the distance between the spatial embedding pixel and the center of the corresponding instance, a Gaussian function for converting the spatial embedding pixel into a probability belonging to the corresponding instance

Each coefficient of

,

and

is derived as,
Each coefficient above

,

and

is the x sigma value (

), y sigma value (

) and derived based on the rotation angle,
A bottom-up instance segmentation method.
[Gaussian function

]

According to claim 5,
In the step of estimating the margin,
Estimating a margin of each instance based on a sigma parameter of the Gaussian function;
The sigma parameter is each coefficient

,

and

x sigma value of (

), y sigma value (

) and rotation angle,
A bottom-up instance segmentation method. According to claim 6,
In the step of estimating the margin,
Estimating the size and shape of the margin based on the sigma parameter according to the size and shape of each instance,
A bottom-up instance segmentation method. According to claim 6,
In the step of estimating the margin,
Further comprising the step of learning the sigma parameter so that the value of the Gaussian function becomes 1 when the value of the Gaussian function is equal to or greater than a preset probability threshold.
A bottom-up instance segmentation method. According to claim 4,
Identifying a boundary of each instance and a shared pixel between the instances,
determining a clustering area based on the estimated margin; and
Determining whether a shared pixel is included in each instance based on the clustering area,
A bottom-up instance segmentation method. According to claim 9,
Identifying a boundary of each instance and a shared pixel between the instances,
Based on the seed map, sampling a spatial embedding pixel for determining the center of each instance,
A bottom-up instance segmentation method. A computer-readable recording medium having recorded thereon a program that, when executed by at least one processor, causes the at least one processor to perform the method of any one of claims 1 to 10. As a bottom-up instance segmentation device,
Memory; and
a processor coupled with the memory and configured to execute computer readable instructions contained in the memory;
The at least one processor,
the operation of acquiring an image;
Encoding the image into a seed map and a plurality of sigma maps based on a pre-learned bottom-up segmentation model to identify boundaries of each instance and shared pixels between instances, and
Set to perform an operation of outputting a segmented image for an object in the image based on a boundary of each instance and a shared pixel between the instances,
Bottom-up instance segmentation device. According to claim 12,
The pre-learned bottom-up segmentation model,
A learning model learned to cluster each pixel into instances based on the optimal offset to the center of the object,
Bottom-up instance segmentation device. According to claim 12,
The operation of identifying the boundary of each instance and the shared pixel between the instances,
An operation of performing pixel-by-pixel spatial embedding of the image; and
Grouping each pixel and shared pixels between the instances based on the spatial embedding result,
Bottom-up instance segmentation device. 15. The method of claim 14,
The operation of identifying the boundary of each instance and the shared pixel between the instances,
Based on the plurality of sigma maps, estimating a margin, which is a distance allowed from the center of each instance to an instance corresponding to the spatial embedding pixel,
The plurality of sigma maps include three sigma maps for estimating margins in the x-axis direction, the y-axis direction, and the rotation direction of each instance,
Bottom-up instance segmentation device. According to claim 15,
The plurality of sigma maps,
Based on the distance between the spatial embedding pixel and the center of the corresponding instance, a Gaussian function for converting the spatial embedding pixel into a probability belonging to the corresponding instance

Each coefficient of

,

and

is derived as,
Each coefficient above

,

and

is the x sigma value (

), y sigma value (

) and derived based on the rotation angle,
Bottom-up instance segmentation device.
[Gaussian function

]

17. The method of claim 16,
The operation of estimating the margin,
Including the operation of estimating the margin of each instance based on the sigma parameter of the Gaussian function,
Estimating the size and shape of the margin based on the sigma parameter according to the size and shape of each instance;
The sigma parameter is each coefficient

,

and

x sigma value of (

), y sigma value (

) and rotation angle,
Bottom-up instance segmentation device. 18. The method of claim 17,
The operation of estimating the margin,
Further comprising the operation of learning the sigma parameter such that the value of the Gaussian function becomes 1 when the value of the Gaussian function is equal to or greater than a preset probability threshold.
Bottom-up instance segmentation device. According to claim 15,
The operation of identifying the boundary of each instance and the shared pixel between the instances,
An operation of determining a clustering area based on the estimated margin, and
Including an operation of determining whether a shared pixel is included in each instance based on the clustering area,
Bottom-up instance segmentation device. According to claim 19,
The operation of identifying the boundary of each instance and the shared pixel between the instances,
Based on the seed map, sampling a spatial embedding pixel for determining the center of each instance,
Bottom-up instance segmentation device.

Images