ES2556558A2 - Method and system for the automatic classification of kidney stones, computer program and computer program product (Machine-translation by Google Translate, not legally binding) - Google Patents

Abstract

Method and system for the automatic classification of kidney stones, computer program and computer program product. The method includes: A) acquire an image of a kidney stone; B) analyzing, using computer vision techniques, information contained in the acquired image referred to the image characteristics associated with at least the texture of the kidney stone; y C) classify the kidney stone according to the result of this analysis. The system is apt to implement the method of the invention. The computer program implements the analysis and classification steps of the method of the invention, and the computer program product incorporates the computer program. (Machine-translation by Google Translate, not legally binding)

Description

5

10

fifteen

twenty

25

30

35

DESCRIPTION

Method and system for the automatic classification of kidney stones, computer program and computer program product

Technology Sector

The present invention concerns in general a method and a system for the automatic classification of renal calculi, based on the analysis of images, and more in particular a method and a system that use computer vision techniques to carry out the analysis of images of kidney stones and their subsequent classification as a result of such analysis.

Other aspects of the invention concern a computer program that implements the stages of analysis and classification of the method of the invention, and a product that incorporates such a computer program.

State of the prior art

Kidney stones are usually classified according to their chemical composition as calcium oxalate monohydrate (COM), calcium oxalate dihydrate (COD), calcium oxalate dihydrate transformed into monohydrate (TRA), brushite (BRU), apatite carbonate (CAP) also known as hydroxyapatite (PAH), struvite (STR), anhydrous uric acid (AUA), uric acid dihydrate (AUD), mixed calcium oxalate and apatite carbonate (MXD) calculations, with remarkable variability within each class.

This chemical classification of renal calculi leads to a description of the metabolic alterations that a patient has gone through, and thus, to the selection of a useful treatment to avoid recurrence of colic (recurrence). The specific treatment that each patient can receive should be based on recommendations and dietary restrictions and suggestions on dietary supplements (which can modify some urinary parameters, such as inhibitors and promoters of calculus formation), along with monitoring the levels of some components of the urine. It is important to note that the formation of kidney stones is a disease with an especially high recurrence rate. Therefore, by proper treatment of the patient, the further formation of calculations is drastically reduced. This provides a better quality of life for the patient, along with considerable savings for health organizations. This is well known to

5

10

fifteen

twenty

25

30

35

urologists, but their access to this information is limited due to the partially incomplete results normally offered by the clinical laboratories.

Existing analysis techniques:

Aware of the problem, some techniques have been applied to the analysis of kidney stones. These types of samples are generally classified, using optical and spectroscopic methodologies, according to their chemical composition.

The infrared (IR) spectroscope is the most widespread technique, since it is simple and allows a classification of stones based on the chemical composition and the percentage of the main components of the sample. The strength of this classification lies in the recognition of spectral bands (in the infrared range, defined by wave numbers ranging from 400 to 4000 cm-1), which are directly related to the chemical composition. The three main drawbacks of this technique are:

- The stone must be milled for analysis, so any spatial distribution of the components is inevitably lost.

- The infrared spectra obtained need to be studied by an expert in the area of spectroscopy and with lithiasis knowledge.

- Only then can the results on composition and quantification be obtained, and translated into the most appropriate treatment suggestions.

Although infrared light is sensitive to the chemical components that appear naturally in kidney stones, it is not sensitive enough to detect minor components.

On the other hand, the nature and distribution of the components found in kidney stones induce a characteristic visual appearance, which can be recognized by optical techniques. On the basis of this, a morpho-constitutional analysis can be carried out based on the optical and physical characteristics of the samples, since it is performed using a stereo microscope. Features such as hardness, color and distribution of the components are used to give a classification of the calculations. The sample is cut to observe the inside of the calculation, if necessary. The classes resulting from this type of analysis are not the same as those obtained with infrared spectroscopy, but an extension / specialization of them because this technique is sensitive to the presence of minor components thus offering! a second order classification. Thus, classes do not only depend on the composition

5

10

fifteen

twenty

25

30

35

chemistry, but also of its spatial distribution. The main drawbacks of this technique are:

- It is a technique that consumes a lot of time and that requires a specialized technician duly trained in the recognition of the constitution of calculations.

- Because the recognition is done visually, this is subject to the aptitude and experience of the technician.

In the International application WO2012136874A1, a method for the characterization and classification of renal calculi is proposed, by analyzing different spectra of pre-cut renal calculi, applying a technique of hyperspectral imaging. The information analyzed is relative to the intensity of the radiation reflected in the calculation at different wavelengths. The method proposed in said application comprises analyzing each pixel of each image, so that the image of the renal calculus is divided into a matrix of pixels and each of them is analyzed independently of the neighboring pixels. Hyperspectral analysis is carried out in the near-infrared spectrum (in a spectral range that covers 10001700 nm), taking into consideration all the variables, relative to reflectance measured for all wavelengths, individually in a multiparametric analysis. That is, in the proposal made in WO2012136874A1, a sample is considered as composed of a given number of pixels, each consisting of variables related to reflectance of energies in the near-infrared spectrum. This spectrum carries information about the chemical composition of the stone.

In US8280496B2 a method is described for determining the type of renal calculus, in particular its composition, by illuminating the renal calculus with different wavelengths and comparing the reflectances generated in the renal calculus for each wavelength, classifying the renal calculation based on the differences found in these reflectances for different wavelengths. The method is implemented by an endoscope that carries both the illumination means of different wavelengths and an image sensor sensitive to said wavelengths and that acquires images that include information related to said reflectances.

In Japanese patent document JP2008197081A it is also proposed to perform a classification of calculations based on a multispectral analysis thereof, in

5

10

fifteen

twenty

25

30

35

particular quantifying the components in the calculation based on absorbance values of some spectral bands in the infrared region.

In none of the cited documents is an image analysis performed, understood as relative to the texture and general characteristics of the renal calculus, but simply of its spectral characteristics.

Explanation of the invention

It seems necessary, therefore, to offer an alternative to the state of the art that overcomes the disadvantages of those who suffer from traditional kidney stone analysis techniques.

To this end, the present invention concerns, in a first aspect, a method for the automatic classification of kidney stones, comprising:

a) acquire an image of a kidney stone;

b) analyze information contained in said acquired image; Y

c) classify said renal calculus according to the result of said analysis.

Unlike the known automatic classification methods, in which a spectral analysis of the different parts of the acquired image is performed, in the method proposed by the first aspect of the present invention, characteristically, the analysis of step b ) is carried out using computer vision techniques, preferably together with computational learning techniques, where said information to be analyzed refers to the image characteristics associated with at least the texture of the renal calculus.

According to an example of preferred realization, the acquired image is a digital image, and the analysis of stage b) is carried out at once on the complete digital image acquired in stage a), without performing an individual classification of each pixel.

For an exemplary embodiment, the information to be analyzed refers to associated image characteristics, in addition to texture, to other visual characteristics of the renal calculus, including size and / or shape and / or color.

According to one embodiment, step a) comprises acquiring at least two images of a fragment of the renal calculus, one corresponding to a view of

5

10

fifteen

twenty

25

30

35

an outer surface and another to a view of an exposed inner surface, and analyze them in step b). When the renal calculus fragment has no exposed inner surface, the method comprises cutting it to expose said inner surface whose image is acquired in step a).

Preferably, the method comprises acquiring in stage a) and analyzing in stage b), a plurality of images of corresponding views of each of the outer and inner surfaces of the renal calculus fragment, each of them under conditions of different lighting and / or exposure time.

According to an example of embodiment, the method comprises carrying out said image acquisitions with the same image sensor sensitive to the wavelengths associated with all the illuminations included in said different lighting conditions, which are within the range from the visible light at the initial near-infrared wavelengths.

Alternatively, the method comprises carrying out said acquisitions with several image sensors sensitive, together, to all said different wavelengths.

The method comprises, according to an embodiment example, making said acquisition, in stage a), and analysis, in stage b), of images of the views of the outer and inner surfaces for a plurality of fragments thereof. sample.

The classification of stage c) explained up to here, that is, based on the analysis of the images of the fragment or fragments of renal calculus, is a classification of a first level, or classification of sight, which includes, for each view, a estimate of the probability of belonging to a class of renal calculus associated with chemical composition, from the calculation of a vector of probabilities for each view that includes information on the spatial distribution of said probabilities.

The result of said classification of a first level is considered as a final result for an example of realization, but for another example of more elaborate realization for which the precision required for the result of the classification is superior, in order to obtain such more precise classification, the method comprises performing, in step c), a second level classification, or classification of

5

10

fifteen

twenty

25

30

35

fragment, which comprises combining the results obtained in the classification of a first level for several views of the same renal calculus fragment, to determine a unique class for each fragment based in addition to the chemical composition also in the localization and distribution of the chemical components associated to these probabilities and to what view they correspond.

In the case of obtaining divergent results for different views, the system contemplates the application of a previously defined cost matrix. The costs related to each class are included in a classification vector of each fragment, and are fixed, contrary to the information obtained as a result of the classification of the first level, which depends on the images and measurements taken. The cost values that are applied allow to correct disparate results to give a single class value to a second level.

Optionally, the method comprises correcting the result of the classification of a second level if the result for a given fragment differs from those obtained for each of the views of the same, in the classification of a first level, above a cost value determined. In general, such a determined cost value is defined prior to the analysis, and included in a classification vector, which includes an associated cost value that marks the dependence of such classification.

One way of carrying out said correction comprises reclassifying said fragment taking less into account, or not taking into account at all, the location and distribution of the chemical components associated with the aforementioned probabilities and to what view they correspond, that is, based on everything, or only (in the most extreme case), in the chemical composition.

The result of said classification of a second level is considered as a final result for an example of realization, but for another example of more elaborate realization for which the precision required for the result of the classification is higher, in order to obtain such Even more precise classification, the method comprises performing steps a), b) and c) for two or more fragments of the same sample, the method comprising performing, in step c), a third level classification, or sample classification , which comprises, if the classification of a second level of said fragments is not coincident, assign a unique class for the sample.

5

10

fifteen

twenty

25

30

35

As already indicated above, depending on the precision required for the result of the classification, this may be terminated after the classification of a first level, after the classification of a second level or after the classification of a third level.

According to an example of realization, the method comprises:

- generate, prior to stage a), a set of training, or learning, for a plurality of samples of kidney stones manually classified by an expert, including, in a correlated manner, information on chemical composition, spatial distribution and of visual appearance for internal and external views of different fragments of each sample represented in images obtained with different types of lighting and exposure times, and

- make the classifications of a first, a second and a third level by consulting the images acquired in said training set and extracting the class information correlated with the images more similar to those consulted.

The method comprises training said training set and / or automatic classifiers used to perform at least said first, second and third level classifications using the results of the classifications.

According to an example of embodiment, the method of the first aspect of the invention comprises carrying out the classification of stage c) complementing the analysis of stage b) with additional information relative to the patient from whom the renal calculation and / or obtained from the renal calculus with non-camera based sensors.

With respect to said information related to the patient, this includes at least one of the following information, or a combination thereof:

- simple data related to sex and / or age and / or race and / or complexion and / or Body mass index and / or health disorders associated with renal lithiasis, and / or

- data linked to collateral analyzes, including at least one of the following data collected in the patient's urine analysis: pH, calcium, oxalate, magnesium, ammonium and phosphate.

5

10

fifteen

twenty

25

30

35

As regards said information obtained from the renal calculation with non-camera based sensors, it includes, according to an example of realization, as minimum reflectivity information in other areas of the electromagnetic spectrum not included in the image acquired in a), such as the spectrum corresponding to infrared.

A second aspect of the invention concerns a system for the automatic classification of kidney stones, comprising:

- means of acquiring images to acquire at least one image of a kidney stone; Y

- an electronic system in connection with said image acquisition means and including processing means to process information contained in said acquired image and to classify said renal calculus according to the result of said analysis.

The system proposed by the second aspect of the invention implements the method according to any one of the preceding claims, implementing the means of processing the electronic system one or more algorithms based on computer vision techniques, and preferably also computer learning, to perform steps b) and c) of the method of the first aspect.

Preferably, said image acquisition means comprise a system that has an image focusing mechanism of renal calculi, manually or automatically controllable by the electronic system.

According to an example of realization, the system comprises a housing that defines an interior space that is luminously isolated from the exterior that houses, supported and / or fixed in an internal support structure:

- to a support for samples of kidney stones, which is preferably removable with respect to said support structure and said housing;

- at lighting means arranged to illuminate, with light of one or more wavelengths, the sample (s) of kidney stones arranged on said support;

- to said image acquisition means, which include an image sensor sensitive to said or said wavelengths; Y

- at least a part of said electronic system, which also includes control means to control at least the lighting means.

5

10

fifteen

twenty

25

30

35

The system also optionally comprises one or more sensors sensitive to a range of the electromagnetic spectrum (such as the corresponding infrared spectrum) different from that associated with the image acquisition means, arranged or arranged facing the kidney stone sample (s). arranged on said support, and in connection with the electronic system, to capture the global reflectivity (without performing a pixel analysis) of the sample or samples in a spectral range suitable for its characterization.

According to an example of preferred realization, the entire electronic system is local and is housed inside the housing.

Alternatively, said part of said electronic system housed within the housing is a local part and the electronic system comprises a remote part, such as a computer, communicated bi-directionally with said local part and with the image acquisition means.

A third aspect of the invention concerns a computer program that includes code instructions which, when executed on a computer, implements steps b) and c) of the method of the first aspect.

A fourth aspect of the invention concerns a computer program product comprising the computer program of the third aspect.

According to an exemplary embodiment, the computer program product comprises or is stored or implemented in a medium that may contain, store, communicate, propagate or transport the computer program for use by a system, apparatus or device of execution of instructions, or in connection with it. Said means is or comprises, according to some examples of realization, a physical and / or logical support readable by a computer and / or an electromagnetic, optical or acoustic signal that transports the computer program.

The present invention allows, therefore, in its different aspects, to perform an automatic classification of renal calculi in a way that is useful for doctors, that is, the results are similar to those they are accustomed to manage and comply with their needs in the allocation of treatments for

5

10

fifteen

twenty

25

30

35

patients Although the definition of the classes used for classification is based on the chemical composition of the calculations, the way in which the samples are analyzed according to the present invention is not based on chemical parameters but on the visible characteristics (mainly the texture of the sample ). The advantages that the present invention offers over known techniques, according to its different realization examples, are:

- The classification of the samples is automatic, because the chemical analysis is replaced by a visual analysis done by computer at once, avoiding the need for a qualified technician and, therefore, without depending on their aptitudes.

- The sample is not destroyed, so it can be re-analyzed if necessary.

- The spatial distribution of the components can be performed, which allows the history of the renal calculus to be traced and used to prevent recurrence.

- As regards the system, this is very robust.

- The analysis is done in a few minutes on the same visit to the urologist.

- The doctor receives the diagnosis and treatment proposed by the device and according to the type of renal calculus generated, thus facilitating the work of the urologist, which can be transmitted directly to the patient.

- Low cost of the system, which can be amortized in a short time, since it can be implemented based on a camera, microprocessor and standard lighting components.

- Low cost of the analysis, even in comparison with a chemical analysis, since it can be performed by the urologist himself and not by a trained technician of an infrared spectroscopy service.

Therefore, the present invention is capable of providing the information required by the urologist, but using unconventional techniques that achieve an equal or improved result. In addition, recommendations on diet and treatment to be followed along with the classification are provided. These recommendations are useful for the urologist and their relationship with the specific type of renal calculus not known to most of them.

The present invention has been developed, in its different aspects, by a team composed of experts in image analysis and experts in kidney stone analysis who are continually in contact with urologists who were asked to

5

10

fifteen

twenty

25

30

35

advice constantly on, for example, what kind of information they need and expect.

Brief description of the drawings

The foregoing and other advantages and features will be more fully understood from the following detailed description of some examples of realization, some of which with reference to the attached drawing, which should be taken by way of illustration and not limitation, in which:

Fig. 1 shows, schematically, the system proposed by the second aspect of the present invention, which is apt to implement the method of the first aspect.

Detailed description of some realization examples

In this section a practical implementation of the present invention will be described, both as regards the method and the system, a prototype of which it has been manufactured and will be described later.

First, the implementation of the method proposed by the first aspect of the invention is described, which includes the selection of the samples and the subsequent classification procedure.

Terminology:

- Sample: calculation or calculation fragments generated by a patient during an episode.

- Fragment: Part of a kidney stone that has been obtained directly from the patient (after treatment with extracorporeal shock wave lithotripsy or ESWL), or after cutting an entire stone, in order to leave the discovered its internal part.

- View: Part of the stone or fragment exposed to the chamber. There are two different types of sight, superficial and cutting; that is, the outer or inner part of a fragment or a stone.

- Image: For each view the camera acquires different images, each under a particular lighting source and a particular exposure time.

5

10

fifteen

twenty

25

30

35

The samples were obtained from the Urologla Service of the Hospital Universitari de Bellvitge, Barcelona (Spain). The stones were expelled either naturally (so the whole stone was received) or collected after breaking it by applying a treatment with extracorporeal lithotripsy by shock waves (receiving fragments of the stone). For calculations or non-fragmented stones (whole), these were cut with a surgical knife in order to reach the nucleus. When the sample contains fragments, both the inner and outer part of the stone is generally visible without the need for any manipulation (unless the fragment does not show the core of the stone, then it must be cut). After collection, the stone or fragments were rinsed with water and ethanol and then stored in clean, individual vials. Samples can be stored as well for years without visible signs of decomposition or damage to the structure.

In order to create a suitable database to train the designed system, that is to say a training set, the samples must be chosen carefully, that is, the samples cannot be chosen at random. For this, the samples were selected from a bank of 1300 samples by an expert (a qualified person) in the analysis and classification of kidney stones. Due to the remarkable variability within each class, the selection criterion was chosen in order to reflect this variation in the group of samples selected for each type of stone. The data set includes the stones that comprise all the possibilities for each class in the internal and external part of the stones. These possibilities include the chemical composition (defined by infrared), the distribution of the components and the visual aspect of the sample (defined by the morpho-constitutional analysis), both performed by a qualified specialist. The creation of this library or database of samples is based on the experience of the present inventors in the study of the causes of the formation of calculations and the classification of renal calculi. The results obtained with the method proposed by the present invention cannot be achieved if the stone data set is chosen at random, or if the selected samples do not cover the full range of possibilities for each kind of stone. This can only be developed by an expert in this field, as a result of experimental work on the classification of samples.

As stated above, the samples were chosen in order to cover all the different classes (which could be done from an analysis of the results obtained by infrared spectroscope) and also the different classification of

5

10

fifteen

twenty

25

30

35

second order (what can only be done by a well trained person). The image acquisition procedure has been designed as follows. One or two fragments were selected from each selected sample so that both the inner and outer surfaces can be observed. Then, for each of the two surfaces or views, a series of 6 images were recorded varying the type of lighting source and the exposure time. The choice of the light source depends on the knowledge and experience acquired after analyzing several spectra of kidney stones. The total number of samples selected was 346, from which 606 fragments were selected giving rise to 1212 interior and exterior surface views (and the acquisition and registration of 6 images of each view). Therefore, a total of 7272 images were selected, all of them used for training and validation of a new hierarchical classification scheme, specifically designed for that purpose on three levels by two image processing experts, which has already been explained in an earlier section for an example of realization of the method proposed by the present invention, but which will be described in more detail below and with reference to the experiment set forth herein.

Classifier Scheme:

First level: Classification of views.

Using the set of 6 images of a fragment view, a class is determined for that view as! as the probability of each class for that view is estimated, almost always coinciding with the class determined with the most probable. This class is based entirely on the visual characteristics of size, shape, color and texture (a feature never used before). At this level it is also possible to take into account the pH level of the urine as another characteristic, if this is known. Although the measurement of the chemical characteristics is not used in the present invention for classification, the output classes are comparable to those obtained by the chemical analysis (COD, COM, STR ...) in order to facilitate a classification to the urologist known. The output of this classification is an estimate of the probability or belief that a fragment with such a view (external or internal) belongs to each of the previous classes. Therefore, the method and system developed by the present invention calculates a probability vector for each view of a fragment. From them you can easily infer also the most probable class for a given view, as a first approximation of the class of the fragment.

5

10

fifteen

twenty

25

30

35

For the creation and training of this classifier, all possible species that appear naturally as main components in the stones were included. The types of calculations in this analysis were defined according to the chemical composition (so they are comparable with the IR spectroscope), but were recognized according to visual characteristics. Calcium oxalate monohydrate (COM), calcium oxalate dihydrate (COD), calcium oxalate dihydrate monohydrate (TRA), brushite (BRU), apatite carbonate (CAP), struvite (STR), anhydrous uric acid (AUA) classes were selected , uric acid dihydrate (AUD), mixed calcium oxalate and apatite carbonate (MXD) calculations. This chemical composition is the classification scheme shared with the infrared spectroscopy technique.

Second level: Classification of fragments.

Once for the two views of a fragment, one of these classes has been computed and also a distribution of probabilities, a second classifier produces a fragment class from the results of outputs of the first level classifier. This is a second-order classification, a very useful information that can only be achieved by an expert in morpho-constitutional analysis and not by one in infrared spectroscopy.

The classification of the fragments is done after both views (inner and outer parts) have been assigned to a class. The system is trained in the definition of a single specific class for the fragment based on the combination of the results for the individual views. Therefore, the classification of each fragment is not only limited to the chemical components present in the stone, but also to its location and distribution. For example, the class assigned to a fragment will be different if compound A is inside the stone and B on the surface, or if the situation is the opposite. The definition of these classes by fragments is based on the differences between the possible treatments administered to the patient. The relationship between the possible combinations for internal and external views is given below in Table 1, referring to the classification of fragments according to their internal (Interior) and external (Surface) views.

5

10

fifteen

twenty

25

30

Inside

CD

or

image 1

COM COD TRA MXD HAP STR

 COM

 2 3t 3t 4 5b

 COD

 2c 3 3t 4 5b

 TRA

 2c 3t 3t 4 5b

 MXD

 2ct 3b 3tb 4 5b

 PAH

 2b 3b 3tb 4 5

 STR BRU AUA AUD CYS

 9 6 7

BRU AUA ADD CYS 6.99. 6. . . .

6,. ,,

6. . . .

6 7 ...

6. . . .

6 7 ...

. . 8 8b.

. . 8 8b.

. . . . 10

Table 1. Second level of classification

However, the system may be wrong, that is, an incorrect class is assigned to a fragment, and depending on the difference between the actual and the assigned class

the associated cost will be different (in the second level of classification). In others

words, not all errors that the system can make have the same influence

About the treatment. To do so, the classes described in Table 1, on which the classification is based, can be reordered or assigned as described in Table 2. If such reordering is done, always according to the potential cost, which also It is shown in Table 2 in the range of 0 to 10, 10 being the one associated with the highest error, the class system can be simplified, getting closer and closer to the first level of classification, based only on chemical composition, not on component distribution. That is, the stone classification algorithm can be very strict, but it has also been designed with an important flexibility component. This feature allows the method and system of the present invention to adapt its performance to the specific conditions that the user needs.

Therefore, the values shown in Table 2 can be used as a percentage of probability of one class being assigned, instead, during the training process. For example, if it is known that a sample is of class 2b, the cost if the assigned class (and learned) is 2ct is low, so the model will be freer to assign this other class to the fragment. However, if the same sample 2b is initially recognized as class 6, this decision will be affected and modified by the risk of assigning that class.

As stated above, the criteria used for this transposition are shown in the cost matrix of Table 2, which is set out below. If classes are combined with a low risk of confusion, the classification is simplified to

The first level of classification. Logically, the percentage accuracy of the classification increases as the subtypes of stones decrease (when different groups are combined), since errors usually occur between similar classes, which give a low value in the cost matrix.

RECOGNIZED CLASSES

 2 2b 2c 2ct 3 3b 3t 3tb 4 5 5b 6 7 8 8b 9 10

 2 0 2 2 2 5 5 2 3 6 8 8 9 8 7 7 5 10

 2b 2 0 2 1 5 5 5 5 6 8 8 9 8 7 7 6 10

 2c 2 2 0 1 4 4 4 4 6 8 8 9 8 7 7 6 10

 2ct 3 2 2 0 4 4 3 3 5 8 7 9 8 7 7 6 10

 3 2 3 3 4 0 1 0 3 6 8 8 9 8 8 8 7 10

 c / o 11 1

 3b 2 3 3 3 2 0 2 0 3 7 4 9 8 8 8 7 10

 1 1 _l

 3t 2 3 1 2 0 2 0 0 2 8 6 9 8 8 8 7 10

 <LU

 3tb 2 3 2 1 2 0 1 0 2 6 4 9 8 8 8 7 10

 to:

 4 6 6 6 6 6 4 6 4 0 3 1 7 8 9 9 7 10

 CO LU

 5 8 6 7 6 7 5 7 5 2 0 1 5 3 9 9 8 10

 C / 0

 5b 8 6 7 6 5 3 5 3 1 1 0 5 3 9 9 8 10

 3rd

 6 10 10 10 10 10 10 10 10 7 5 5 0 5 10 10 10 10

 7

 8 7 8 7 8 7 8 7 4 2 2 5 0 9 9 9 10

 8 5 6 5 5 5 6 5 6 8 9 9 10 9 0 0 2 10

 8b 5 6 5 5 5 6 5 6 8 9 9 10 9 0 0 2 10

 9 2 3 2 3 5 6 5 6 8 9 9 10 8 2 2 0 10

 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 0

Table 2. Cost matrix

Third level: classification of the samples.

10 Taking advantage of the fact that two fragments were selected for each stone, a third classifier has been designed in order to offer a final classification. If the most probable class of each of the two fragments both have a probability greater than a certain threshold, if they are the same class, this is the one assigned. If they are not but the two exceed this threshold, the method and system of the present invention has been

15 designed to assign a single class for the sample, using the combination table as seen in Table 3, designed based on experience and knowledge in the analysis of kidney stones. When either of the two classes does not exceed this probability threshold, it is classified based on the output results of the second level classifier.

twenty

 Possible combinations if 2 fragments of the same sample give different K RESULTING CLASSES results

 Fragment one

 Fragment 2 Resulting class Fragm. 1 Fragm. 2 Resulting class Fragm. 1 Fragm. 2 Resulting class

 z2

 2c 2c 2c 2ct 2ct 2ct 9 9

 2

 2ct 2ct 2c 2b 2ct 2ct 3t 3t

 2

 2b 2b 2c 9 9 2ct 3tb 3tb

 2

 9 9 2c 3t 3t 2ct 4 4

 2

 3t 3t 2c 3tb 3tb 2ct 5b 5b

 2

 3 3t 2c 4 4 2ct 5 5b

 2

 3b 3b 2c 5b 5b 2ct 6 6

 2

 3t 3t 2c 5 5b 2ct 7

 2

 3tb 3tb 2c 6 6 2ct 8 9

 2

 4 4 2c 7 2ct 8b 9

 2

 5b 5b 2c 8 9 2ct 10

 2

 5 5b 2c 8b 9

 2

 6 6 2c 10

 2

 7

 2

 8 9 3t 3tb 3tb 3b 3t 3tb

 2

 8b 9 3t 4 4 3b 3tb 3tb

 2

 10 3t 5 5b 3b 4 4

 3t 5b 5b 3b 5b 4

 3

 3t 3t 3t 6 6 3b 5 5b

 3

 3b 3b 3t 7 7 3b 6 6

 3

 4 4 3t 8 9 3b 7 7

 3

 5b 5b 3t 8b 9 3b 8 9

 3

 5 4 3t 9 9 3b 8b 9

 3

 6 6 3t 10 3b 9 9

 3

 7 7 3b 10

 3

 8 9 4 5b 4

 3

 8b 9 4 5 5b 5b 5 5b

 3

 9 9 4 6 6 5b 6 6

 3

 10 4 7 7 5b 7 7

 4 8 5b 8

 3tb

 4 4 4 8b 5b 8b

 3tb

 5b 5b 4 9 5b 9

 3tb

 5 5b 4 10 5b 10

 3tb

 6 6

 3tb

 7 7 6 7 6 8 8b 8b

 3tb

 8 9 6 8 8 9 9

 3tb

 8b 9 6 8b 8 10

 3tb

 9 9 6 9

 3tb

 10 6 10 8b 9 9

 8b 10

 5

 6 6 7 8

 5

 7 7 7 8b 9 10

 5

 8 7 9

 5

 8b 7 10

 5

 9

 5

 10

. Invalid combinations

Table 3. Results of the third level of classification. Fragm.1 or 2 corresponds to the individual fragments. The resulting class is the final result assigned.

5 The output or final result offered by the method and system of the present invention consists not only in the classification of kidney stones, but also in treatment recommendations for the physician. These recommendations depend

5

10

fifteen

twenty

25

30

35

directly from the type of kidney stone detected, related to the direct origin of the stone. The recommendations are linked to a strict classification system that has been specifically defined for this case. In the literature, some classification systems for kidney stones have been defined, but based on the experience of the present inventors these systems have been reconstructed and adapted to this specific problem and combined with the requirements of image analysis. In accordance with the present invention, the stones are classified according to a combination of classes defined in a table designed specifically for this purpose, which considers the outer part and the inner part of the stones. This form of kidney stone classification has never been used previously.

The classification of images based purely on optical parameters is complemented, optionally, as already indicated in a previous section, with additional information from the patient's clinical history. Therefore, the rate of correctly classified samples is improved by considering some parameters such as urine pH, since this is directly related to the type of kidney stone formed. The inclusion of the pH of the urine in the classification of the stone is another exclusive feature of the present invention, since no other methodology using said parameter has been described previously.

The present inventors have designed a prototype of the system proposed by the second aspect of the invention, especially configured to implement the method of the invention. All components were chosen independently to meet both high performance and low cost criteria. The lighting configuration was chosen based on the experience of the present inventors in stone classification (thus the most useful energy ranges for this application were chosen) and image treatment (the physical configuration of the hardware was based on the experience in image analysis and laboratory work, so a high number of lighting configurations were tested before selecting the most appropriate one).

The system proposed by the present invention is schematically illustrated in Figure 2, for an example of realization, for which the system comprises:

- A lighting plate 3, with L LEDs of different wavelengths in the visible and near-infrared spectrum,

5

10

fifteen

twenty

25

30

35

- A control unit 5 (or local part of the electronic system, according to the terminology used in a previous section) to control the intensity and type of light emitted by the LEDs L of the lighting board 3, connected to an external computer 7 (or remote part of the electronic system, according to the terminology used in a previous section),

- A digital camera 4, also connected to the external computer 7, in addition to a suitable optics 8, which includes an extension tube and a lens,

- A housing 1 defining an interior space that is luminously isolated from the exterior to isolate the sample S from ambient light,

- An infrared sensor 6,

- A stand or sample tray 2,

- An internal support structure (not illustrated).

The images of the samples, which are used for the description of the characteristics of the stone necessary for the classification (texture, shape and color), are taken using a conventional camera, equipped with a Silicon sensor. The lighting energies used are in the Visible - Near Infrared (400-1000 nm) range within the sensitivity range of this sensor. The image information can be complemented with the use of other LEDs, which emit at specific wavelengths, allowing measurements of particular reflectance intensities. The illumination plate 3 has been specially designed so that the lenses allow us to observe the renal calculus S, and the location of the LEDs L has been chosen to avoid shadows in the sample S. In addition, an infrared sensor 6 arranged in this plate 3 provides a signal of the reflected light to the computer 7. The operating parameters of the different types of lighting are controlled by the software.

As regards the sample tray 2, in the prototype built this is a mobile sample tray 2 with a homogeneous bottom, designed for the placement of the different samples S in the field of vision of the camera 4, and for optically distinguish the background stone fragment (segmentation). The sample tray 2 has been placed on a mobile platform, which is used for the best placement of the S stone for a simpler approach and lighting adaptation. For another example of realization, the sample tray is not mobile.

A suitable support structure (not shown) has also been designed to support the system elements inside the housing 1. This keeps all

5

10

fifteen

twenty

25

components in the defined position, allowing at the same time that the sample tray 2 can be moved as necessary, both to place the sample in the field of view of the camera and to adjust the focus of the image.

The system operates from a computer 7 using special software designed for this purpose. This software controls the captured image, as well as the lighting conditions. In addition, an appropriate graphical user interface has been designed for the acquisition of stone images, classification and visualization of the result. This interface allows you to name the sample and the collection and storage of several images, as well as the associated near-infrared data for each stone fragment, and also for more than one stone fragment for each sample (stones with ESWL origin generally consist of several fragments). Once all the images and data of the stones of a given sample were recorded, they were classified using a set of supervised algorithms designed exclusively for the present invention. In addition to these images and IR information, other patient data, relevant for classification, such as age, sex and urine pH level can be entered. The results of the classification are given as the probability distribution for each class based on the degree of belief at the end of perception, illustrated, for example, by a diagram.

A person skilled in the art could introduce changes and modifications in the examples of realization described without departing from the scope of the invention as defined in the appended claims.

Claims (26)

5 10 fifteen twenty 25 30 35 1. - Method for the automatic classification of kidney stones, comprising: a) acquire an image of a kidney stone; b) analyze information contained in said acquired image; Y c) classifying said renal calculus according to the result of said analysis; the method being characterized in that said analysis of said stage b) is carried out using computer vision techniques, where said information to be analyzed refers to the image characteristics associated with at least the texture of the renal calculus. 2. - Method according to claim 1, characterized in that for the analysis of stage b) computational learning techniques are also used. 3. - Method according to claim 1, characterized in that said acquired image is a digital image, and that the analysis of stage b) is carried out at once on the complete digital image acquired in stage a), without performing a individual classification of each pixel. 4. - Method according to claim 1, 2 or 3, characterized in that said information to be analyzed refers to associated image characteristics, in addition to texture, to other visual characteristics of the renal calculus, including size and / or shape and / or color . 5. - Method according to any one of the preceding claims, characterized in that said step a) comprises acquiring at least two images of a fragment of said renal calculus, one corresponding to a view of an outer surface and another to a view of an inner surface exposed, and analyze them in stage b). 6. - Method according to claim 5, characterized in that if said renal calculus fragment has no exposed inner surface, the method comprises cutting it to expose said inner surface whose image is acquired in step a). 7. - Method according to claim 5 or 6, characterized in that it comprises acquiring in stage a) and analyzing in stage b), a plurality of images of corresponding views of each of said exterior and interior surfaces of said calculation fragment renal, each of them under different lighting conditions and / or exposure time. 8. - Method according to claim 7, characterized in that it comprises carrying out said acquisitions with the same image sensor sensitive to the wavelengths associated with all the illuminations included in said different lighting conditions, which are within the range from visible light to 5 10 fifteen twenty 25 30 35 near infrared, or with several image sensors sensitive, together, to all such different wavelengths. 9. - Method according to any one of claims 5 to 8, characterized in that it comprises performing said acquisition, in stage a), and analysis, in stage b), of images of said views of said exterior and interior surfaces for a plurality of fragments of the same sample. 10. - Method according to any one of claims 5 to 9, characterized in that said classification of said stage c) based on the analysis of said images of said fragment or said fragments of renal calculus, is a classification of a first level, or classification of view, which includes, for each view, an estimate of the probability of belonging to a class of renal calculus associated with chemical composition, from the calculation of a vector of probabilities for each view that includes information on the spatial distribution of said odds 11. - Method according to claim 10, characterized in that it comprises performing, in step c), a classification of a second level, or fragment classification, which comprises combining the results obtained in the classification of a first level for several views of a same fragment of renal calculus, to determine a unique class for each fragment based in addition to chemical composition also in the localization and distribution of the chemical components associated to said probabilities as they correspond. 12. - Method according to claim 11, characterized in that it comprises correcting the result of said classification of a second level if the result for a given fragment differs from those obtained for each of the views thereof, in the classification of a first level, above a certain cost value. 13. - Method according to claim 12, characterized in that said correction comprises reclassifying said fragment taking less into account, or not taking into account at all, the location and distribution of the chemical components associated with said probabilities and to which view they correspond. 14. - Method according to claim 11, 12 or 13, characterized in that It comprises performing said steps a), b) and c) for at least two fragments of the same sample, the method comprising performing, in step c), a third level classification, or sample classification, comprising, if the classification of a second level of said fragments, which are at least two, it is not coincidental, to assign a unique class for the sample. 15. - Method according to claim 10, 11 or 14, characterized in that understands: 5 10 fifteen twenty 25 30 35 - generate, prior to said stage a), a set of training, or learning, for a plurality of samples of kidney stones manually classified by an expert, including, in a correlated manner, information on chemical composition, spatial distribution and of visual appearance for internal and external views of different fragments of each sample represented in images obtained with different types of lighting and exposure times, and - make these classifications of a first, a second and a third level by consulting the images acquired in said training set and extracting the class information correlated with the images more similar to those consulted. 16. - Method according to claim 15, characterized in that it comprises training said training set and / or automatic classifiers used to perform at least said classifications of a first, a second and a third level using the results of the classifications. 17. - Method according to any one of the preceding claims, characterized in that it comprises carrying out said classification of stage c) by complementing the analysis of stage b) with additional information relative to the patient from whom the renal calculation comes from and / or obtained from the renal calculus with non-camera based sensors. 18. - Method according to claim 17, characterized in that said information relating to the patient includes at least one of the following information, or a combination thereof: - simple data related to sex and / or age and / or race and / or complexion and / or Body mass index and / or health disorders associated with renal lithiasis, and / or - data linked to collateral analyzes, including at least one of the following data collected in the patient's urine analysis: pH, calcium, oxalate, magnesium, ammonium and phosphate. 19. - Method according to claim 17 or 18, characterized in that said information obtained from the renal calculation with non-camera based sensors includes at least reflectivity information in other areas of the electromagnetic spectrum not included in said image acquired in a). 20. - System for the automatic classification of kidney stones, comprising: - means of acquiring images to acquire at least one image of a kidney stone; 5 10 fifteen twenty 25 30 35 - an electronic system in connection with said image acquisition means and including processing means to process information contained in said acquired image and to classify said renal calculus according to the result of said analysis; the system being characterized in that it implements the method according to any one of the preceding claims, said processing means implementing at least one algorithm based on computer vision techniques to perform steps b) and c) of the method. 21. - System according to claim 20, characterized in that said algorithm implementing in said processing means is also based on computational learning, to perform steps b) and c) of the method according to claim 2. 22. - System according to claim 21, characterized in that it comprises a housing (1) defining an interior space insulated externally from the exterior that houses, supported and / or fixed in an internal support structure: - to a support (2) for samples (S) of kidney stones; - at lighting means arranged to illuminate, with light of one or more wavelengths, the sample (S) of kidney stones arranged on said support (2); - to said image acquisition means, which include at least one image sensor (4) sensitive to said or said wavelengths; Y - to at least part of said electronic system which also includes control means (5) to control at least the lighting means. 23. - System according to claim 22, characterized in that it also comprises one or more sensors (6) sensitive to a range of the electromagnetic spectrum different from that associated with said image acquisition means, arranged or arranged facing the sample (s) (S) of kidney stones arranged on said support (2), and in connection with the electronic system, to capture the overall reflectivity of the sample or samples (S) in a spectral range suitable for its characterization. 24. - System according to claim 22 or 23, characterized in that said part of said electronic system housed within the housing (1) is a local part (5) and that the electronic system comprises a remote part (7) communicated bi-directionally with said local part (5) and with the means of image acquisition. 25. - Computer program that includes code instructions that, when executed on a computer, implements steps b) and c) of the method according to any one of claims 1 to 19. 26. - Computer program product comprising the computer program of claim 25.