ES2948438A1 - Computer-implemented method to obtain the fertility status of a woman (Machine-translation by Google Translate, not legally binding) - Google Patents

Abstract

Computer-implemented method to obtain the fertility status of a woman that comprises: obtaining an image (1) of the cervical mucus of the woman in stretching; extract pixel values (2) from the image; processing the pixel values in a module (A) to obtain a feature map (3) in several channels (d); apply to the feature map at least a first two-dimensional convolution operation (B), obtaining a gradient map (4); apply to the gradient map at least a second two-dimensional convolution operation (C) to restore the same number of features as in the feature map, obtaining an attention map (5); for each channel (d) of the feature map perform a weighted average (D) of the feature map, to obtain a weighted feature map (6); and process the weighted feature map to obtain a candidate probability (7) of fertility status. (Machine-translation by Google Translate, not legally binding)

Description

DESCRIPTION

Computer-implemented method to obtain the fertility status of a woman

Technical sector of invention

The computer-implemented method of the present invention allows the fertility status of a woman to be reliably obtained. In this way, it is possible to determine if it is in a fertile or non-fertile state of its fertility cycle.

Background of the invention

There are known methods to try to predict the fertility status of a woman from the study of her cervical mucus, which analyze physical characteristics of the cervical mucus to try to predict the fertility status.

The patent document US2011118996 presents a method and apparatus that, based on the viscoelasticity of cervical mucus, allows predicting the state of fertility. The device is provided with a motor to extrude the cervical mucus, and a camera to obtain photographs of the extruded cervical mucus, predicting the state of fertility from the expansion characteristics obtained from the analysis of the photographs of the cervical mucus. However, in this case a complex and bulky device is required to predict fertility status, which in turn has to be in contact with cervical mucus.

Patent document WO2015/143259 A1 presents a system and method that, by means of a dedicated device in contact with cervical mucus, determines its physical characteristics to compare them with standard characteristics through neural networks and thus determine the state of fertility. However, contact of the device or its sensors with cervical mucus is necessary, which means that it must be cleaned periodically for hygienic reasons and to avoid contaminating subsequent samples, which could give erroneous results.

Methods are also known to obtain fertility status from the automated analysis of a photograph of crystallized cervical mucus, using a fern test. For example, patent document WO2020/092901 A1 presents a system and method that, by means of an optical support system for a sample of cervical mucus, allows obtaining, by a mobile device, an enlarged photograph of a sample of dry cervical mucus and analyze it using neural networks to determine the fertility status. In this case It is also necessary to immediately clean the optical sample support system after use for hygienic reasons and to avoid contaminating samples that are to be analyzed later, which could give erroneous results.

It is therefore an object of the present invention to provide a method that is able to determine the fertility status in a very rapid, hygienic and reliable manner without requiring dedicated devices, so that the fertility status can be determined reliably and safely by part of a user who provides a sample of cervical mucus without specialized knowledge.

Explanation of the invention

The computer-implemented method of the present invention allows obtaining the fertility status of a woman using common devices, such as a mobile phone equipped with a camera.

In essence, this method, which can be programmed to be executed on electronic devices such as computers, comprises the steps of obtaining an image of the woman's cervical mucus stretched, such as stretched between two fingers of a hand, for example, between the finger thumb and index finger. The method further comprises extracting pixel values from the image and processing the pixel values in a processing module to obtain a feature map with two-dimensional features distributed over several channels; apply to the feature map at least a first two-dimensional convolution operation to reduce the number of channels, obtaining a gradient map, and apply to the gradient map at least a second two-dimensional convolution operation to increase the number of channels and restore the same number of features than in the previous feature map, thus obtaining an attention map; for each channel of the feature map, perform a weighted average of the two-dimensional features of the feature map, taking as weights the respective two-dimensional features of the attention map of the same channel, to obtain a weighted feature map; and processing the weighted feature map by a first neural network to obtain a candidate fertility state probability.

Advantageously, through this method it is possible to obtain the fertility status in a reliable and simple way for the user, avoiding that the computing devices have to come into contact with the cervical mucus.

In a variant embodiment, the method further comprises the steps of obtaining a set of additional characteristics based on characteristics of the woman; and processing the candidate probability and the set of additional features through a second neural network to obtain a refined candidate probability.

Thus, it is possible to confirm the previously calculated candidate probability by combining it with other characteristics. This stage is especially useful when the candidate probability is around 0.5, which allows us to finish determining the probability of fertility status, thus obtaining a refined candidate probability.

In a variant embodiment, the processing module comprises a neural network based on Inception v3, so that a previously trained network is used to obtain the image feature map.

In a variant embodiment, the first neural network comprises a fully connected neural network, thus allowing this first neural network to be trained.

In a variant embodiment, the set of additional characteristics comprises characteristic data of the woman categorized according to previously determined characteristic categories.

In a variant embodiment, the method further comprises obtaining a heat map image from the feature map to visually report the most relevant parts of the image, corresponding to the cervical mucus, allowing the method to be performed again, with a new image if the relevant part of the image does not correspond to the cervical mucus, thus avoiding erroneous results due to poor image capture, for example if it is too dark or without contrast.

In a variant embodiment, the activation function of the first neural network is a TanhSoft type function as described in the article "TanhSoft - a family of activation functions combining Tanh and Softplus" (https://arxiv.org/abs /2009.03863).

Also presented is a data processing apparatus comprising means for carrying out the method for obtaining the fertility status of a woman as described above.

In one embodiment, the apparatus comprises a client component and a server component.

Preferably the client component is a portable device that comprises a photographic camera to obtain the image of the cervical mucus, such as a mobile phone, which incorporates an application to execute part or all of the computer-implemented method described.

A computer program is also presented that comprises software code adapted to carry out the method for obtaining the fertility status of a woman when executed on a device of those described above.

Furthermore, a computer-readable storage medium is disclosed that comprises the computer program presented above.

Brief description of the drawings

To complement the description that is being made and in order to facilitate the understanding of the characteristics of the invention, this descriptive report is accompanied by a set of drawings in which, for illustrative and non-limiting purposes, the following has been represented:

Fig. 1 presents an image of a cervical mucus in a stretching arrangement for use in the method of the present invention;

Fig. 2 presents first stages of the method according to the present invention;

Fig. 3 presents second stages of the method according to the present invention;

Fig. 4 presents a gradient map and heat map image of the image in Fig. 1.

Fig. 5 presents third steps of the method according to the present invention;

Fig. 6 presents a formula for carrying out the weighted average of the method step of Fig. 5;

Fig. 7 presents fourth steps of the method according to the present invention;

Fig. 8 presents additional fifth steps of the method of the procedure;

Fig. 9 presents a general scheme of a method according to the present invention; and Fig. 10 presents an apparatus for executing the method of the present invention.

Detailed description of the drawings

Fig. 1 presents a remotely taken image 1 of a stretching cervical mucus for use in the method of the present invention. Specifically in this case, the stretching disposition that the cervical mucus has to adopt is a stretch between the index fingers and thumb of the hand, making a clamp. In this way it is achieved that a user can Hold a sample of your cervical mucus outstretched with one hand, while with the other hand you can hold a device, such as a mobile phone, to capture image 1 of the cervical mucus. Thus, the same user can easily and without additional help capture the image to obtain her fertility status on her own. In other embodiments, it is also provided that the cervical mucus can be stretched using accessory devices, such as forceps. It is expected that a set of instructions will inform the disposition that the cervical mucus must adopt for the capture of image 1 for use in the method, in each case.

Fig. 2 presents some first stages of the computer-implemented method of the present invention to obtain the fertility state, in which, from the previous image 1 taken at a distance, which can be considered an RGB type image, pixel values 2 in each of its three different channels d, performing, if necessary, scaling and adaptation operations for their correct processing in a processing module A to obtain a feature map 3 with two-dimensional features w, h distributed in several channels d. Naturally, the distribution of pixel values 2, as well as feature map 3, must be appropriate for use in processing module A.

This processing module A comprises a neural network, specifically an Inception v3 neural network previously trained to obtain the feature map 3 of the pixel values 2. Although usually the Inception v3 neural network includes a final layer of fully connected type to categorize the images 1 from which the pixel values 2 have been extracted, in this embodiment this final layer of fully connected type has been removed, thus obtaining the feature map 3 with two-dimensional features w, h distributed in several channels d. Naturally, other types of neural networks could be used. The advantage of using a pre-trained neural network is that its structure and weights can be frozen during the training phases of the rest of the neural networks in the set. This advantage is known in the state of the art of pre-trained neural networks. Naturally, in other embodiments other pre-trained neural networks could be used, for example others also based on version 3 of the Inception network created and trained by Google based on ImageNet images, or based on other architectures, such as EfficientNet.

Fig. 3 presents second stages of the method in which it is observed that at least a first two-dimensional convolution operation B will be applied to the feature map 3. to reduce the number of channels d, obtaining a gradient map 4. Next, as shown in the same Fig. 2, at least a second two-dimensional convolution operation C is applied to the gradient map 4 to increase the number of channels d and reset the same number of features as in feature map 3, obtaining an attention map 5.

Fig. 4 presents a visualization of the gradient map 4 in the form of a grayscale image, in which each value of the gradient map 4 would correspond to a pixel gray level at the position of the two-dimensional features w, h, naturally the gradient map 4 would be appropriately scaled to the same size as image 1. By applying this gradient map 4, for example to one or more d RGB channels of image 1, a heat map image 10 identifying the points is obtained. most relevant images of image 1. In this way, this heat map image 10 can be presented to the user during carrying out the method.

As presented in Fig. 5, for each channel d of the feature map 3, perform a weighted average D of the two-dimensional features w, h of the feature map 3, taking as weights the respective two-dimensional features w, h of the attention map 5 of the same channel d, to obtain a weighted feature map 6, in the manner presented in detail in the formula of Fig. 6. Through this weighted average D, it is possible to reinforce the characteristics of the key feature map 3, in the map of characteristics weighted 6, which will later allow us to reliably determine the woman's fertility status.

Fig. 7 presents a fourth stage of the method, in which the weighted feature map 6 previously obtained is processed by a first neural network E, which may be or comprise fully connected neural networks, to thus obtain a candidate probability 7 of state of fertility. Preferably, the activation function of the first neural network E will be of the TanhSoft type since it has the ability to improve the performance of the most common ones, avoiding problems related to gradient fading and also being parameterizable, so that it can be adjust. TanhSoft type activation functions are described in the article "TanhSoft -a family of activation functions combining Tanh and Softplus” (https://arxiv.org/abs/2009.03863).

Fig. 8 presents an additional fifth step of the method of the procedure in which processes the candidate probability 7 and a set of additional features 8 by a second neural network F, which may be or comprise fully connected neural networks, to obtain a refined candidate probability 9. Preferably, the activation function of this second neural network F will be also TanhSoft type.

This second neural network F receives as inputs the candidate probability 7 and the additional feature set 8 that may be based on characteristics of the woman, categorized according to predetermined feature categories. Thus, for example, the set of additional characteristics 8 can be obtained from age, height, weight, eating/smoking/alcohol habits, duration of their fertility cycles, medical and menstrual history, categorized using techniques such as distances between points, clustering/segmentation or embeddings.

Fig. 9 presents a general scheme of the method according to the present invention formed from standard blockchains, in which the processing module A can first be observed, which, based on pixel values 2, generates a map of features 3 using a neural network based on Inception v3, for incorporation into a first two-dimensional convolution operation B, to obtain a gradient map 4, which is subsequently processed in a second two-dimensional convolution operation C to obtain an attention map 5 This attention map 5 together with the feature map 3 is processed in a weighted average module D to obtain the weighted feature map 6.

Finally, this weighted feature map 6 is processed in a first neural network E comprising a fully connected neural network to obtain a candidate probability 7, which could be subsequently refined into a second neural network F along with a set of additional features 8 to thus obtain a refined candidate probability 9.

In a known manner in neural networks, in order to train the network as a whole, a set of training data is provided, in this case a set of images of stretching cervical mucus, labeled according to whether they correspond to a woman in a fertile state, whose candidate probability 7 and/or refined candidate probability 9 of fertility would be '1', or to a woman in a non-fertile state, whose candidate probability 7 and/or refined candidate probability 9 of fertility would be '0'. In a known manner, part of the training data set could form a validation data set, which would be used to validate the training performed, removing them from the training data set.

Thus, during a previous training stage, a backward propagation process would be carried out, according to known techniques in the training of neural networks, where the weights of processing module A would be frozen, if it is a previously trained network, starting the rest of the weights of the neural networks randomly.

In this way, starting from a respective image of the training data set, its candidate probability 7 would be calculated and a back propagation would be performed to adjust the calculated candidate probability 7 so that it was close to the probability of the training data set corresponding to the image of the training data set, using known techniques, such as gradient descent. Thus, each of the chained standard blocks shown in Fig. 9 would be trained by means of backward propagation. In this way, in a known way by means of backward propagation, the weights of the first neural network E, the kernels, would be adjusted. ") of the second two-dimensional convolution operation C and the kernels ("kernels") of the first two-dimensional convolution operation B.

Additionally, if the woman's data is also available to calculate the set of additional characteristics 8, starting from a respective image 1, its refined candidate probability 9 would be calculated and a backward propagation would be carried out to adjust the refined candidate probability 9 calculated so that will approach the probability of the training data set corresponding to image 1, using known techniques, such as gradient descent. Thus, first the second neural network F would be trained by backward propagation, adjusting its weights, and it would continue backwards by training the first neural network E, the second two-dimensional convolution operation C and the first two-dimensional convolution operation B in the manner previously described. .

Naturally the same conveniently adapted method could be used to obtain the fertility status of females of another species of mammal, and could be used, for example, in the field of livestock farming to obtain the fertility status of other species. In this case, the training data set is expected to correspond to images of the stretched cervical mucus of females of the species of interest, labeled according to whether they correspond to a fertile state, whose candidate probability 7 and/or refined candidate probability 9 of fertility would be '1', or to a non-fertile state, whose candidate probability 7 and/or refined candidate probability 9 of fertility would be '0'.

Fig. 10 presents a data processing apparatus 100 comprising means for carrying out the method presented above. This apparatus 100 comprises a client component 110, such as a portable device, for example a mobile phone, and a server component 120, such as a remote server connected via the Internet 112 to the client component 110, for example to perform distributed computing of the method of the present invention. As can be seen, the client component 110 comprises a photographic camera 111 to obtain image 1 of the cervical mucus, the client component 110 being adapted to send the image 1 to the server component 120 through Internet 112, so that the server 120 performs the stages of the method in order to return to the client 110 either the candidate probability 7 or the refined candidate probability 9. Naturally, it is also expected that the client himself will carry out the entire method.

Claims (11)

1. Computer-implemented method to obtain the fertility status of a woman that includes the stages of: - obtain an image (1) of the woman's cervical mucus in stretch; - extract pixel values (2) from the image; - process the pixel values (2) in a processing module (A) to obtain a feature map (3) with two-dimensional features (w, h) distributed in several channels (d); - apply to the feature map (3) at least a first two-dimensional convolution operation (B) to reduce the number of channels (d), obtaining a gradient map (4), - apply to the gradient map (4) at least a second two-dimensional convolution operation (C) to increase the number of channels (d) and restore the same number of features as in the feature map (3), obtaining a map of attention (5); - for each channel (d) of the feature map (3), perform a weighted average (D) of the two-dimensional features (w, h) of the feature map (3), taking the respective two-dimensional features (w, h) as weights. from the attention map (5) of the same channel (d), to obtain a weighted feature map (6); and - process the weighted feature map (6) using a first neural network (E) to obtain a candidate probability (7) of fertility state. 2. Method according to the previous claim, characterized in that it also comprises the steps of - obtain a set of additional characteristics (8) based on characteristics of the woman categorized according to predetermined characteristic categories; - process the candidate probability (7) and the set of additional features (8) using a second neural network (F) to obtain a refined candidate probability (9). 3. Method according to the previous claim, characterized in that the processing module (A) comprises a neural network based on Inception v3. 4. Method according to any one of the preceding claims, characterized in that the first neural network (E) comprises a fully connected neural network. 5. Method according to any one of the preceding claims, characterized in that it further comprises obtaining a heat map image (10) from the feature map (3). 6. Method according to any one of the preceding claims, characterized in that the activation function of the first neural network (E) is a TanhSoft type function. 7. Data processing apparatus (100) comprising means for carrying out the method according to any one of claims 1 to 6. 8. Device (100) according to the preceding claim, characterized in that it comprises a client component (110) and a server component (120). 9. Device (100) according to the preceding claim, characterized in that the client component (110) is a portable device that comprises a photographic camera (111) to obtain the image (1) of the cervical mucus. 10. A computer program comprising software code adapted to perform the method according to any one of claims 1 to 6. 11. A computer readable storage medium comprising the computer program of the preceding claim.