RU2795658C1 - Device and method for hip joint diagnosis - Google Patents

Abstract

FIELD: medical images processing.

SUBSTANCE: group of inventions relates to supporting medical decision-making based on the analysis of medical images. A device and method for diagnosing hip joints is proposed, comprising a block for obtaining a three-dimensional medical image of the pelvic region; a landmark search unit configured to automatically detect anatomical landmarks in the obtained image for diagnosing the hip joints and obtain their coordinates, wherein the landmark detection includes generating an initial set of candidate landmarks and iteratively refining the position of landmarks in the set based on the analysis of spatial relationships between landmarks in a set of candidate landmarks using reinforcement learning models based on the training set of images; a diagnostic unit configured to calculate clinical indicators describing the health status of the hip joint based on the data obtained by the landmark search unit and generate a report; and a block for transmitting a report to the device requesting the diagnostics.

EFFECT: group of inventions provides an increase in the speed, accuracy and automation of diagnostics.

16 cl, 7 dwg

Description

The field of technology to which the invention belongs

The present invention relates to the field of image processing, and more specifically to medical decision support based on medical image analysis.

State of the art

Morphological anomalies of the hip joint are among the most common diseases of the human musculoskeletal system, and their diagnosis is usually based on a combination of clinical and radiographic studies. Radiological studies aim to quantify the shape and spatial relationships between the femur, femoral head, and acetabulum using anteroposterior computed tomography (CT) or magnetic resonance (MR) imaging of the joint. The orthopedic community has compiled a list of anatomical indicators characterizing various hip joint anomalies, many of which are asymptomatic in the early, easily treatable stages. In various systematic reviews conducted in the prior art, it was shown, for example, that about 22% of asymptomatic volunteers had signs of hip impingement that could be observed on medical images, and up to 19% of asymptomatic volunteers had undiagnosed hip dysplasia. Identification of asymptomatic or early stage hip abnormalities can be achieved by examining images of the pelvic/abdominal regions of patients obtained for reasons other than imaging of the hip joint. Finding hip anomalies on all such images is time consuming and resource intensive and should not be performed by clinicians. This frees up space for computer-aided diagnostic (CAD) solutions that can independently scan images of asymptomatic patients and alert the user to potential anomalies detected.

Automatic detection and segmentation of the femur and acetabulum on medical images are usually the initial steps in computerized diagnosis of hip anomalies. Early methods in this area were based on heuristics and combined image thresholding, filtering, coupled component decomposition, and so on. to improve bone and bone edge recognition. Many of these algorithms are based on the relative sphericity of the femoral head, allowing it to be reliably identified by matching spherical templates or finding the intersection point of gradient vectors initialized at the edges of the bone. Subsequently, a number of papers were released that tested the idea that the femoral heads and acetabular cups could be approximated by spherical shapes and found that a sphere is already a good approximation to a healthy femoral head with a surface fitting error of around 0. 87 mm, while a more complex superovoid can capture the bone surface with a fitting error of 0.68 mm. However, it has been observed that heuristic-based methods do not work in the presence of hip pathology, especially in people with arthritis. As a result, other methods such as graph analysis and anatomical atlas subsequently emerged for segmenting the femur. One of the disadvantages of such methods is the impossibility of using them to perform morphometry of the structures of the hip joint, since even accurate segmentation of the femoral head and acetabulum does not in itself indicate the presence of insufficient or excessive coverage of the acetabulum. On the other hand, the morphology of the hip joint can be automatically analyzed using statistical shape modeling, in which the tissues of the hip joint are usually defined using cells, and the allowable deformations of such cells, which capture the anatomical variability of normal and pathological joints, are learned from the training set of labeled images. A number of prior art solutions have used statistical shape models to segment the femur, femoral head, acetabulum, other bones, and cartilage. One of the inherent properties of statistical shape models is that they attempt to approximate the shape of the patient's hip joint from the shapes in the training set, potentially making it difficult to segment hips with severe anomalies that were not present in the training set. However, this inability to detect previously unknown anomalies, on the contrary, began to be used to accurately determine their presence. Several studies have demonstrated that the cellular structures of a healthy pelvis can be aligned with a patient with an acetabular deformity to quantify the level of bone loss, and then suggest an optimal acetabular surface for bone reconstruction planning for revision total hip arthroplasty. Despite more accurate segmentation than previously described methods, statistical cellular structures can still fail in the event of unpredictable or hard-to-fix anomalies.

The known approaches do not offer a tool for fast and accurate diagnosis of 3D medical images of the pelvis.

The essence of the invention

In order to overcome at least some of the aforementioned shortcomings of the prior art, the present invention is directed to improving the efficiency of systems, devices and methods for diagnosing hip joints.

According to a first aspect of the present invention, there is provided a device for diagnosing hip joints, comprising:

an image acquisition unit configured to acquire a three-dimensional medical image including the pelvic region;

a landmark search unit configured, using at least one processor and memory, to automatically detect anatomical landmarks in the obtained image for diagnosing hip joints and output their coordinates, the landmark detection including:

- generating an initial set of candidate landmarks based on randomly selected points in the image, based on the average locations of the respective landmarks calculated from the training images, or based on data received from the first pre-trained neural network analyzing the received image;

- iterative refinement of the position of landmarks in the set based on the analysis of spatial relationships between landmarks in the set of candidate landmarks using reinforcement learning models based on the training set of images, and the analysis of spatial relationships includes the sequential application of the graphical lasso method, the second pretrained neural network and Morris sensitivity analysis on the acquired image;

a diagnostic unit configured, using at least one processor and memory, to calculate clinical indicators describing the state of health of the hip joint, based on the data output by the landmark search unit, and generate a report on the state of health of the hip joints, containing at least one calculated clinical score; And

a reporting unit configured to send a report to the device requesting the diagnostics.

In one of the embodiments, the diagnostic unit is further configured to compare the calculated clinical indicators with acceptable limits or ranges, conclude the presence or absence of hip pathologies based on the comparison, and include said conclusion in the report.

In one of the embodiments, the diagnostic unit is configured to generate a report if a pathology is detected in the image, and the report contains an indication of the presence of a pathology and a corresponding at least one calculated clinical indicator confirming the presence of a pathology.

In one embodiment, the pelvic region in the image includes the entirety of both acetabulums and both femoral heads, the space between them, and adjacent portions of the femurs.

In one of the embodiments, the landmark search block is additionally configured to generate a copy of the input image with landmarks applied to it.

In one embodiment, the implementation of the anatomical landmarks for the diagnosis of the hips are the reference points on the image needed to calculate clinical indicators that describe the state of health of the hip joint.

In one embodiment, the clinical measures include: lateral central-octane angle (LCEA), cervical-diaphyseal angle (NSA), anterior acetabular angle (AASA), and posterior acetabular angle.

In one of the embodiments, the diagnostic unit is additionally configured to include in the report a copy of the input image with landmarks applied to it.

In one embodiment, the reporter is further configured to send a copy of the input image with the landmarks detected by the landmark finder to the remote user interface;

the diagnostic unit is further configured to receive input from a user received through the user interface indicating appropriate landmark locations and calculate clinical scores based on the received input from the user.

In one of the embodiments, the diagnostic unit is configured to determine that it should be transmitted to the user interface if the difference between at least one of the calculated clinical indicators and the boundary of the corresponding range of indicators that determine the state of health of the hip joint does not exceed a predetermined threshold .

In one of the embodiments, the diagnostic unit is configured to determine that it should be transmitted to the user interface, based on whether a possible deviation caused by an error in determining the position of the landmark can lead to the value of the corresponding clinical indicator falling outside the range in which it is located. the initially calculated value of the clinical indicator.

In one of the embodiments, the diagnostic unit is configured to determine the position of the deviated landmark by selecting a random point within the area of possible deviation.

In one of the embodiments, the diagnostic unit is configured to determine the point corresponding to the deviated landmark within the area of possible deviation in such a position that provides the maximum deviation of the value of the corresponding clinical indicator in the direction of the nearest limit of the range of this indicator.

In one of the embodiments, the area of possible deviation is specified as a circle with a radius equal to a predetermined error in determining a given landmark, or as a closed curve corresponding to a predetermined spatial distribution of the probability of deviating a given landmark.

In one embodiment, the diagnostic unit is configured to calculate or refine the landmark deviation radius and/or the spatial distribution of the landmark deviation probability by calibrating the system, wherein a plurality of images are transmitted to the user interface for calibration without determining whether they should be transmitted to the user interface.

In one embodiment, the reporter is further configured to send a copy of the input image with the landmarks detected by the landmark finder to the remote user interface;

the landmark search unit is additionally configured to receive from the user interface a copy of the input image with user-specified landmarks printed on it and store it in the training set.

According to a second aspect of the present invention, there is provided a method for diagnosing hip joints, comprising the steps of:

using the imaging unit, a three-dimensional medical image containing the pelvic region is obtained;

using the landmark search block, anatomical landmarks are automatically detected on the received image for the diagnosis of the hip joints and their coordinates are output, and the identification of landmarks includes:

- generating an initial set of candidate landmarks based on randomly selected points in the image, based on the average locations of the respective landmarks calculated from the training images, or based on data received from the first pre-trained neural network analyzing the received image;

- iterative refinement of the position of landmarks in the set based on the analysis of spatial relationships between landmarks in the set of candidate landmarks using reinforcement learning models based on the training set of images, and the analysis of spatial relationships includes the sequential application of the graphical lasso method, the second pretrained neural network and Morris sensitivity analysis on the acquired image;

using the diagnostic unit, calculating clinical indicators describing the state of health of the hip joint, based on the data output by the landmark search unit, and generating a report on the state of health of the hip joints, containing at least one calculated clinical indicator; And

using the reporting transmission unit, transmitting a report to the device requesting the diagnostics.

Technical result

The present invention improves the efficiency of systems and methods for diagnosing hip joints. This provides:

- improving the accuracy of diagnostics;

- increasing the speed of processing three-dimensional images;

- simplification of interpretation and description of images;

- the ability to perform early diagnosis of diseases of the hip joints at stages that are easily treatable;

- release of medical personnel from the routine processes of analyzing images of patients who do not have pathologies and symptoms in the pelvic area;

- reducing the influence of the human factor (mindfulness, fatigue, responsibility).

These and other advantages of the present invention will become apparent upon reading the following detailed description with reference to the accompanying drawings.

Brief description of the drawings

Fig. 1 - examples of magnetic resonance images of the pelvis.

Fig. 2 - examples of anatomical landmarks for the diagnosis of the hip joints.

Fig. 3 - block diagram of the system for diagnosing the hip joints.

Fig. 4 - visualization of the process running in the landmark search block.

Fig. 5 - block diagram of the system for diagnosing hip joints with external correction.

Fig. 6 is an example of ranges of clinical indicators that determine the state of health of the hip joint.

Fig. 7 is an example of landmark deviation.

It should be understood that the figures may be represented schematically and not to scale and are intended primarily to improve understanding of the present invention.

Detailed description

Anatomical landmarks and clinical indicators

Anomalies of the hip joint in practice are usually diagnosed by doctors by manually placing anatomical reference points (landmarks) on the image and calculating, based on them, anatomical distances and angles that characterize the health of the hip joints. The present invention provides a method and system that mimics this diagnostic workflow by automatically detecting anatomical landmarks in pelvic images, allowing for easier interpretation and description of the images, as well as early diagnosis of easily treatable stages of hip disease.

The authors of the present invention have collected an anonymous database of patients with pelvic anomalies. Each patient case consists of one or more T1- (Fig. 1a), T2- (Fig. 1b), and/or PD (proton density)-weighted (Fig. 1c) MRI images and associated diagnostic notes. It should be noted that the example MRI images shown in FIG. 1, for ease of understanding, are only selected two-dimensional coronal sections (slices) from MRI volumetric images.

On each MRI image, an experienced radiologist manually placed 16 anatomical landmarks to assess the pathology of the hip joint, namely two landmarks in the centers of the femoral heads, four landmarks on the axes of both hips, four landmarks on both axes of the femoral head and femoral neck, and six landmarks on the lateral, anterior and posterior edges of both acetabulums. On FIG. 2 illustrates all landmarks (marked with large numbered dots) for one hip. Landmarks include:

1) (Fig. 2a, 2c) the center of the femoral head,

2) (Fig. 2a) lateral edge of the acetabulum,

3), 4) (Fig. 2b) orientation of the femoral shaft (2 landmarks),

5), 6) (Fig. 2b) orientation of the femoral neck (2 landmarks),

7) (Fig. 2c) anterior edge of the acetabulum,

8) (Fig. 2c) the posterior edge of the acetabulum.

These eight landmarks are determined for each femur, resulting in a total of 16 landmarks. Based on these benchmarks, the four most widely used clinical indicators describing hip health can be calculated:

1) Lateral Central-Extreme Angle (LCEA, Wiberg's angle): the angle between the longitudinal axis of the body and the line connecting the center of the femoral head with the lateral edge of the acetabulum (Fig. 2a). Serves to detect and quantify tick-borne impingement of the hip joint, acetabular dysplasia, etc.

2) Neck-diaphyseal angle (NSA): the angle between the axis of the femur and the axis of the head-neck of the femur (Fig. 2b). It serves to characterize the development of the lower extremities and helps to diagnose various diseases.

3) Anterior acetabular angle (AASA): the angle between the line connecting the centers of the heads of the left and right femoral heads and the anterior edge of the acetabulum (Fig. 2c). Serves to quantify under or over coverage of the acetabulum.

4) Posterior acetabular angle (PASA): the angle between the line connecting the centers of the left and right femoral heads and the posterior edge of the acetabulum (Fig. 2c). Serves to quantify under or over coverage of the acetabulum.

The present invention is directed to a novel system and method in which landmark detection is performed automatically or automated. The proposed method is based on the concept that anatomical landmarks have pairwise and group spatial relationships that can be extracted and used to improve the accuracy of landmark detection. From the prior art, systems are known in which to detect landmarks in 2D (two-dimensional) images, models for searching for relationships between landmarks were used, but unlike 2D, in application to 3D, the complexity of traditional models for searching for spatial relationships between landmarks is an obstacle, since the number of landmarks in 3D can exceed thousands when these landmarks, for example, cover the 3D surface of organs. However, the present invention effectively overcomes this obstacle by suggesting that landmark detection be performed as a multi-agent optimization problem, where each landmark is treated as a Reinforcement Learning (RL) intelligent agent and the agent's environment depends on the relationships between the landmarks, whereby the resulting optimization problem is proposed to be solved using the concept of graphical lasso and Morris sensitivity analysis in order to identify the most representative relationships between landmarks that best describe the target object.

General description of the system

As shown in the block diagram in FIG. 3, the pelvic pathology diagnostic system 300 according to the present invention comprises an image acquisition unit 310, a landmark search unit 330, a diagnostic unit 340, and a report unit 350. The image acquisition unit 310 in one embodiment may be a network card or other means of receiving data. In another embodiment, the imaging unit 310 may be an MRI scanner or a CT scanner itself. The landmarking unit 330 and the diagnostic unit 340 are implemented on the basis of a processing device containing a computing module (for example, a central processing unit), a memory module (for example, random access memory and permanent memory), and, if necessary, an additional computing module (for example, a graphics processor or neural CPU). Thus, the locator 330 and the diagnostics 340 may be implemented on a computer or server basis. Reporting unit 350 may be a network card, a hardware data/video interface, or other means of communication.

The method for diagnosing pathologies of the pelvis is performed by the system for diagnosing pathologies of the pelvis and generally consists in the following.

First, a three-dimensional image (MRI or CT) containing the pelvic region is obtained using an imaging unit. In particular, the required pelvic region must contain the entirety of both acetabulums and both heads of the femurs, the space between them, and the adjacent portions of the femurs. In a preferred embodiment, the image should completely contain the pelvic bone, with the acetabulum and femoral heads located approximately in the zone of the central horizontal axis of the image. However, an embodiment is possible in which the upper part of the pelvic bone does not completely fall on the image - this may somewhat reduce the accuracy, but will not fundamentally affect the feasibility of the task, since the upper part of the pelvic bone does not contain the desired anatomical landmarks.

As mentioned above, in one embodiment, if the imaging unit is an MRI or CT scanner, it can acquire an image on its own during an MRI or CT scan. In another embodiment, the image acquisition unit, being part of a pathology diagnostic service, may receive from outside (for example, from a medical organization, from a doctor, from a regional medical data storage system, etc.) a diagnostic request or input data containing one or more (for example, a batch) of images, and extract the desired image from the received input or query.

If necessary, the resulting image is pre-processed in a pre-processing block 320 (not shown in FIG. 3) so that it is suitable for processing by further blocks. For example, if the original image has insufficient or excessive resolution (or the number of slices) in any direction, then the image can be transformed or reconstructed to the required resolution. In a specific non-limiting example, if the original 3D image has 180 spatial slices (cross sections) in one of the directions, and a resolution of 256 slices is required, then a transformation can be performed - for example, by an approximation method, taking into account the physical dimensions of this area. Brightness, contrast and other image parameters can also be adjusted.

The image is then fed to a landmark search box, which automatically detects anatomical landmarks in the image for hip diagnosis. The process performed in the landmark search block will be described in more detail later in this document. As a result of processing, the landmark search block provides the coordinates of 16 landmarks. Also in one of the embodiments, the implementation of the landmark search block generates a copy of the input image with landmarks applied to it.

The output from the landmark finder is passed to the diagnostic unit, which automatically determines the state of health of the hip joints based on them, calculating clinical indicators that describe the state of health of the hip joint, namely: lateral central-contact angle (LCEA) (Fig. 2a), cervical diaphyseal angle (NSA) (Fig. 2b), anterior acetabular angle (AASA) (Fig. 2c) and posterior acetabular angle (PASA) (Fig. 2c); and generates the corresponding report. The diagnostic unit can compare the calculated clinical indicators with acceptable limits or ranges and make a conclusion about the presence or absence of pathologies of the hip joints.

In one embodiment, a hip health report may be generated for each image. In another embodiment, the request for diagnostics or settings of the diagnostic system may include the need to generate a report only for cases in which pathology of the hip joints is detected. For example, if in a received batch of 10 images, the diagnostic unit detects one case of an unacceptable LCEA deviation from the norm, it can generate a report for the corresponding image (or patient) containing the calculated clinical indicators (either all, or at least one that indicates pathology). – in this case, LCEA) and an indication of the likely presence of tick-borne impingement of the hip joint or acetabular dysplasia, and for the remaining images (or patients) either do not generate a report at all, or generate a report indicating that no pathologies were detected on the corresponding images. The report can also contain a copy of the input image generated by the landmark search block with landmarks applied to it.

Further, the report on the state of health of the hip joints is displayed on the screen of a monitor connected to the server, or on the screen of the MRI scanner or CT scanner itself, or transmitted using the report transmission unit to the device that requested diagnostics - for example, to a remotely located workstation ( AWP) doctor. To simplify the description in this document, all of these cases are considered as transfers to the device that requested the diagnostics.

Using the image with landmarks printed on it and other data contained in the report (calculated clinical indicators, indication of the presence / absence of pathology), the doctor makes a decision about the health of the hip joints (the presence / absence of pathology) and, if necessary, decides on treatment patient.

Finding Landmarks

Next, the process performed in the landmark search unit will be described in more detail with reference to FIG. 4.

Proposal of candidate landmarks

RL-based landmark detection is based on the concept of iteratively optimizing the location of landmarks using image intensity information and possibly spatial relationships between landmarks. Iterative optimization requires some initialization in the form of candidate landmark proposals. Different approaches can be used to create candidate proposals.

The first approach defines candidate landmark sentences as random points selected from an image. This initialization is applicable in a situation where the original landmark sentences do not provide any information.

The second approach defines the candidate landmark suggestions as the average locations of the respective landmarks computed from the training MR images. For such an initialization, RL-based optimization starts from the mid-pelvic anatomy.

Also, initializations can be based on candidate sentences discovered using neural networks, such as deep learning models such as UNet and pyramidal feature network (FPN) with region suggestion network (RPN). These networks can learn the position of landmarks on training MR images and then scan previously unprocessed MR images to determine the most likely possible locations for the landmarks. The landmarks and coordinates of the landmark proposals can be normalized to the range [0,1] with respect to the respective image sizes. After the final determination of the position of landmarks on the test image, their coordinates return to the original scale.

UNet for Landmark Candidate Proposal

The UNet architecture was developed for image segmentation and is now widely used in medical imaging due to its versatility and highly accurate segmentation results. It consists of a downsampling encoder path that extracts image features at different scales and complexity, and an upsampling decoder path that generates segmentation results using the extracted features. The basic concept of UNet is to use skip connections that go from the encoder to the decoder and go from the early levels of the encoder path, where fine image details are still preserved, to the later levels of the decoder path, which contain information from coarse features. required for object recognition and segmentation. The encoder path consists of four blocks, each of which consists of two 3 × 3 × 3 convolutional layers, followed by a ReLU activation layer and a 2 × 2 × 2 max-pooling layer. The decoder path consists of four blocks , each consisting of an upsampling layer, a 2×2×2 transposition convolution layer, and two 3×3×3 convolution layers, each followed by a ReLU layer.

To get the landmark suggestions from UNet, the landmark detection problem is first converted to a segmentation problem. For each image l of size l × m × n, the mask of landmark J of size l × m × n with |P| channels, where |P| is the number of references. For a landmark p ∈ P, a binary sphere of radius r = 10 mm is created and inserted into the pth channel J at point p. UNet is trained to generate J masks for I images using a linear combination of binary cross entropy and Dice loss functions with weights of 0.5. For test image I, the candidate sentence for landmark p is computed as the center of mass of the largest connected component from the p-th channel of the corresponding threshold outcome J. In a specific non-limiting example, UNet can be trained using the Adam optimizer with a learning rate of 0.0002 and weight reduction of 0. 0001.

FPN for Landmark Proposal Candidate

The FPN architecture with RPN is based on the principle of efficient multi-scale feature extraction, which allows the combination of semantically strong low-resolution features and semantically weak high-resolution features. This approach is effective for the object detection problem. The FPN component extracts features from the input image, and during this process, it has a certain resemblance to UNet. In FPN, the encoder part is a 3D-ResNet, in a specific example, pre-trained on the combined "Kinetics-700" and "Moments in Time" datasets. The decoder part generates higher resolution feature maps by upsampling the results of previous layers, and each decoder path block generates a separate output for network training. Closing connections consist of 1×1 convolution layers applied to encoder blocks, which are then added to the corresponding decoder blocks. As a result, the FPN tends to generate variable resolution features that can describe the target object. The RPN component analyzes the output of the FPN encoder block to evaluate how descriptive the FPN features are, and simulates a class-independent sliding window object detector that performs object/non-object binary classification and bounding box regression on feature maps. For this purpose, the RPN applies a set of multiple (eg 15) rectangular anchor filters to each voxel of the FPN output array and measures the probability that that voxel matches a landmark from P using the functions in the output array. If anchor filters are computed for a voxel located in close proximity to the true reference, the FPN learns to improve that voxel. The desired neighborhood is modeled by calculating the ratio of bounding box areas (IoU) between anchor filters and a predefined rectangular bounding box enclosing the landmark, where IoU values above 0.7 are considered positive examples of the landmark's location. The FPN output at different scales is then parsed by the RPN to extract the voxels that are most likely to represent landmarks. The coordinates of these voxels are converted to a vector and then passed through two fully connected layers with 1024 features that predict the coordinates of each landmark from P, and the optimal coordinates are used as the final landmark suggestions. In a specific example, an FPN can be trained using the Adam optimizer with a learning rate of 0.0002 and a weight reduction of 0.0001.

Multi-agent RL for landmark correction

After candidate proposals are received, the present invention proposes to iteratively correct them by solving the multi-agent RL problem. To apply the RL methodology, the landmark position optimization is reformulated as a multi-agent Markov decision process, where each agent is associated with the corresponding landmarks from P. The state space S ^p observed by the landmark agent p is defined as follows:

where I(p) is the region of the local MR image extracted around the estimated location p = [x _p , y _p , z _p ] for landmark p, P is a vector of proposals for landmark coordinates, and v ^p is a binary vector indicating which landmarks p are visible ( v ^p = 1 if all landmarks are visible and their coordinates can affect p's coordinates; v ^p contains at least one non-zero element, since p can always see its own coordinates when position is optimized). In other words, the landmark agent p can, while optimizing its position, observe its current location p, the intensity of the image around p, and the locations of some other landmarks from P\p.

. Вектор

пространственной информации определяется как все ненулевые элементы v ^p P, т.е. как текущие координаты всех ориентиров, видимых для p. Векторы

представляют собой

и объединяются в представление текущего состояния w ^p. Представление w ^p затем анализируется с помощью модели RL, в качестве которой может использоваться глубокая Q-сеть (DeepQN), градиент глубокой детерминированной политики (DDPG), градиент глубокой детерминированной политики с двойной задержкой (TD3) или градиент политики актер-критик (A2C). Функция вознаграждения R каждой модели RL определяется как взвешенная сумма между качеством текущего шага агента и историей его перемещений:At each optimization stage, the image patch I( p ) and landmarks v ^p P visible for p are first analyzed by the neural network to create a hidden representation of the current state of the landmark agent p. A given image patch passes through three 3 × 3 × 3 convolutional layers, each followed by a ReLU layer. The output of the convolutional layers is flattened into a one-dimensional vector

. Vector

spatial information is defined as all non-zero elements v ^p P , i.e. as the current coordinates of all landmarks visible to p. Vectors

represent

and are combined into a representation of the current state w ^p . The representation w ^p is then analyzed using an RL model, which can be a Deep Q-Network (DeepQN), a Deep Deterministic Policy Gradient (DDPG), a Double Delay Deep Deterministic Policy Gradient (TD3), or an Actor-Critic Policy Gradient (A2C) . The reward function R of each RL model is defined as a weighted sum between the quality of the agent's current step and the history of his movements:

where p ^k is the position of the landmark at the kth optimization iteration, p ^* is the correct position of the landmark, and λ is a weighting factor for balancing the R terms. The first term of the R function rewards the agent for approaching the correct position of the landmark, and the second term rewards the agent for that he continues to move in the same direction and avoids zigzags.

The loss function L is the weighted combination between the square of the distance error and the change direction penalty:

where d is the number of steps the agent needs to move from the initial position p ⁰ to the correct position p ^* . For DeepQN, DDPG and TD3, the agent can move a certain distance (eg 1mm) in any of the x, y and z directions, while for A2C the agent's movements are unlimited in 3D space. The RL models are trained to improve the position of the landmarks in P given the location of each other, that is, the relationship between the landmarks.

Accordingly, RL-based landmark detection according to the present invention is performed on a new image as follows. For image J, an initial set of candidate landmark sentences P ₀ is first generated. As stated above, these proposals can be determined based on the results of FPN, UNet, or using some other approach. The initial state {J( p ₀ ), v ^p P ₀ } for landmark p is transformed into a state representation w ^p using the network described above and fed through the corresponding RL model for landmark p to generate an optimal move that should improve the current proposed location from r ₀ to r ₁ . After finding improved sentences for all landmarks, the set of landmark sentences is updated to P ₁ . The procedure is repeated until the displacements for all landmarks are less than the threshold value δ or until the specified number of iterations is reached.

Sparse undirected connection graph

The vectors v ^p (Equation 1) are predetermined for each landmark p ∈ P before RL optimization and represent one of the most important components of the proposed system. If v ^p = 1 (i.e., all landmarks are visible for p), then RL models can become too restrictive and bias the landmarks towards average pelvic morphology. On the other hand, if v ^p contains a single non-zero element (that is, landmark p "sees" only itself), then RL models will ignore the spatial relationships between landmarks. Meanwhile, it is necessary to find the optimal multi-orientation environment for each agent, taking into account the spatial relationships between landmarks in P. Three-dimensional coordinates of landmarks for all M images from the training set fill the distance matrix D with size 3|P| × M. The D values are normalized according to the values of each landmark coordinate from the training set. The covariance matrix D describes the degree of correlation of different coordinates of landmarks and, therefore, estimates the spatial relationship between these landmarks. To find the set of the most correlated landmarks, it is proposed to optimize the sparseness of the accuracy matrix Θ = Σ ^-1 . The optimal precision matrix Θ ^* can be found by minimizing the L1-regularized negative log-likelihood:

where | Θ | is the determinant of the matrix Θ , S is the empirical covariance matrix evaluated by D , the trace function tr( ) calculates the sum of the diagonal elements of its input, || Θ || ₁ is the sum of the absolute values, and the parameter γ controls the sparseness of the resulting optimal estimate. Equation (4) is called the graphical lasso problem and can be solved using duality or interior point methods. The solution Θ ^* allows you to select among the remaining coordinates the most correlated landmarks in the training set. After aggregating the Θ ^* values using the 3ⅹ3 kernel, a sparse matrix of the most correlated landmarks is obtained.

Neural networks for estimating the position of a landmark

To perform an analysis of the relationships between a set of landmarks, the predictability of the normalized coordinates of the landmark using the locations of the remaining landmark is estimated by training |P| neural networks. A network _Fp for a landmark p ∈ P is defined as a four-block multilayer perceptron, where the first three blocks consist of dense layers of n _land neurons, followed by layers of batch normalization, ReLU activation, and dropout, and the last block is defined as a dense layer with three neurons , followed by a softmax activation layer, and predicts the 3D coordinates of the p landmark. The network is trained using a loss function based on the root mean square error, supplemented with L1 regularization.

For a landmark p, the corresponding network F _p is trained on the normalized coordinates of the landmarks P\p to predict the coordinate for p using all samples from the training set. When a plausible set of coordinates x _P\p for landmarks P\p passes through the network F _p , it predicts a coordinate x _p = F _p ( x _P\p ) for landmark p. In addition to predicting p coordinates, the F _p network can quantify the relative influence of P\p landmarks on p. For example, if the coordinates of the landmark x _P\p are changed to Δ x _P\p , then the resulting estimate will change to x _p + Δ x _p = F _p ( x _P\p + Δ x _P\p ). This increment Δ x _p allows us to estimate how changes in the coordinates of landmarks P\p affect the coordinate p, and to identify the most spatially dependent subgroups of landmarks.

Analysis of spatial relationships between multiple landmarks

суммарных вкладов каждого ориентира из N(p) и инициализируется нулями (

= 0). Для подмножества ориентиров S из N(p) выборочный вектор вклада

вычисляется как:The purpose of this step is to identify the landmarks that contribute the most to the location of the landmark p ∈ P. First, the neighborhood set N(p) of all potentially related landmarks is filled with the indices of the nonzero elements in the pth row and column Θ ^* . To find out how changes in different subsets of landmarks from N(p) affect the coordinates of the point p, a vector is given

the total contributions of each landmark from N(p) and is initialized to zero (

= 0). For a subset of landmarks S from N(p), the sample contribution vector

calculated as:

where rand(0, σ _k ) is a random 3D bias selected from a normal distribution with zero mean and standard deviation σ _k , such that the values σ = {σ _k } determine the degree of flexibility of the landmark bias. In other words, the contribution vector estimates how random changes in the position of the landmark from S change the predicted position of the landmark p. To calculate σ, the _pI landmark locations for each image I are normalized with respect to translation and rotation:

- центр тяжести ориентиров p _I для изображения I, а R(I_ref, I) - оптимальная матрица поворота, максимально приближающая нормализованные ориентиры (p _I −

) для изображения I к нормализованному ориентиру (p _Iref −

) для предварительно выбранного случайным образом эталонного изображения I_ref. Трехмерное стандартное отклонение σ_i местоположения ориентира p ∈ P вычисляется из нормализованных местоположений

для всех обучающих изображений.Where

is the center of gravity of the p _I landmarks for image I, and R (I _ref , I) is the optimal rotation matrix that approximates the normalized landmarks ( p _I −

) for image I to the normalized landmark ( p _Iref −

) for a pre-selected random reference image I _ref . The three-dimensional standard deviation σ _i of the landmark location p ∈ P is calculated from the normalized locations

for all training images.

увеличивается в соответствии с вкладом выборки

:Vector of total contributions

increases according to the contribution of the sample

:

взвешивается в соответствии с абсолютным смещением |y _S|. Уравнения (5)–(7) повторяются предварительно заданное число раз для оценки изменения координат точки p при различных случайных смещениях ориентиров из S при сохранении оставшихся координат ориентиров x _P\{p,S} неизменными. Такой анализ повторяется для подмножеств ориентиров S разного размера, и полученный вектор

оценивает относительный вклад каждого ориентира из окрестности N(p) в координаты точки p. Ориентиры l с наибольшим относительным вкладом определяют ненулевые элементы v ^p для мультиагентной оптимизации RL (уравнение 1). Следует отметить, что нет необходимости нормализовать вектор

, так как выбираются ориентиры с наибольшим вкладом, и каждый ориентир из N(p) участвует одинаковое количество раз в формировании

. Алгоритм, описанный в этом подразделе, расширяет концепции анализа чувствительности Морриса до оценки пространственной взаимосвязи множества ориентиров.so the sample contribution

weighted according to absolute offset | y _S |. Equations (5)–(7) are repeated a predetermined number of times to estimate the change in the coordinates of the point p for various random displacements of landmarks from S while maintaining the remaining coordinates of the landmarks x _P\{p,S} unchanged. This analysis is repeated for subsets of landmarks S of different sizes, and the resulting vector

evaluates the relative contribution of each landmark from the neighborhood N(p) to the coordinates of the point p. Landmarks l with the largest relative contribution determine non-zero elements v ^p for multi-agent RL optimization (equation 1). Note that there is no need to normalize the vector

, since the landmarks with the largest contribution are chosen, and each landmark from N(p) participates the same number of times in the formation

. The algorithm described in this subsection extends the concepts of Morris sensitivity analysis to assess the spatial relationship of multiple landmarks.

Thus, increased accuracy of determining landmarks in the landmark search block is provided.

External correction

In one embodiment, an external correction of system results may be performed. An example of an externally corrected hip diagnostic system 500 is shown in FIG. 5. Blocks 510-550 in system 500 are similar to blocks 310-350 in system 300 described above and will therefore not be described again here.

The main difference between the system 500 and the system 300 is that the hip health report from the reporting unit 550 can be provided to the doctor via the user interface 560. The user interface can be implemented as a client installed on the doctor's equipment (workstation, computer, laptop , tablet, smartphone, etc.), or as a remote connection to the diagnostic service (for example, via a browser). The diagnostic service is one or more processors, computers, or servers, collectively comprising blocks 510-550, and provides a service or performs the function of diagnosing pathologies on the medical images provided to it. A diagnostic service may also be referred to as a diagnostic device in this document.

The user interface 560 may provide a flat screen 3D image or a virtual 3D image within virtual, augmented, mixed reality (VR, AR, MR), etc. Image rendering can be done either completely on the user's side, or completely or partially on the side of the diagnostic service. The clinician can visually verify that anatomical landmarks have been correctly identified and use the user interface to indicate the correct locations for the landmarks (using a mouse, touch input, VR manipulator, keyboard, etc.). It should be understood that the doctor performing external correction through the user interface 560 is not necessarily the doctor for whom the end result of the system is intended - for example, an external correction may be performed by a specialist specifically engaged for this purpose, while the diagnostic service may provide a service on the diagnosis of pathologies for other doctors and organizations.

The correction in this document is considered as external to the block 530 search for landmarks, that is, the block that first automatically determines the position of anatomical landmarks.

Data about the corrected location of anatomical landmarks (and, if necessary, the resulting image generated in the user interface with corrected landmarks applied to it) is transmitted from the user interface 560 to the diagnostic unit 540, which performs correction of clinical indicators based on them and adjusts the report accordingly.

The corrected report on the state of health of the hip joints is displayed on the screen of the monitor connected to the server, or on the screen of the MRI scanner or CT scanner itself, or is transmitted using the report transmission unit to the device that requested diagnostics.

Thus, an increase in the accuracy of the diagnosis of the hip joints is ensured by checking and correcting the data obtained by a specialist.

In one of the embodiments, it is also possible to transfer to the doctor in the user interface 560 not the health report prepared by the diagnostic block 540, but directly the image with landmarks printed on it, formed by the landmark search block 530, that is, bypassing the diagnostic block 540 (shown in dotted line in Fig. . 5). This allows you to somewhat reduce the time to provide the image to the doctor. However, the doctor may feel uncomfortable not having before his eyes additional data on clinical indicators generated by block 540 diagnostics. As will be shown below, this approach can be useful at the stage of initial setup of the system during its commissioning.

In one embodiment, only cases in which at least one of the calculated clinical scores (LCEA, NSA, AASA, and PASA) is close to the boundaries of the hip health ranges may be submitted for review. For example, as shown in FIG. 6, for the LCEA angle, the normal range of adult hip health may be 25 to 39 degrees, 20 to 25 degrees may indicate borderline dysplasia, 20 degrees or less may indicate dysplasia, and 39 degrees and more may indicate impingement. In this case, the range boundaries are the LCEA angle values of 20, 25, and 39 degrees. If the calculated clinical score is close to the range limit (for example, the difference between the calculated clinical score and the range limit does not exceed a predefined threshold of 0.5 degrees), then such a case can be considered as requiring agreement with the doctor, and the corresponding image is sent to the user interface for verification. interface 560. The proximity threshold to a particular range boundary may be different and depends on the requirements of a particular application. Thus, considering the above statistics on asymptomatic patients and healthy patients, and also that the proportion of patients in whom the calculated clinical indicators are close to the range limit is small relative to the total number of patients, most images in this embodiment will be immediately sent to the device, requesting diagnostics, since they do not need to be coordinated, and only a part will be sent for verification to an external specialist. This ensures simultaneous improvement in diagnostic accuracy, increased processing automation and reduced workload on an external specialist.

In one embodiment, the diagnostic unit 540 may determine whether to send the obtained result to an external specialist for verification, based on whether there is an uncertainty in the diagnostic result, taking into account the possible error in determining the position of landmarks. To this end, the diagnostic block 540, in addition to calculating clinical indicators based on the data on the position of anatomical landmarks received from the landmark search block 530, also determines the possible values of clinical indicators when the landmarks are deviated. The magnitude of the possible deviation of the landmarks is determined by the error in their determination (mean, root mean square, etc.) and is predetermined by the settings of the manufacturer, the organization that installs the system, or the settings of the medical institution itself or the doctor using the hip diagnostic system 500. A circle is built around each landmark with a radius equal to the value of the possible deviation of this landmark, which determines the area of possible deviation. Within each such area, a point different from the center of the area is selected as a possible deviated reference point. Further, based on the positions of the rejected landmarks, the corresponding possible value of the clinical indicator in case of deviation is calculated.

In one embodiment, in each area of possible deviation, one point is randomly selected corresponding to the deviated landmark. The resulting clinical score probable value on deviation is compared to the range limits of that clinical score, and if the computed clinical score probable value on deviation is within the same clinical score range as the originally calculated clinical score value (without benchmark deviation), i.e. both indicators will be on the same side of the range limit, this indicates that external correction may not be appropriate, since even in the case of correction, the alleged diagnosis may not change. In a specific, non-limiting example, diagnostic unit 540 may initially calculate a clinical score of 25.4 degrees, and then, based on a random deviation of the landmark, calculate a possible clinical score with a deviation of 25.2 degrees - in which case both calculated scores will be in range from 25 to 39 degrees, on one side of the border of 25 degrees, which means that the diagnosis, even if the landmark deviates, will remain the same: within the normal range. Accordingly, such a case is not sent for external correction. If the calculated possible value of the clinical indicator with a deviation is within a different range of the clinical indicator compared to the one in which the initially calculated value of the clinical indicator is located (without the deviation of the reference point), that is, the indicators will be on opposite sides of the range boundary, then this indicates that there is uncertainty as a result of the diagnosis, and external correction can significantly affect the proposed diagnosis. Accordingly, such a case is sent for external correction. Thus, the system independently refines the feasibility of correction even near the boundaries of the ranges, which provides an additional increase in processing automation and a reduction in the load on an external specialist while maintaining high diagnostic accuracy.

In another embodiment, the point corresponding to the deviated landmark is not chosen randomly, but in a position that provides the maximum deviation of the value of the corresponding clinical indicator towards the nearest end of the range. In a specific non-limiting example, diagnostic unit 540 may initially calculate a clinical index value of 25.4 degrees. The difference between this value and the limit of the range is 0.4 degrees and does not exceed the pre-set threshold of 0.6 degrees, that is, it may be necessary to refine the position of the landmark. The nearest limit of the range is at the level of 25 degrees, therefore, in order to change the clinical indicator in the direction of this limit, it is necessary that the clinical indicator in case of deviation be less than the calculated one.

On FIG. 7 shows one way to deviate landmarks to decrease the LCEA angle. Points 1 and 2 are landmarks determined by the landmark search block 530 . Bold lines show the sides of the LCEA angle based on points 1 and 2 - namely, the longitudinal axis of the body and the line connecting the center of the femoral head with the lateral edge of the acetabulum. The circles around points 1 and 2 define the areas of possible deviation of landmarks. Point 1* is a deviated landmark obtained at the intersection of the boundary of the area of possible deviation of the landmark 1 and the normal to the longitudinal axis of the body drawn from the landmark 1 in the direction to point 2. Point 2* is the deviated landmark obtained at the intersection of the boundary of the area of possible deviation of the landmark 2 and the normal drawn from landmark 2 in the direction of the tangent circle around point 2, constructed from point 1*. The directions of landmark deflection are shown in Fig. 7 by arrows, and the new longitudinal axis of the body and the line connecting the center of the femoral head with the lateral edge of the acetabulum are shown by dotted lines.

If the possible clinical score value at deviation (LCEA* in the example of Fig. 7) is within a different clinical score range than the originally calculated clinical score value (LCEA in the example of Fig. 7, without benchmark deviation) , that is, the readings will be on opposite sides of the range boundary (for example, LCEA * will be 24.8 degrees, while LCEA is 25.4 degrees), this indicates that the external correction can significantly affect the proposed diagnosis. Accordingly, such a case is sent for external correction. If, however, the calculated possible clinical score at deviation is within the same clinical score range as the originally calculated clinical score (e.g., LCEA* is 25.1 degrees while LCEA is 25.4 degrees), then both scores turn out to be on the same side of the range boundary, this indicates that an external correction may not be appropriate, and such a case is not sent for an external correction. Thus, when checking the feasibility of external correction, the system takes into account the worst-case scenario, which makes it possible to avoid errors that could occur in the absence of external correction, which provides an additional increase in diagnostic accuracy while maintaining high automation of processing and freeing an external specialist from routine correction processes that do not affect diagnosis.

In some embodiments, the selection of a possible deviated landmark may be based on a predetermined probability distribution of the deviation of that landmark. To do this, the boundary of the area of possible deviation around the landmark can correspond to a predetermined threshold value of the probability of deviation - accordingly, in the general case, it can be an arbitrary closed curve other than a circle. The search for the maximum deviation of the value of a clinical indicator can be performed using this boundary. Also, within this boundary, a random selection of the deviated landmark can be performed - based on a uniform distribution or on the basis of the aforementioned known distribution of the deviation probability. Thus, the least probable deviations that could erroneously lead to sending data to an external specialist are eliminated, that is, the accuracy of deciding whether an external correction is appropriate is further increased, which provides an additional increase in processing automation and a reduction in the load on an external specialist while maintaining high diagnostic accuracy.

In one embodiment, the radius of the circle that defines the area of possible deviation of the landmark and/or the spatial distribution of the probability of deviation of the landmark may be predefined by the settings of the manufacturer, the organization that performs the installation of the system, or the settings of the medical institution itself or the doctor using the hip diagnostic system 500. In a specific non-limiting example, the source of data for setting the settings may be studies, results of trial operation, own knowledge of specialists, etc. This speeds up the system commissioning process.

In another embodiment, the landmark deviation radius and/or the spatial distribution of the landmark deviation probability can be calculated or refined by calibration directly in the process of using a particular instance of the system. To do this, the diagnostic unit 540 does not check for the need for external correction, and each image is sent for correction to an external specialist. After receiving corrected data for multiple images via user interface 560 from an external expert, diagnostic unit 540 may calculate the landmark error (mean, root mean square, etc.) and/or the spatial distribution of the landmark deviation probability. The exact number of images required for calibration is determined by the requirements of the particular application. Thus, an increase in the accuracy of determining landmarks is provided, since a medical organization using the proposed system 500 may specialize in the treatment of patients of a certain gender or age category, patients with certain diseases, may have equipment that produces medical images with certain specifics, etc. .d., and this organization may not be suitable for the standard factory settings of the system, while calibration allows you to maximally adapt the system to the specific application.

Calibration can be carried out at the stage of initial setup of the system during its commissioning or, if necessary, already during operation. During calibration, since all images are sent for external correction, the above approach of sending images to the clinician to the user interface 560 directly from the landmark finder 530, i.e. bypassing the diagnostic unit 540 (shown in dotted line in Fig. 5), can be applied. This allows not only to reduce the time for providing the image to the doctor, but also to avoid the overhead of computing resources and time for intermediate data processing by the diagnostic block 540.

Calibration can be performed using data or settings that are already in the system. In this case, presettings can be taken into account with a certain weight. In a specific, non-limiting example, for calculating a landmark error, the pre-setting may have a weight of 0.7, and new data on the deviation of landmarks obtained as a result of external correction can be taken into account with a weight of 0.01, which will allow you to adjust the degree of influence of individual cases on the overall result. and avoid retraining the system according to the set of images used for calibration. The exact weight values depend on the requirements of the particular application. Also, to avoid overfitting, a minimum landmark error threshold can be pre-set so that the system does not over-adjust to the set of images used for calibration.

In one embodiment, the image with the corrected landmarks applied to it, generated in the user interface as a result of external correction, is transmitted from the user interface 560 to the landmark finder 530, which stores this image in the training set. Thus, regular replenishment of the training set is ensured, which makes it possible to improve the accuracy of determining the position of landmarks by the landmark search block 530 . Moreover, when combined with the aforementioned processing performed in the diagnostic block 540, this achieves a synergistic effect whereby reducing the error in the landmarking block 530 can further reduce the error in the operation of the diagnostic block 540, which further increases processing automation and reduces the burden on external specialist.

It should be understood that the arrows from the user interface in FIG. 5 are shown to make it easier to understand which block can be the recipient of data from the user interface, and input from the user received through the user interface 560 can be sent to the landmark finder 530 or diagnostic block 540 not directly, but through the data receiving means, such as the image acquisition block 510 .

Implementation example

The proposed system was tested on a database of 260 patients containing 337 MRI images (in particular, 115 T1, 109 T2 and 113 PD-weighted images), where 180 patients / 251 images (78 T1, 92 T2 and 81 PD) were used for training and 80 patients/86 images (37 T1, 17 T2 and 32 PD) were used for testing. The images were coronally reconstructed to isotropic resolution with a coronal slice size from 320 x 320 mm2 to 370 x 370 mm2 and a number of cross sections from 58 to 180.

The age of patients ranged from 5 to 86 years with a mean of 56 years, 63% of patients were women. Statistics by sex and age are consistent with clinical evidence that pelvic disease is more common in women. All patients suffered from one or another pathology of the pelvis with cartilage thinning, the most common diagnosis was left- and right-sided coxarthrosis. Also, some patients had other anomalies, including pathologies not directly related to the hip joint, and diseases specific to women.

The parameters of the individual components of the system were determined as follows. The RL models for feature location optimization (DeepQN, DDPG, TD3, and A2C) used 10 mm3 I(p) patches of 3D MR images to cover the outer boundaries of the feature location, and the loss functions had an orientation penalty weighted with λ = 0.3, taking into account the root mean square error. The RL-based landmark position optimization continues if there is at least one landmark candidate that moves more than δ = 0.1 mm from its location in the previous iteration. The graphical lasso algorithm was optimized for 100 iterations with a regularization parameter γ = 0.01, which controls the sparseness of the results, and was solved using the python-based scikit-learn package.

Neural networks for analyzing relationships between a set of landmarks had, respectively, n _land = 48, 32 and 16 features for the first three network blocks. The maximum number of landmarks k = 3 determined a random combination of landmarks to test their joint influence on the landmark p ∈ P, and 1000 random movements were generated for each combination.

The learning rate for the RL models and the landmark-based shape network was chosen to be 0.001. RL optimization was terminated after 50 iterations if the landmark position correction did not converge. The framework was trained on a server equipped with a Tesla V100-SXM2-16 GB GPU, 80-core Intel(R) Xeon(R) E5-2698 v4 @ 2.20 GHz, RAM ) 503 GB.

The effectiveness of the system was assessed by evaluating the accuracy of landmarking, calculating the anatomical angle, and detecting and quantifying hip anomalies. When combining the FPN-based landmark suggestions and the proposed shape model, the resulting accuracy was 81%, 100%, 99%, and 100% for LCEA, NSA, AASA, and PASA, respectively. The results obtained were compared with other systems and with human assessments, and it was found that the proposed system can successfully quantify the morphological abnormalities of the hip joint, demonstrating increased accuracy and speed of diagnosis in almost all indicators. In the presence of an external correction mechanism, it is possible to further improve the accuracy of diagnosis, while the help of an external specialist is required only in cases where there is a possibility of an incorrect automatic diagnosis.

Application

The systems, devices, and methods of the present invention can be used to process diagnostic medical images containing the pelvic area for signs of hip joint pathology.

Additional Implementation Features

Although this document may indicate that data is being transmitted/sent or received/received by a person (e.g., medical professional, physician, expert), one skilled in the art should understand that such indication is used solely for the purpose of simplifying the description. , while in fact it is understood that the data is transmitted / sent or received / received by the corresponding device used and / or controlled by this person.

One or more transmission units or devices (transmitters) described herein and one or more receiver units or devices (receivers) may be physically implemented in the same transceiver unit or device or in different units or devices.

A device or a transmission unit in this document, for the sake of simplicity of description, may be referred to as a device or unit having the functions of not only transmitting, but also receiving data, information and/or signals. Similarly, the receiving device or unit may also include data, information and/or signaling functions.

Various illustrative blocks and modules described in connection with the disclosure herein may be implemented or executed by a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA), or other programmable logic device (PLD), discrete logic element or transistor logic, discrete hardware components, or any combination of the foregoing, designed to perform the functions described in this document. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (eg, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors together with a DSP core, or any other similar configuration).

Some blocks individually or collectively may be, for example, a computer, and include a processor that is configured to call and execute computer programs from memory to perform method steps or block functions in accordance with embodiments of the present invention. According to embodiments, the device may further include a memory. The processor may call and execute computer programs from memory to execute the method. The memory may be a separate device independent of the processor or may be integrated with the processor. The memory may store code, instructions, instructions, and/or data for execution on a set of one or more processors of the device described. Codes, instructions, commands may direct the processor to perform the steps of a method or device function.

The functions described herein may be implemented in hardware, software running on one or more processors, firmware, or any combination of the foregoing. The hardware and software implementing the functions may also be physically located in different locations, including such a distribution that parts of the functions are implemented in different physical locations, that is, distributed processing or distributed computing may be performed.

If necessary (for example, if the amount of data and / or calculations that need to be performed on this data is large), multi-threaded data processing can be performed, which in a simple representation can be expressed in the fact that the entire set of data to be processed is divided into a set of subsets , and each processor core performs processing on its assigned subset of data.

The above memory may be volatile or non-volatile memory, or may include both volatile and non-volatile memory. One of ordinary skill in the art will also understand that when referring to memory and storage of data, programs, codes, instructions, instructions, and the like, it is understood that there is a machine-readable (or computer-readable, processor-readable) storage medium. Computer-readable storage media includes both non-transitory computer storage media and communication media, including any transmission medium that facilitates movement of a computer program, or portion thereof, from one place to another. A non-transitory computer-readable storage medium can be any available medium that can be used to carry or store a desired piece of program code in the form of instructions or data structures, and that can be accessed by a computer, processor, or other general purpose or special processing device. destination.

By way of example, and not limitation, computer-readable media may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), flash memory. , Random Access Memory (RAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM) , accelerated speed synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct access bus random access memory (DR RAM), register, cache, semiconductor memories, magnetic media such as internal hard disks and removable disks, magneto-optical media and optical media such as CD-ROMs and digital versatile disks (DVDs), as well as any other storage media known in the art.

The information and signals described herein may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, and elementary signals that may be exemplified in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combinations of the above, if applicable to the present invention.

At least one of the steps in the method or blocks in the device may use an artificial intelligence (AI) model to perform the respective operations. The AI-related function may be executed via a processor and non-volatile and/or volatile memory.

The processor may include one or more processors. At the same time, one or more processors may be a general purpose processor such as a central processing unit (CPU), an application processor (AP) and the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU). ), and/or a specialized AI processor such as a Neural Processing Unit (NPU).

One or more processors direct the processing of input data in accordance with a predefined operating rule or artificial intelligence (AI) model stored in non-volatile memory and/or non-volatile memory. A predetermined operating rule or artificial intelligence model can be obtained through training. In doing so, the processor may perform a pre-processing operation on the data to convert it into a form suitable for use as input to the artificial intelligence model.

"Obtained by training" means that by applying a learning algorithm to a trainable artificial intelligence model using a set of training data, a predefined operation rule or AI model with a desired characteristic is created. The training may be performed on the device itself running the AI according to the embodiment and/or may be implemented via a separate server/system.

An artificial intelligence model may include multiple neural network layers. Each of the multiple layers of the neural network includes a plurality of weights (coefficients) and performs a work operation for a given layer by computing using the plurality of weights of a given layer with respect to the input or calculation result of a previous layer.

Examples of neural networks include, but are not limited to, Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN). ), generative adversarial networks (GANs), and deep Q-nets.

A learning algorithm is a method of training a predetermined target device (eg, a GPU or NPU based neural network) using a set of training data to call, enable, or control the target device to perform a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, partially supervised learning, or reinforcement learning.

It should be understood that although terms such as "first", "second", "third" and the like may be used in this document to describe various blocks, modules, networks, elements, components, regions, layers and/or sections ., these blocks, modules, networks, elements, components, areas, layers and/or sections should not be limited by these terms. These terms are only used to distinguish one block, module, network, element, component, region, layer, or section from another block, module, network, element, component, region, layer, or section. Thus, a first block, module, network, element, component, region, layer, or section may be referred to as a second block, module, network, element, component, region, layer, or section, without departing from the scope of the present invention. As used herein, the term "and/or" includes any and all combinations of one or more of the respective listed positions. Elements mentioned in the singular do not exclude the plurality of elements, unless otherwise specified.

The functionality of an element specified in the description or claims as a single element may be practiced by means of several components of the device, and conversely, the functionality of elements indicated in the description or claims as several separate elements may be practiced by means of a single component.

In one of the embodiments, some or all elements/blocks/modules of the proposed device are located in a common housing, can be placed on the same frame/structure/printed circuit board/crystal and are structurally connected to each other through assembly (assembly) operations and functionally through communication lines . The mentioned communication lines or channels, unless otherwise indicated, are typical communication lines known to specialists, the material implementation of which does not require creative efforts. The communication link may be a wire, a set of wires, a bus, a track, a wireless link (inductive, RF, infrared, ultrasonic, etc.). Communication protocols over communication lines are known to those skilled in the art and are not disclosed separately.

The functional connection of elements should be understood as a connection that ensures the correct interaction of these elements with each other and the implementation of one or another functionality of the elements. Particular examples of functional communication may be communication with the ability to exchange information, communication with the ability to transmit electric current, communication with the ability to transmit mechanical motion, communication with the ability to transmit light, sound, electromagnetic or mechanical vibrations, etc. The specific type of functional connection is determined by the nature of the interaction of the mentioned elements, and, unless otherwise indicated, is provided by well-known means, using principles well-known in the art.

Although the exemplary embodiments have been described in detail and shown in the accompanying drawings, it should be understood that such embodiments are illustrative only and are not intended to limit the present invention, and that the present invention should not be limited to the particular arrangements and structures shown and described, since a person skilled in the art on the basis of the information set forth in the description and the knowledge of the prior art may be obvious various other modifications and embodiments of the invention, without going beyond the essence and scope of this invention.

Claims (30)

1. A device for diagnosing hip joints, comprising: an image acquisition unit configured to acquire a three-dimensional medical image including the pelvic region; a landmark search unit configured, using at least one processor and memory, to automatically detect anatomical landmarks in the obtained image for diagnosing hip joints and output their coordinates, the landmark detection including: - generating an initial set of candidate landmarks based on randomly selected points in the image, based on the average locations of the respective landmarks calculated from the training images, or based on position data of the candidate landmarks obtained from the first pretrained neural network analyzing the received image; - iterative refinement of the position of landmarks in the set based on the analysis of spatial relationships between landmarks in the set of candidate landmarks using reinforcement learning models based on the training set of images, and the analysis of spatial relationships includes the sequential application of the graphical lasso method, the second pretrained neural network and Morris sensitivity analysis on the acquired image; a diagnostic unit configured, using at least one processor and memory, to calculate clinical indicators describing the state of health of the hip joint, based on data output by the landmark search unit, and clinical indicators include: lateral central contact angle (LCEA ), cervical-diaphyseal angle (NSA), anterior acetabular angle (AASA), and posterior acetabular angle (PASA), and generate a hip health report containing at least one calculated clinical score; And a reporting unit configured to send a report to the device requesting the diagnostics. 2. The device according to claim 1, wherein the diagnostic unit is further configured to compare the calculated clinical indicators with acceptable limits or ranges, conclude the presence or absence of pathologies of the hip joints based on the comparison, and include said conclusion in the report. 3. The device according to claim 2, in which the diagnostic unit is configured to generate a report if a pathology is detected in the image, and the report contains an indication of the presence of a pathology and a corresponding at least one calculated clinical indicator confirming the presence of a pathology. 4. The device according to claim. 1, in which the pelvic region in the image includes both acetabulum and both heads of the femur, the space between them and adjacent parts of the femur. 5. The device according to claim. 1, in which the landmark search unit is additionally configured to form a copy of the input image with landmarks applied to it. 6. The device of claim. 1, in which the anatomical landmarks for diagnosing the hip joints are reference points in the image necessary to calculate clinical indicators that describe the state of health of the hip joint. 7. The device according to claim 1, in which the diagnostic unit is additionally configured to include in the report a copy of the input image with landmarks printed on it. 8. The device according to any one of paragraphs. 5 or 7, wherein the reporting unit is further configured to send a copy of the input image marked with the landmarks detected by the landmark finder to the remote user interface; the diagnostic unit is further configured to receive input from a user received through the user interface indicating appropriate landmark locations and calculate clinical scores based on the received input from the user. 9. The device according to claim 8, in which the diagnostic unit is configured to determine that it should be transmitted to the user interface, if the difference between at least one of the calculated clinical indicators and the border of the corresponding range of indicators that determine the health status of the hip joint, does not exceed a predetermined threshold. 10. The device according to claim 8, wherein the diagnostic unit is configured to determine that it should be transmitted to the user interface, based on whether a possible deviation caused by an error in determining the position of the landmark can lead to the output of the value of the corresponding clinical indicator beyond that the range in which the value of the clinical indicator is initially calculated. 11. The device according to claim 10, in which the diagnostic unit is configured to determine the position of the deviated landmark by selecting a random point within the area of possible deviation. 12. The device according to claim 10, in which the diagnostic unit is configured to determine the point corresponding to the deviated landmark within the area of possible deviation in such a position that provides the maximum deviation of the value of the corresponding clinical indicator in the direction of the nearest border of the range of this indicator. 13. The device according to any one of paragraphs. 11 or 12, in which the area of possible deviation is specified as a circle with a radius equal to a predetermined error in determining a given landmark, or as a closed curve corresponding to a predetermined spatial distribution of the probability of deviation of a given landmark. 14. The device according to claim 13, in which the diagnostic unit is configured to calculate or refine the radius of deviation of the landmark and / or the spatial distribution of the probability of deviation of the landmark by calibrating the system, and for calibration, images are transmitted to the user interface without determining whether they should be transmitted to the user interface. 15. The device according to any one of paragraphs. 5 or 7, wherein the reporting unit is further configured to send a copy of the input image marked with the landmarks detected by the landmark finder to the remote user interface; the landmark search unit is additionally configured to receive from the user interface a copy of the input image with user-specified landmarks printed on it and store it in the training set. 16. A method for diagnosing hip joints, comprising the steps of: using the imaging unit, a three-dimensional medical image containing the pelvic region is obtained; using the landmark search block, anatomical landmarks are automatically detected on the received image for the diagnosis of the hip joints and their coordinates are output, and the identification of landmarks includes: - generating an initial set of candidate landmarks based on randomly selected points in the image, based on the average locations of the respective landmarks calculated from the training images, or based on position data of the candidate landmarks obtained from the first pretrained neural network analyzing the received image; - iterative refinement of the position of landmarks in the set based on the analysis of spatial relationships between landmarks in the set of candidate landmarks using reinforcement learning models based on the training set of images, and the analysis of spatial relationships includes the sequential application of the graphical lasso method, the second pretrained neural network and Morris sensitivity analysis on the acquired image; using the diagnostic unit calculates clinical indicators describing the state of health of the hip joint, based on the data output by the landmark finder, and the clinical indicators include: lateral central-area angle (LCEA), neck-diaphyseal angle (NSA), angle of the anterior sector acetabulum (AASA) and posterior acetabular angle (PASA), and generate a hip health report comprising at least one calculated clinical score; And using the reporting transmission unit, transmitting a report to the device requesting the diagnostics.

Images