WO2023088275A1

WO2023088275A1 - Automatic roi positioning method and apparatus, surgical robot system, device and medium

Info

Publication number: WO2023088275A1
Application number: PCT/CN2022/132130
Authority: WO
Inventors: 白全海; 刘鹏飞; 刘赫
Original assignee: 苏州微创畅行机器人有限公司
Priority date: 2021-11-19
Filing date: 2022-11-16
Publication date: 2023-05-25
Also published as: CN114255329A

Abstract

The present invention discloses an automatic ROI positioning method, comprising: obtaining original image data; preprocessing the original image data to obtain a coronal image sequence and a sagittal image sequence; positioning the ROI of the images in the coronal image sequence and the sagittal image sequence; integrating the ROI positioned in the coronal image sequence to obtain a coronal plane ROI integration area, and integrating the ROI positioned in the sagittal image sequence to obtain a sagittal plane ROI integration area; and performing coordinate transformation on the coronal plane ROI integration area and the sagittal plane ROI integration area to obtain three-dimensional coordinates of a target ROI in the original image data.

Description

ROI automatic positioning method, device, surgical robot system, equipment and medium

This application claims the priority of the Chinese patent application No. 202111391566.8 filed on November 19, 2021, which is hereby incorporated by reference in its entirety.

technical field

The present application relates to the technical field of image processing, in particular to an ROI automatic positioning method, device, surgical robot system, equipment and media.

Background technique

The main functions of human joints are connection and movement. The non-invasive acquisition of medical images of the internal joints of a human body or part of a human body for medical treatment or medical research. The medical image may be a magnetic resonance imaging (Magnetic Resonance Imaging, MRI) image, may also be a computerized tomography (Computed Tomography, CT) image, or may be a positron emission computed tomography (Positron Emission Computed Tomography, PET) image, Ultrasound images and the like may also be used. Process the obtained medical images and use them as auxiliary means for subsequent clinical diagnosis and treatment.

Contents of the invention

Based on this, according to the embodiments of the present application, a ROI automatic positioning method, device, surgical robot system, equipment, and medium are provided to reduce the operator's manual positioning of joints and other ROI operations, improve the accuracy of ROI positioning, and improve work efficiency. Simplify the operation process of navigation software for related operations, and improve the versatility of navigation software.

The present application provides a ROI automatic positioning method, comprising: acquiring original image data; preprocessing the original image data to obtain a coronal image sequence and a sagittal image sequence; The ROI in the image is positioned; the ROI positioned in the coronal image sequence is integrated to obtain the coronal ROI integration area, and the ROI located in the sagittal image sequence is integrated to obtain the sagittal ROI integration area; the coronal ROI integration area is obtained; Coordinate transformation is performed on the plane ROI integration area and the sagittal plane ROI integration area to obtain the three-dimensional coordinates of the target ROI in the original image data.

In an embodiment of the present application, before the positioning of the ROI in the images of the coronal view sequence and the sagittal view sequence, further includes: Classifying the images in the sequence, determining whether each image in the coronal image sequence and the sagittal image sequence contains an ROI; filtering images that do not contain the ROI in the coronal image sequence and the sagittal image sequence.

In an embodiment of the present application, the preprocessing the original image data to obtain the coronal image sequence and the sagittal image sequence includes: performing standardization processing on the original image data to obtain standardized three-dimensional image data; according to the Standardize the three-dimensional image data to obtain the coronal image sequence and the sagittal image sequence.

In an embodiment of the present application, the standardization processing of the original image data to obtain standardized three-dimensional image data includes: obtaining parameters of the original image data; obtaining target image parameters and an image transformation interpolation algorithm; The target image parameters and the image transformation interpolation algorithm perform standardization processing on the original image data to obtain the standardized three-dimensional image data.

In an embodiment of the present application, the parameters of the original image data include at least one of the shooting direction angle, resolution, origin coordinates and three-dimensional size of the original image; the target image parameters include at least the target shooting direction Angle, target resolution, target origin coordinates, and target 3D size.

In an embodiment of the present application, by classifying the images in the coronal image sequence and the sagittal image sequence, it is determined whether each image in the coronal image sequence and the sagittal image sequence is Before including the ROI, the ROI automatic positioning method further includes: performing window width and window level processing on the images in the coronal image sequence and the sagittal image sequence.

In an embodiment of the present application, by classifying the images in the coronal image sequence and the sagittal image sequence, it is determined whether each image in the coronal image sequence and the sagittal image sequence is Before including ROI, further include:

Perform normalization processing on the images in the coronal image sequence and the sagittal image sequence.

In an embodiment of the present application, the locating the ROI in the images of the coronal image sequence and the sagittal image sequence includes: locating all ROIs in the coronal image sequence and the sagittal image sequence performing feature extraction on the image; predicting position information of the ROI contained in the images in the coronal image sequence and the sagittal image sequence according to the extracted features.

In an embodiment of the present application, the location information includes center point coordinates and size information of the ROI.

In an embodiment of the present application, integrating the ROI positioned in the coronal image sequence to obtain a coronal ROI integration area, and integrating the ROI positioned in the sagittal image sequence to obtain a sagittal ROI integration area includes : Based on the non-maximum value suppression algorithm, integrate the overlapping parts of the ROI positioned in the coronal image sequence to obtain the target coronal ROI; based on the non-maximum value suppression algorithm, integrate all the positioned in the sagittal image sequence The overlapping part of the ROI is used to obtain the target sagittal plane ROI; the target coronal plane ROI and the target sagittal plane ROI are respectively clustered to obtain the coronal plane ROI integration area and the sagittal plane ROI integration area.

In an embodiment of the present application, performing clustering processing on the target coronal ROI and the target sagittal ROI respectively to obtain the coronal ROI integration area and the sagittal ROI integration area, comprising: performing clustering processing on the target coronal ROI according to a k-means clustering algorithm to obtain an integrated area of the coronal ROI; performing clustering processing on the target sagittal ROI according to a k-means clustering algorithm, The sagittal plane ROI integration area is obtained.

In an embodiment of the present application, performing clustering processing on the target coronal ROI according to the k-means clustering algorithm to obtain the coronal ROI integration area includes: selecting a plurality of the target coronal ROIs As a cluster center; according to the intersection of each cluster center and other said target coronal planes, the target coronal plane ROI is clustered to obtain the coronal plane ROI integration area; said clustering according to k-means The similar algorithm performs clustering processing on the target sagittal plane ROI to obtain the integrated region of the sagittal plane ROI, including: selecting a plurality of the target sagittal plane ROIs as cluster centers; The intersection and union comparison of the target sagittal plane performs clustering processing on the target sagittal plane ROI to obtain the integration region of the coronal plane ROI.

In an embodiment of the present application, performing coordinate transformation on the coronal plane ROI integration area and the sagittal plane ROI integration area to obtain the three-dimensional coordinates of the target ROI in the original image data includes: according to the formula P _ct (X, Y, Z)=P _ct (x _i , x _j , (y _i +y _j )/2), performing coordinate transformation on the coronal plane ROI integration area and the sagittal plane ROI integration area, Obtain the three-dimensional coordinates of the target ROI; wherein, P _ct (X, Y, Z) represents the three-dimensional coordinates of the points in the target ROI in the original image data, and P _i ( _xi , y _i ) represents the The coordinates of the point in the i-th coronal ROI integration area, and P _j (x _j , y _j ) represent the coordinates of the point in the j-th sagittal ROI integration area.

The present application also provides a ROI automatic positioning device, including: a data acquisition module for acquiring original image data; a data preprocessing module for preprocessing the original image data to obtain a coronal image sequence and a sagittal image sequence The positioning module is used to locate the ROI of the image in the coronal image sequence and the sagittal image sequence; the integration module is used to integrate the ROI positioned in the coronal image sequence to obtain a coronal ROI integration area, Integrating the ROIs positioned in the sagittal image sequence to obtain a sagittal plane ROI integration area; a coordinate transformation module for performing coordinate transformation on the coronal plane ROI integration area and the sagittal plane ROI integration area to obtain the The 3D coordinates of the target ROI in the original image data.

The present application also provides a surgical robot system configured to execute any one of the methods for automatic ROI positioning described above.

The present application also provides a computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, any one of the above ROI automatic positioning methods is implemented.

The present application also provides a computer-readable storage medium, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by one or more processors, the one or more processors perform any of the above-mentioned The ROI automatic positioning method.

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features, objects and advantages of the invention will be apparent from the description, drawings and claims.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present application, and those of ordinary skill in the art can also obtain other drawings according to the disclosed drawings on the premise of not paying creative efforts.

Fig. 1 is a schematic diagram of the joint position in the coronal view and sagittal view of the medical image manually marked by the operator based on the navigation software in the related art of the present application;

FIG. 2 is an application environment diagram of the ROI automatic positioning method according to an embodiment of the present application;

FIG. 3 is a flowchart of an ROI automatic positioning method according to an embodiment of the present application;

Fig. 4 is the flowchart of the preprocessing of the original image data according to an embodiment of the present application;

FIG. 5 is a flow chart of standardization processing of original image data according to an embodiment of the present application;

FIG. 6 is a flowchart of determining whether an ROI is included in a coronal image sequence and a sagittal image sequence according to an embodiment of the present application;

Fig. 7 is a flow chart of classifying coronal image sequences and sagittal image sequences according to an embodiment of the present application;

FIG. 8 is a flow chart of image preprocessing in a coronal image sequence and a sagittal image sequence according to an embodiment of the present application;

FIG. 9 is a flowchart of ROI positioning in an image according to an embodiment of the present application;

Fig. 10a is a heat map of a target area according to an embodiment of the present application;

Fig. 10b shows the prediction result of the offset of the center point of the target area according to an embodiment of the present application;

Fig. 10c shows the prediction results of the length and width of the target area according to an embodiment of the present application;

FIG. 11 is a flow chart of integrating multiple overlapping ROIs based on a non-maximum value suppression algorithm according to an embodiment of the present application;

FIG. 12 shows a process of integrating multiple overlapping ROIs based on a non-maximum value suppression algorithm according to an embodiment of the present application;

FIG. 13 is a flow chart of integrating multiple overlapping ROI coordinate frames using an NMS algorithm combined with a clustering algorithm according to an embodiment of the present application;

14 is a schematic diagram of multiple ROI coordinate frames that integrate and overlap multiple ROIs using an NMS algorithm combined with a clustering algorithm according to an embodiment of the present application;

Fig. 15 is a schematic diagram of transforming the coordinates of the ROI in the coronal plane and the sagittal plane to the three-dimensional coordinates of the ROI in the original image;

FIG. 16 is a schematic structural diagram of an automatic ROI positioning device according to an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, not to limit the present application.

FIG. 1 is a schematic diagram of an operator's known manual marking of joint positions in coronal and sagittal views of a medical image based on navigation software. As shown in Figure 1, before joint surgery, the operator first manually marks the joint position in the coronal view and sagittal view of the medical image based on the navigation software, and then the navigation system automatically gives the three-dimensional coordinates of the joint. This manual marking method is time-consuming and labor-intensive, which increases the workload of the operator. The operator needs to locate different joints in different views. For example, in the navigation software, the left and right knee joints and hip joints need to be marked in the coronal view and sagittal view. The operator needs to perform 8 operations, which greatly Increase the labor intensity of the operator. In addition, the joint positioning standards of different operators are not uniform, which leads to inconsistent positions of the selected joints, low joint positioning accuracy, and further leads to inaccurate 3D reconstruction models of joints. Finally, the operator needs to manually mark the joint position. There is no general positioning method to adapt to the positioning of different joints. Different program codes need to be developed for different joints, which is not conducive to the platform construction of navigation software.

This application provides a method for automatically locating a region of interest (region of interest, ROI), which can be applied to the application environment shown in FIG. 2 . The application environment includes a surgical trolley 1, a mechanical arm 2, a tool target 21, a femoral target 22, a tibial target 23, a base target 24, a pointed target 241, an osteotomy guide tool 31, an oscillating saw 41, and an NDI navigation device 51 , auxiliary display 52, navigation trolley 61, main display 62, keyboard 63 and operating bed 81. Using the ROI position automatically positioned by the application, the operator can perform preoperative operations such as key point marking in the corresponding three-dimensional model through the navigation software. During surgery, surgical treatment is performed using each device in the application environment diagram.

This application provides a method for automatic ROI positioning, and does not limit the subject of its execution. Optionally, the ROI automatic positioning method may be implemented by an ROI positioning device, and its executing body may be a computer processor. Before the operation, the operator imports the three-dimensional model described by the medical image sequence, and triggers the ROI automatic positioning method of the present application. By running the ROI automatic positioning method of the present application, the operator can determine the position in the imported three-dimensional model in the display interface Each ROI. The specific process of data processing from the operator importing the medical image sequence until triggering the ROI automatic positioning method may include: reading user data through navigation software; opening up shared memory and writing data; starting the ROI automatic positioning device through message notification.

Fig. 3 is the flowchart of the ROI automatic positioning method of an embodiment of the present application, as shown in Fig. 3, the ROI automatic positioning method provided by the present application includes the following steps:

Step S11, obtaining original image data;

Step S12, preprocessing the original image data to obtain a coronal image sequence and a sagittal image sequence;

Step S13, locating the ROI in the images of the coronal view sequence and the sagittal view sequence;

Step S14, integrating the ROI positioned in the coronal image sequence to obtain a coronal ROI integration area, and integrating the ROI located in the sagittal image sequence to obtain a sagittal ROI integration area;

Step S15 , performing coordinate transformation on the coronal plane ROI integration area and the sagittal plane ROI integration area to obtain the three-dimensional coordinates of the target ROI in the original image data.

In step S11, the original image data is acquired.

The original image data in the embodiments of the present application is, for example, a three-dimensional medical image, and the computer device can obtain the medical image by performing three-dimensional reconstruction on the data of the patient's part to be examined collected by the scanning device. The medical image in this application is described by taking a computerized tomography (Computed Tomography, CT) image as an example.

In step S12, the original image data is preprocessed to obtain a coronal image sequence and a sagittal image sequence.

Specifically, the preprocessing of the original image data in this application includes standardizing the original image data, so as to convert the original image data (such as the 3D model of the whole lower limb) into coronal and sagittal image sequences under unified image rules . Fig. 4 is a flow chart of preprocessing the original image data according to an embodiment of the present application. As shown in Fig. 4, according to the embodiment of the present application, step S12 preprocesses the original image data to obtain the coronal image sequence and the coronal image sequence The sagittal image sequence includes the following steps S121 and S122.

In step S121, standardization processing is performed on the original image data to obtain standardized three-dimensional image data.

FIG. 5 is a flow chart of standardization processing of original image data according to an embodiment of the present application. As shown in FIG. 5 , step S121 performs standardization processing on the original image data to obtain standardized three-dimensional image data, including the following steps S1211 to S1213.

In step S1211, parameters of the original image data are obtained.

In an embodiment of the present application, the parameters of the original image data include at least one of shooting direction angle, resolution, origin coordinates and three-dimensional size of the original image.

In step S1212, the target image parameters and image transformation and interpolation algorithm are acquired.

In an embodiment of the present application, the target image parameters include at least one of target orientation angle, target resolution, target origin coordinates, and target three-dimensional size. The image transformation interpolation algorithm may be nearest neighbor method, bilinear interpolation method or cubic interpolation method, which is not limited in this application.

In step S1213, standardize the original image data according to the target image parameters and the image transformation and interpolation algorithm to obtain the standardized three-dimensional image data.

In an embodiment of the present application, the standardization process of image data can also be min-max standardization, and the original image data is linearly transformed, so that the result value is mapped to between 0-1, and the conversion formula is specifically:

Where max is the maximum signal value of the original image data, and min is the minimum signal value of the original image data. P _i is the signal value of the i-th point in the original image data, and P _i ^* is the signal value of the i-th point in the normalized image data.

In step S122, the coronal image sequence and the sagittal image sequence are obtained according to the standardized three-dimensional image data.

In an embodiment of the present application, a sequence of coronal images is obtained by extracting each 2D image section along the coronal direction in the 3D image data, wherein each 2D image section is regarded as a coronal image.

In an embodiment of the present application, a sequence of sagittal images is obtained by extracting each two-dimensional image slice along the sagittal plane direction in the three-dimensional image data, wherein each two-dimensional image slice is regarded as a sagittal image.

Through the above-mentioned preprocessing of the original image data, a coronal view sequence and a sagittal view sequence are obtained. In order to reduce the processing amount of image data and improve the speed of image data processing, it is necessary to filter out coronal images, sagittal images containing ROIs and coronal images and sagittal images without ROIs from the coronal image sequence and the sagittal image sequence. state diagram. For coronal and sagittal images without ROI, do not enter into the next step of data processing.

Fig. 6 is a flow chart of determining whether an ROI is included in a coronal image sequence and a sagittal image sequence according to an embodiment of the present application. The processing speed further includes steps S125 and S126 before step S13 locates the ROI of the images in the coronal view sequence and the sagittal view sequence.

In step S125, by classifying the images in the coronal image sequence and the sagittal image sequence, it is determined whether each image in the coronal image sequence and the sagittal image sequence contains an ROI.

Some of the images in the coronal image sequence and the sagittal image sequence contain ROIs, and some do not contain ROIs. By performing binary classification on the images in the coronal image sequence and the sagittal image sequence, it is possible to determine the Whether each image in the coronal image sequence and the sagittal image sequence contains ROI. Specifically, in an embodiment of the present application, the ROI pre-classification network is used to classify the images in the coronal image sequence and the sagittal image sequence, and whether the classification label output by the pre-classification network contains ROI, It is determined whether each image in the sequence of coronal images and the sequence of sagittal images contains a ROI. Fig. 7 is a flow chart of classifying coronal image sequences and sagittal image sequences according to an embodiment of the present application. As shown in Fig. 7, step S125 classifies images in the coronal image sequences and the sagittal image sequences , determining whether each image in the coronal image sequence and the sagittal image sequence contains ROI includes the following steps S1251-S1253.

In step S1251, feature extraction is performed on the images in the coronal image sequence and the sagittal image sequence respectively.

In an embodiment of the present application, a backbone network is used to perform feature extraction on images in the coronal image sequence and the sagittal image sequence respectively. The backbone network is any one of VGG series and Resnet series.

In step S1252, the extracted features are mapped to a binary classification space.

In an embodiment of the present application, a fully connected network is used to map the features extracted by the backbone network to a binary classification space. The classification result is 0 or 1, for example, 0 represents an image containing only the background, and 1 represents an image containing ROI.

In step S1253, according to the output classification result, it is determined whether the images in the coronal image sequence and the sagittal image sequence contain ROI.

In an embodiment of the present application, the determination is made according to the classification results of 0 and 1, for example, the coronal and sagittal images with a classification result of 1 are determined to include ROI images; the coronal and sagittal images with a classification result of 0 and Sagittal images, identified as images that do not contain ROIs.

In step S126, the images in the coronal image sequence and the sagittal image sequence that do not contain ROI are filtered.

The coronal image and the sagittal image containing the ROI are respectively composed of a coronal image sequence and a sagittal image sequence for further target detection of the ROI. By filtering images that do not contain ROIs in the coronal image sequence and the sagittal image sequence, the number of images to be processed can be reduced and the image processing speed can be improved.

Fig. 8 is a flow chart of image preprocessing in coronal image sequence and sagittal image sequence in an embodiment of the present application. Image requirements, before the coronal image sequence and the sagittal image sequence are input into the pre-classification network, the images in the coronal image sequence and the sagittal image sequence need to be preprocessed, that is, before step S125, further include the step S123 and S124.

In step S123, window width and window level processing is performed on the images in the coronal image sequence and the sagittal image sequence.

In an embodiment of the present application, the original image is, for example, a CT image, and the window width and level of the CT image in the coronal view sequence and the sagittal view sequence can be set, and based on the window width and level of the CT image, performing window width and window level processing on the CT images in the coronal image sequence and the sagittal image sequence, so as to enhance the ROI data of the CT images in the coronal image sequence and the sagittal image sequence.

Further, in order to meet the requirements of the pre-classification network for the size of the input image, in an embodiment of the present application, it also includes adjusting the size of the images in the coronal image sequence and the sagittal image sequence, for example, adjusting the image size to 1024× 512. There are two ways to adjust image size including edge cropping and padding.

In step S124, preprocessing is performed on the enhanced images in the coronal image sequence and the sagittal image sequence.

In an embodiment of the present application, after performing window width and window level processing on the images in the coronal image sequence and the sagittal image sequence, the enhanced coronal image can also be processed according to the requirements of the pre-classification network sequence and the sagittal image sequence were normalized.

Specifically, in order to further unify the distribution of data and accelerate network convergence, the enhanced coronal image sequence and the sagittal image sequence are normalized, and the normalization method can be a Z-score normalization method:

Where μ is the mean value of the image data in the enhanced coronal image sequence and the sagittal image sequence, and σ is the standard deviation of the image data in the enhanced coronal image sequence and the sagittal image sequence.

In step S13, the ROIs of the images in the coronal view sequence and the sagittal view sequence are located.

In an embodiment of the present application, the target detection network is used to locate the ROI of each coronal image in the coronal image sequence, and to locate the ROI of each sagittal image in the sagittal image sequence to obtain the desired target.

FIG. 9 is a flow chart of ROI positioning in an image according to an embodiment of the present application. As shown in FIG. 9, step S13 locates the ROI of the images in the coronal image sequence and the sagittal image sequence, including the following Step S131 and Step S132.

In step S131, feature extraction is performed on images in the coronal image sequence and the sagittal image sequence.

In an embodiment of the present application, a feature extraction network is used to perform feature extraction on the images in the coronal image sequence and the sagittal image sequence respectively. The feature extraction network can be Resnet50, Resnet101, HourglassNet or MobelNet. The feature extraction network uses the extracted features for target prediction.

In one embodiment of the present application, the convolution operation is performed on the images in the coronal image sequence and the sagittal image sequence through a convolutional neural network to realize feature extraction. The convolutional neural network uses a 3×3 filter, and the filter is scanned to the right and down in sequence, and the value of each element of the output matrix can be obtained to realize the filtering of the image to be processed. The calculation process can be as follows, for example:

Convolution process

Dotted frame: 4=1×1+1×0+1×1+0×0+1×1+0×1+0×1+0×0+1×1

Black line frame: 3=1×1+0×1+0×1+1×0+1×1+1×0+0×1+1×0+1×1

In an embodiment of the present application, in order to reduce the image size more effectively, speed up image processing and prevent overfitting, a pooling layer is added between adjacent convolutional layers in the convolutional neural network. The pooling layer, for example, can use a 2×2 filter to perform a maximum pooling operation on a 4×4 image, and the result takes the corresponding maximum value in the 2×2 window, and finally obtains a 2×2 image. The maximum pooling provides a way to down-sample the convolutional matrix for subsequent network layers to continue processing until the image features used to determine whether the image input to the target prediction network contains ROI are obtained.

In step S132, the position information of the ROI included in the images in the coronal image sequence and the sagittal image sequence is predicted according to the extracted features.

The position information of the ROI includes the center point coordinates and size information of the ROI. In an embodiment of the present application, taking the ROI as an example of a joint area in an image, after processing the features of the extracted image through a target detection network, the results shown in Figures 10a-10c can be predicted. The heat map shown in Fig. 10a shows the probability that each image block in the image contains a joint. Figure 10b shows the offset between the center point of the joint area and the actual center point in the heat map, and the vertical and horizontal coordinate values represent the offset angle and sum of the predicted center point of the joint area relative to the actual center point of the joint Offset length. Figure 10c shows the prediction of the length and width of the joint area, and using the predicted coordinate offset of the center point of the joint area to modify the predicted center point coordinates of the joint area, according to the size of the joint area predicted by the target detection network Information (such as length and width) and the corrected center point, the joint area in the image can be determined.

According to the above-mentioned embodiments of the present application, the ROI of each image in each coronal image sequence and sagittal image sequence is obtained through detection by the target detection network. In order to obtain the ROIs covering the dimension of the coronal view and the sagittal view of the three-dimensional model, it is necessary to integrate the detected ROIs of the images.

In step S14, the ROIs located in the coronal image sequence are integrated to obtain a coronal ROI integration area, and the ROIs located in the sagittal image sequence are integrated to obtain a sagittal ROI integration area.

In order to obtain the coronal plane ROI integration area and the sagittal plane ROI integration area, in an embodiment of the present application, based on the non-maximum suppression (Non-Maximum Suppression, NMS) algorithm, multiple overlapping ROIs are integrated to obtain the coronal The coronal ROI integration area of the ROI of the coronal image of the map sequence, and the sagittal ROI integration area of the ROI of the sagittal image of the overlay sagittal image sequence.

Fig. 11 is a flow chart of integrating multiple overlapping ROIs based on the non-maximum suppression algorithm in one embodiment of the present application. As shown in Fig. 11, step S14 integrates the ROIs positioned in the coronal image sequence to obtain the coronal ROI integration area , integrating the ROIs positioned in the sagittal image sequence to obtain a sagittal image ROI integration area, including the following steps S1401-S1403.

In step S1401, based on a non-maximum value suppression algorithm, the overlapping parts of the ROIs positioned in the coronal image sequence are integrated to obtain a target coronal ROI.

In step S1402, based on the non-maximum value suppression algorithm, the overlapping parts of the ROIs positioned in the sagittal image sequence are integrated to obtain a target sagittal ROI.

In step S1403, cluster processing is performed on the target coronal ROI and the target sagittal ROI to obtain the coronal ROI integration area and the sagittal ROI integration area.

Figure 12 shows the process of integrating multiple overlapping joint areas based on the non-maximum value suppression algorithm in an embodiment of the present application. As shown in Figure 12, the joint area with a non-maximum value of 0.78 and the joint area with a non-maximum value of 0.80 The joint area with a maximum value of 0.86 is suppressed, and the joint area corresponding to the maximum value of 0.92 is retained, thereby removing redundant joint areas and retaining the best joint area.

In an embodiment of the present application, a clustering algorithm may also be used to eliminate redundant joint regions and retain the best joint regions.

In an embodiment of the present application, since there is a small probability of false detection during the prediction period, after performing non-maximum value suppression, there may be some boxes that deviate from the center of the cluster, and it is necessary to further obtain the coronal plane and vector The ROI position of the surface. Specifically include:

According to the k-means clustering algorithm, the target coronal plane ROI is clustered to obtain the coronal plane ROI integration area; according to the k-means clustering algorithm, the target sagittal plane ROI is clustered to obtain the target coronal plane ROI. Said sagittal plane ROI integration area.

Wherein, according to the k-means clustering algorithm, the target coronal ROI is clustered, and the step of obtaining the coronal ROI integration area includes: selecting a plurality of the target coronal ROIs as cluster centers; The intersection and union comparison between the cluster center and the other target coronal planes is used to cluster the target coronal plane ROIs to obtain the coronal plane ROI integrated regions. According to the k-means clustering algorithm, the target sagittal plane ROI is clustered, and the step of obtaining the sagittal plane ROI integration area includes: selecting a plurality of the target sagittal plane ROIs as cluster centers; The intersection and union comparison between the cluster center and other target sagittal planes is used to cluster the target sagittal plane ROI to obtain the coronal plane ROI integration area.

In an embodiment of the present application, as shown in FIG. 13 , step S14 uses the NMS algorithm combined with the clustering algorithm to integrate multiple overlapping ROIs, and step S14 integrates the ROIs positioned in the coronal image sequence to obtain the coronal plane The ROI integration area, integrating the ROIs positioned in the sagittal image sequence to obtain the sagittal plane ROI integration area specifically includes the following steps S1411-S1418.

In step S1411, multiple overlapping ROIs of each ROI are integrated based on a non-maximum value algorithm.

In step S1412, the initial value of the number n of cluster centers is set to 0.

In step S1413, a cluster center corresponding to the current ROI is selected.

In step S1414, the number n of cluster centers is accumulated by 1, and an intersection over union (IOU) between the current ROI and the current cluster center is calculated.

In step S1415, it is judged whether the IOU is greater than a given threshold.

In step S1416, if the IOU is greater than a given threshold, the current ROI is classified into the current cluster center.

In step S1417, if the IOU is less than or equal to a given threshold, it is judged whether the number n of the calculated cluster centers is greater than or equal to k, where k is the total number of cluster centers corresponding to each type of ROI set ; If not, reselect a cluster center from unselected cluster centers as the current cluster center, and return to step S1414.

In step S1418, it is judged whether all ROIs have been traversed, if not, then select an ROI from unclustered ROIs as the current ROI, and return to execute step S1412; if yes, end the integration of the ROI, The coronal plane ROI integration area and the sagittal plane ROI integration area are obtained.

In an embodiment of the present application, the step S1413 selects a cluster center corresponding to the ROI, specifically, randomly selects a cluster center corresponding to the ROI.

In an embodiment of the present application, the re-selecting a cluster center from unselected cluster centers may be re-randomly selecting a cluster center from unselected cluster centers.

As shown in FIG. 14 , for each type of joint, the above-mentioned steps S1411 - S1418 are performed respectively, so as to realize the integration of multiple overlapping joint areas of all joints. Through the NMS algorithm combined with the clustering algorithm to integrate multiple joint areas of the joint, even after the non-maximum value suppression is performed, there may be some frames that deviate from the center of the cluster, through the joint area integration method of this application, it can also be obtained Accurate coronal and sagittal joint positions.

In step S15, coordinate transformation is performed on the coronal image region of the ROI and the sagittal image region of the ROI to obtain the three-dimensional coordinates of the target ROI in the original image data.

In an embodiment of the present application, as shown in FIG. 15 , the three-dimensional coordinates of the ROI are calculated according to the obtained coordinates of the ROI on the coronal plane and the sagittal plane. Assume that P _ct (X,Y,Z) represents the three-dimensional coordinates of the ROI in the original CT image, P _i ( _xi ,y _i ) represents the coordinates of the ROI on the i-th coronal plane, and P _j (x _j ,y _j ) represents the coordinates of the ROI on the jth sagittal plane, then the three-dimensional coordinates of P _i ( _xi ,y _i ) transformed into the original CT image are CT( _xi ,Y,y _i ), P _j (x _j ,y _j ) The three-dimensional coordinates transformed into the original CT image are CT(X, x _j , y _j ), according to the above formula, the three-dimensional coordinates of the ROI under the original CT image can be deduced as:

P _ct (X,Y,Z)=P _ct ( _xi ,x _j ,(y _i +y _j )/2).

Due to errors in the calculation process, y _i and y _j will not be completely equal, so the coordinates in the Z direction are replaced by the mean value of y _i and y _j .

The ROI automatic positioning method of the present application does not need to perform calculations on three-dimensional data, reduces computational complexity, reduces data processing time, can realize automatic positioning of ROI parts in three-dimensional space, and detects differences in medical images at one time. The ROI position of the navigation software simplifies the operation process of the navigation software, improves the versatility of the navigation software, improves work efficiency, and reduces the workload of the navigation software operators.

It should be understood that although the various steps in the flow charts in the drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in each flow chart may include multiple sub-steps or multiple stages, these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, the execution of these sub-steps or stages The order is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

The present application provides an automatic ROI positioning device. As shown in FIG.

The data acquisition module 101 is configured to acquire original image data.

The data preprocessing module 102 performs preprocessing on the original image data to obtain a coronal image sequence and a sagittal image sequence.

The positioning module 103 locates the ROI of the images in the coronal view sequence and the sagittal view sequence.

The integration module 104 integrates the ROIs located in the coronal image sequence to obtain a coronal ROI integration area, and integrates the ROIs located in the sagittal image sequence to obtain a sagittal ROI integration area.

The coordinate transformation module 105 performs coordinate transformation on the coronal plane ROI integration area and the sagittal plane ROI integration area to obtain the three-dimensional coordinates of the target ROI in the original image data.

This application provides an ROI automatic positioning device, which does not need to perform calculations on three-dimensional data, reduces computational complexity, reduces data processing time, can realize automatic positioning of ROI parts in three-dimensional space, and detects medical images at one time The different ROI parts on the map simplifies the operation process of the navigation software, improves the versatility of the navigation software, improves the work efficiency, and reduces the workload of the navigation software operators.

Different embodiments of the ROI automatic positioning device of the present application can be obtained by referring to the above descriptions of the various embodiments of the ROI automatic positioning method, and will not be repeated here. Each module in the above-mentioned ROI automatic positioning device can be fully or partially realized by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of the above-mentioned modules. It should be noted that the division of modules in the embodiment of the present application is schematic, and is only a logical function division, and there may be other division methods in actual implementation.

The present application provides a surgical robot system. The robot system is configured to execute the ROI automatic positioning method in each of the above embodiments. In the process of realizing the automatic positioning of the ROI of the human body, the robot system does not need to perform calculations on the three-dimensional data, which reduces the computational complexity and the time of data processing, and can realize the automatic positioning of the ROI in the three-dimensional space. The different ROI parts on the medical image can be detected accurately, which simplifies the operation process of the navigation software, improves the versatility of the navigation software, improves the work efficiency, and reduces the workload of the navigation software operators.

The present application provides a computer device, including a memory and a processor, the memory stores a computer program, and the processor implements the ROI automatic positioning method in any embodiment of the present application when executing the computer program.

The present application provides a computer-readable storage medium on which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the ROI automatic positioning method of any embodiment of the present application.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above-mentioned embodiments can be completed by instructing related hardware through computer programs, and the computer programs can be stored in a non-volatile computer-readable memory In the medium, when the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include non-volatile and/or volatile memory. Nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The various technical features of the above embodiments can be properly combined according to the actual situation. To make the description concise, all possible combinations of the various technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features , should be considered as within the scope of this specification.

The above-mentioned embodiments only represent several implementation modes of the present application, and the description thereof is relatively specific and detailed, but it should not be construed as limiting the scope of the patent for the invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the scope of protection of the patent application should be based on the appended claims.

Claims

A ROI (region of interest, region of interest) automatic positioning method, comprising:

Get raw image data;

Preprocessing the original image data to obtain a coronal image sequence and a sagittal image sequence;

Locating ROIs in the images of the coronal sequence and the sagittal sequence;

Integrating the ROIs positioned in the coronal image sequence to obtain a coronal ROI integration area, and integrating the ROIs positioned in the sagittal image sequence to obtain a sagittal ROI integration area;

Coordinate transformation is performed on the coronal plane ROI integration area and the sagittal plane ROI integration area to obtain the three-dimensional coordinates of the target ROI in the original image data.
The ROI automatic positioning method according to claim 1, wherein, before positioning the ROI in the images of the coronal view sequence and the sagittal view sequence, further comprising:

determining whether each image in the coronal image sequence and the sagittal image sequence contains an ROI by classifying the images in the coronal image sequence and the sagittal image sequence;

and filtering images that do not contain the ROI in the coronal image sequence and the sagittal image sequence.
The ROI automatic positioning method according to claim 1, wherein said preprocessing the original image data to obtain a coronal image sequence and a sagittal image sequence comprises:

performing standardized processing on the original image data to obtain standardized three-dimensional image data;

According to the standardized three-dimensional image data, the coronal image sequence and the sagittal image sequence are obtained.
The ROI automatic positioning method according to claim 3, wherein said performing standardization processing on said original image data to obtain standardized three-dimensional image data comprises:

Obtain parameters of the original image data;

Obtain target image parameters and image transformation interpolation algorithm;

Standardize the original image data according to the target image parameters and the image transformation and interpolation algorithm to obtain the standardized three-dimensional image data.
ROI automatic positioning method according to claim 4, wherein:

The parameters of the original image data include at least one of the shooting direction angle, resolution, origin coordinates and three-dimensional size of the original image;

The target image parameters include at least one of target shooting direction angle, target resolution, target origin coordinates and target three-dimensional size.
The ROI automatic positioning method according to claim 2, wherein, before said classifying said coronal image sequence and said sagittal image sequence, it also includes:

performing window width and window level processing on the images in the coronal image sequence and the sagittal image sequence.
The ROI automatic positioning method according to claim 2, wherein, by classifying the images in the coronal image sequence and the sagittal image sequence, determining the coronal image sequence and the sagittal image sequence Before each image contains the ROI, further include:

Perform normalization processing on the images in the coronal image sequence and the sagittal image sequence.
The ROI automatic positioning method according to claim 1, wherein said positioning the ROI in the images of the coronal view sequence and the sagittal view sequence comprises:

performing feature extraction on the images in the coronal sequence and the sagittal sequence;

Predicting position information of the ROI included in the images in the coronal image sequence and the sagittal image sequence according to the extracted features.
The ROI automatic positioning method according to claim 8, wherein the position information includes the center point coordinates and size information of the ROI.
The ROI automatic positioning method according to claim 1, wherein said integrating the ROI positioned in the coronal image sequence obtains a coronal ROI integration area, and integrates the ROI positioned in the sagittal image sequence to obtain a sagittal plane ROI integration areas, including:

Based on a non-maximum suppression algorithm, integrating the overlapping parts of the ROI positioned in the coronal image sequence to obtain a target coronal ROI;

Based on a non-maximum value suppression algorithm, integrating the overlapping parts of the ROI positioned in the sagittal image sequence to obtain a target sagittal ROI;

The target coronal plane ROI and the target sagittal plane ROI are respectively clustered to obtain the coronal plane ROI integration area and the sagittal plane ROI integration area.
The ROI automatic positioning method according to claim 10, wherein said target coronal plane ROI and said target sagittal plane ROI are respectively clustered to obtain said coronal plane ROI integration area and said sagittal plane ROI Surface ROI integration area, including:

Carry out cluster processing to described target coronal plane ROI according to k-means clustering algorithm, obtain described coronal plane ROI integration region;

The target sagittal plane ROI is clustered according to the k-means clustering algorithm to obtain the sagittal plane ROI integration area.
The ROI automatic positioning method according to claim 11, wherein said target coronal plane ROI is clustered according to the k-means clustering algorithm to obtain the coronal plane ROI integration area, comprising:

Selecting multiple target coronal plane ROIs as cluster centers;

Perform clustering processing on the target coronal ROI according to the intersection and union ratio between each cluster center and other target coronal planes, to obtain the coronal plane ROI integration area;

According to the k-means clustering algorithm, the target sagittal plane ROI is clustered to obtain the sagittal plane ROI integration area, including:

Selecting multiple target sagittal plane ROIs as cluster centers;

The target sagittal plane ROIs are clustered according to the intersection ratio between each cluster center and other target sagittal planes to obtain the coronal plane ROI integration area.
The ROI automatic positioning method according to claim 1, wherein the coordinate transformation is performed on the coronal plane ROI integration area and the sagittal plane ROI integration area to obtain the three-dimensional coordinates of the target ROI in the original image data, include:

According to the formula P ct (X, Y, Z)=P ct (x i , x j , (y i +y j )/2), the coronal plane ROI integration area and the sagittal plane ROI integration area are performed Coordinate transformation to obtain the three-dimensional coordinates of the target ROI;

Wherein, P ct (X, Y, Z) represents the three-dimensional coordinates of the point in the target ROI in the original image data, P i (xi , y i ) represents the point in the ith coronal Coordinates of the plane ROI integration area, P j (x j , y j ) represents the coordinates of the point in the jth sagittal plane ROI integration area.
A ROI automatic positioning device, comprising:

A data acquisition module, configured to acquire raw image data;

A data preprocessing module, configured to preprocess the original image data to obtain a coronal image sequence and a sagittal image sequence;

A positioning module, configured to position the ROI of the images in the coronal view sequence and the sagittal view sequence;

An integration module, configured to integrate the ROIs positioned in the coronal image sequence to obtain a coronal ROI integration area, and integrate the ROIs positioned in the sagittal image sequence to obtain a sagittal ROI integration area;

A coordinate transformation module, configured to perform coordinate transformation on the coronal plane ROI integration area and the sagittal plane ROI integration area to obtain the three-dimensional coordinates of the target ROI in the original image data.
A surgical robot system configured to perform the method of any one of claims 1-13.
A computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 13 when executing the computer program.
A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by one or more processors, cause the one or more processors to perform claims 1 to 13 any one of the methods described.