CN116153469A

CN116153469A - Interactive medical image segmentation platform

Info

Publication number: CN116153469A
Application number: CN202111329369.3A
Authority: CN
Inventors: 金博; 李志谦; 王祥丰; 李郁欣; 胡斌
Original assignee: East China Normal University; Huashan Hospital of Fudan University
Current assignee: East China Normal University; Huashan Hospital of Fudan University
Priority date: 2021-11-10
Filing date: 2021-11-10
Publication date: 2023-05-23

Abstract

The embodiment discloses an interactive medical image segmentation platform. The platform medical image display subsystem and the interactive segmentation subsystem; wherein: the medical image display subsystem comprises a display module, a display module and a display module, wherein the display module is used for receiving and displaying medical images and medical information of the medical images; the interactive segmentation subsystem comprises an operation instruction response module, and is used for responding to an operation instruction of a user and feeding back the operation instruction to the medical image display subsystem for display adjustment; the medical image segmentation method further comprises an algorithm segmentation module, a segmentation module and a segmentation module, wherein the algorithm segmentation module is used for determining an initial segmentation result of the medical image based on a segmentation algorithm; and the algorithm interaction module is used for adjusting the initial segmentation result according to the error correction interaction information if the error correction interaction information is detected, so as to obtain an image segmentation result. By adopting the technical scheme of the embodiment of the invention, a doctor can be liberated from manual labeling by using an interactive segmentation algorithm, and the segmentation result can be obtained quickly only by correcting the segmentation result of the deep learning algorithm model.

Description

Interactive medical image segmentation platform

Technical Field

The embodiment of the application relates to the technical field of medical image processing, in particular to an interactive medical image segmentation platform.

Background

With the rapid development of artificial intelligence, artificial intelligence+medical treatment has become one of the important scenes in which artificial intelligence falls to the ground. Among them, computer-aided diagnosis and treatment is the most common way to combine medical and intelligent technologies. In the computer aided diagnosis and treatment process, the segmentation of the focus area of the medical image is one step which is critical.

In the prior art, segmentation of images is usually realized in a mode based on a threshold value and a graph theory, but the mode is usually shallow, and has no guiding significance to clinic. With the high-speed development of deep learning, dynamic segmentation of medical images is developed to full-automatic segmentation, and segmentation based on a neural network has become an important mode, so that the accuracy of segmentation of medical images is improved, and a certain guiding effect is provided for clinic. In general, a segmentation method based on deep learning requires a large amount of manual labeling information as training data, and if the data is insufficient, the model is over-fitted, so that the robustness of the model is reduced. Manual labeling is very specialized for the annotators and is time and effort consuming for labeling of some 3D medical images.

Disclosure of Invention

The embodiment of the invention provides an interactive medical image segmentation platform, which can be used for freeing doctors from purely manual labeling by utilizing an interactive segmentation algorithm, and the doctors can quickly obtain a final segmentation result by only carrying out a small amount of error correction on the segmentation result of a deep learning algorithm model.

The embodiment of the invention provides an interactive medical image segmentation platform, which comprises: a medical image display subsystem and an interactive segmentation subsystem; wherein:

the medical image display subsystem is connected with the interactive segmentation subsystem;

the medical image display subsystem comprises a display module, a display module and a display module, wherein the display module is used for receiving and displaying medical images and medical information of the medical images;

the interactive segmentation subsystem comprises an operation instruction response module, and is used for responding to an operation instruction of a user, feeding back the operation instruction to the medical image display subsystem and enabling the medical image display subsystem to display and adjust according to the operation instruction;

the interactive segmentation subsystem further comprises an algorithm segmentation module, which is used for determining an initial segmentation result of the medical image based on a segmentation algorithm;

the interactive segmentation subsystem further comprises an algorithm interaction module, wherein the algorithm interaction module is used for adjusting the initial segmentation result according to the error correction interaction information if the error correction interaction information is detected, so that an image segmentation result is obtained.

Further, the interactive segmentation subsystem further comprises an algorithm offline training module;

the algorithm off-line training module is used for combining the original training set stored in the file system and error correction interaction information provided by the interaction segmentation of the user in the preset time stored in the database to update and train the segmentation algorithm.

Further, the interactive segmentation subsystem further comprises an image marking module;

the image marking module is used for providing at least one marking mode of ruler function, dotting, drawing of rectangle and free painting brush for users to mark images.

Furthermore, the image mark adopts data conversion to obtain serializable mark data so as to construct error correction interaction information.

Further, the medical image display subsystem further comprises an operation module;

the operation module is used for switching medical image slices, adjusting window level and window width, and performing at least one operation of enlarging, reducing and translating the medical images.

Further, the medical image display subsystem further comprises a three-view module;

the three-view module is used for controlling at least one of three-view display, three-view control and three-view linkage.

Further, the medical image display subsystem further comprises a multi-window module;

the multi-window module is used for carrying out multi-window layout on at least one medical image and controlling multi-window display or hiding.

Further, the medical image display subsystem is realized by using a reaction front-end framework based on a B/S architecture.

Further, the data transmission of the interactive segmentation subsystem is realized based on a flash interface.

Further, the medical image display subsystem and the interactive segmentation subsystem are realized by adopting containerized deployment.

The embodiment of the invention discloses an interactive medical image segmentation platform. The doctor can be liberated from purely manual labeling, and the problem of insufficient robustness and precision of the full-automatic algorithm segmentation result is solved. The doctor does not need to label the image pixel by pixel, only provides corresponding error correction information on the basis of algorithm segmentation, and can obtain a final segmentation result through a few rounds of simple interaction.

Drawings

Fig. 1 is a schematic structural diagram of an interactive medical image segmentation platform according to a first embodiment of the present invention;

FIG. 2 is a schematic diagram of a medical image display subsystem according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of an interactive segmentation subsystem according to a first embodiment of the present invention;

FIG. 4 is a schematic flow chart of interactive segmentation of medical images according to an embodiment of the present invention;

fig. 5 is an access schematic diagram of a medical image interaction segmentation platform according to an embodiment of the present invention.

Detailed Description

The present application is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present application are shown in the drawings.

Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart depicts steps as a sequential process, many of the steps may be implemented in parallel, concurrently, or with other steps. Furthermore, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figures. The processes may correspond to methods, functions, procedures, subroutines, and the like.

Example 1

Fig. 1 is a schematic structural diagram of an interactive medical image segmentation platform according to an embodiment of the present invention. The technical scheme of the embodiment of the invention can be applied to the scenes of medical image segmentation.

As shown in fig. 1, the interactive medical image segmentation platform provided in the embodiment of the present invention includes: a medical image presentation subsystem 110 and an interactive segmentation subsystem 120; wherein:

the medical image display subsystem 110 is connected with the interactive segmentation subsystem 120;

the medical image display subsystem 110 includes a display module 111 for receiving and displaying medical images, and medical information of the medical images;

the interactive segmentation subsystem 120 comprises an operation instruction response module 121, which is used for responding to an operation instruction of a user and feeding back the operation instruction to the medical image display subsystem 110, so that the medical image display subsystem 110 can perform display adjustment according to the operation instruction;

the interactive segmentation subsystem 120 further comprises an algorithm segmentation module 122 for determining an initial segmentation result of the medical image based on a segmentation algorithm;

the interactive segmentation subsystem 120 further includes an algorithm interaction module 123, configured to adjust the initial segmentation result according to the error correction interaction information if the error correction interaction information is detected, so as to obtain an image segmentation result.

In this scenario, the processing of medical images is mainly performed, wherein the medical images (Digital Imaging and Communications in Medicine, DICOM) are software integration standards for medical imaging. DICOM files are binary files stored in a format on a computer hard disk, from which information cannot be obtained directly by the human eye. In order to facilitate the viewing by professionals, such as doctors and researchers, the medical image display subsystem 110 needs to provide for the extraction and display of the image information in the DICOM file. In order to facilitate the professional's view of each detail on the medical image, the present invention provides the functions of zooming in, zooming out, panning, and adjusting the window level and width of the image. Medical images are typically 3D maps made up of multiple 2D images, and have the characteristics of multiple modalities. The present invention provides slice switching, multi-window presentation, and three-view presentation functions. The main implementation comprises the following contents:

in connection with the first aspect, embodiments of the present invention provide two ways of loading DICOM files: uploading local files and remotely loading the files through the URL. In local file uploading, the scheme can utilize the < input > tag of the browser, and the file can be selected to be uploaded by setting the attribute 'type' of the input tag as 'file'. Because this system only supports displaying DICOM files, setting the "accept" attribute to ". Dcm" can be uploaded with other types of files. As described above, generally, a patient's photographed medical image is composed of a plurality of DICOM files, so that it is necessary to support uploading of a plurality of images, and setting "multiple" on the < input > tag may allow a user to upload a plurality of files. And when the user selects the remote import data, a mode frame is popped up, and the user is allowed to fill in the URL address of the file to be imported. Uploading the loading file locally is an asynchronous task, and the task to be executed after loading is completed needs to be placed in the callback function. In the callback function, the uploaded data can be obtained from the end object, then the data of the scheme is required to be submitted to the Dicom Parser for analysis, and the analyzed data is stored in the Dicom instance. The remote loading file firstly needs to provide a loading address by a user, and after the address is acquired, the loading module circularly sends an Ajax request to remotely acquire data. Each request requires a callback function in which an event object is obtained, from which the return code of the request can be obtained, the successful return code of the request being 200. If the request fails, the scheme returns error information and prompts the user at the front end. And if the request is successful, resolving the data by using the DicomParse, and storing the resolved data into the global.

In this embodiment, optionally, the medical image display subsystem is implemented using a reaction front-end framework based on a B/S architecture.

Wherein, based on the B/S framework, the medical image display subsystem is realized by utilizing a React front-end framework. The method mainly provides a completely interactive segmentation environment for doctors, provides the functions of loading DICOM files and displaying images and information of the DICOM files, and can amplify, shrink, translate and adjust window width of the images, display images in multiple windows, display three views and the like after data are loaded so as to facilitate the doctors to observe medical images.

In this embodiment, optionally, data transmission of the interactive segmentation subsystem is implemented based on a flash interface.

The interactive segmentation subsystem is mainly divided into two parts: web front end and background services. The Web front end mainly provides an image annotation tool and an interaction function with a background segmentation algorithm on the basis of a medical image display subsystem. Background services are mainly based on a flash framework, and the main functions of the background services are as follows: providing service for storing and acquiring DICOM files; providing an interface for front-end call; calling an interactive segmentation algorithm; collecting interactive error correction data; periodic training of the algorithm model. The system realizes the deployment of Docker containerization, shields the environment problem and solves the problem of difficult deployment of different systems. This constitutes a one-stop system integrating the functions of Dicom viewing, interactive segmentation, data collection, model training, etc.

Fig. 2 is a schematic structural diagram of a medical image display subsystem according to an embodiment of the present invention. As shown in fig. 2, the medical image presentation subsystem has the following functional modules:

the medical image display subsystem consists of five modules, namely a data loading module, a display module, an operation module, a three-view module and a multi-window module. In the data loading module, the data loading module is mainly responsible for loading local data or remote data into a memory and analyzing the loaded dicom file. The display module is mainly responsible for displaying the parsed data at the position of the response, wherein the parsed data mainly comprises images and image information in the dicom file. In the operation module, mainly operations on images are provided: the method comprises the steps of slice switching, image window level and window width adjustment, image amplification, image reduction and image translation. The three-view module is mainly responsible for displaying front, top, side views of the 3D dicom image, and providing control over and controlling the linkage between the three views.

In this scheme, specifically, the image display function provided by the embodiment of the invention mainly uses the new tag < canvas > of HTML5 to draw the required image. The < canvas > tag may draw the picture directly, but since the image of the solution is a binary number parsed from the DICOM file, the ImageData object provided by the < canvas > is employed. The ImageData object can be transmitted into a one-dimensional array corresponding to the height and width of the picture, and the return value is 4 x high x wide, and each pixel point consists of four numbers which respectively represent color spaces of Red (Red), green (Green), blue (Blue) and Alpha. The solution then takes the image binary stream of the desired drawing from the DICOM instance, converts the binary into RGBA and fills it onto the ImageData object. And finally, drawing the constructed ImageData object on the canvas. After the picture is displayed, meta information corresponding to the DICOM file needs to be displayed. The scheme displays meta information by adding a transparent lamination layer on a canvas drawing board, and the data elements of the meta information are derived from a DICOM example. Since the meta information is displayed or not, which can be decided by the user, it is also necessary to add a control button to the user's upper operation field of the page. A state named display is newly added on the page component to control the style of the button for controlling the meta information, and the render life cycle updating view is executed according to the state change by the action, so that the meta information is controlled to be displayed and hidden.

In this aspect, optionally, the medical image display subsystem 110 further includes an operation module;

The embodiment of the invention provides a slice switching function. Slice switching is mainly achieved in that the index of the slice increases when the user mouse wheel is slid up and the slice index decreases when the wheel is slid down. Firstly, registering a global mouse rolling monitoring event in an image window, and then, reacting to be made when the mouse is rolled, namely, realizing a callback event of the mouse rolling. The browser transmits an event object to a callback function, the scheme can judge whether the mouse is scrolled upwards or downwards through deltaY in the event object in the callback function, and when deltaY <0 indicates to scroll upwards, the scheme indicates to scroll downwards. Then, the corresponding effect is performed according to deltaY, and if deltaY <0, the slice index is decreased, whereas the index is increased. During the slice index increase and decrease, the system will obtain the corresponding indexed slice from the raw data of the DICOM instance and update it into the DICOM instance rendering data. The image is then redrawn and the user observes the change.

The embodiment of the invention provides an image adjusting function. The main realization of adjusting the image mode is that the picture is enlarged when the user slides the roller upwards, when the user moves towardsThe image is reduced when the scroll wheel is slid down, and moves with the mouse when the user presses the left key and moves. The proposal registers the global monitoring event of mouse movement and mouse rolling in the image window, and the callback function after the event occurs is needed to be realized. Similar to the slice switching mode in the implementation of enlargement and reduction, the scheme can judge the direction and the rolling force of the roller through the wheelDelta on the event object, so that the scheme is determined to be the magnification scale of enlargement and reduction, and the scheme can obtain the magnification. At the same time obtain the position (x) _center ,y _center ) As the center point for image magnification. A new center point (x _origin ,y _origin )。

In the implementation of movement, when the left button of the mouse is pressed, the start of the talk flag bit is set to true to indicate that movement is started, and the initial position x of the mouse is remembered when the mouse is clicked ₀ And y ₀ . During the movement of the mouse, the current position of the mouse can be obtained through the event object and is marked as x _s And y _s . Obtaining the offset x of the picture _t ＝x _s -x ₀ ，y _t ＝y _s -y ₀ . Finally, the magnification scale and the image center point (x _origin ，y _origin ) Offset (x) _t ，y _t ) And finally, transmitting the parameters into a canvas provided setTransform interface to convert the image.

The embodiment of the invention provides a window level and window width adjusting function. Similar to the implementation of image movement in image conversion mode, the start of the marker bit is set to true when the left button of the mouse is pressed, and the initial position x of the mouse is remembered when the mouse is clicked ₀ And y ₀ . During the movement of the mouse, the current position of the mouse can be obtained through the event object and is marked as x _s And y _s . The window width w can be obtained by the following formula _width Sum window level w _center . According to the window levelThe window width value, the scheme can use DicomParser to convert to obtain a new image so as to update the view.

d _x ＝x _s -x ₀ ；

d _y ＝y _s -y ₀ ；

w _width ＝w _width +d _y ；

w _center ＝w _center +d _x ；

The view assembly mainly consists of two large modules. The first is the display module described above, whose main function is to convert binary data into macroscopic picture data for display on a screen. The second module is a translation rotation module, the translation rotation module mainly provides an operation interface for a user, and the user can influence the display of other two views by operating a certain view component so as to achieve the effect of observing images from more angles.

The main implementation mode of the translation rotation module is that the image is displayed<canvas>A transparent stack is added to the label, on which two mutually perpendicular axes are drawn, representing the projections of the other two views onto the present view, respectively. The user interaction is mainly the translation and rotation of the two axes, which is also achieved by pressing the mouse main key to move the mouse. First a listening event for a left mouse button press is added to the view component, after which the mouse movement affects the axis on the stack. The left mouse button is released and movement of the mouse cannot affect the axis on the stack. The on-axis region is divided into a moving and rotating region. The movement function calculates the distance of the mouse movement mainly according to the event object on the mouse event, thereby using the transfer x of CSS _diff ,y _diff (x _diff ,y _diff Representing the distance the mouse moves) to move the stack, ensuring that the intersection of the two axes coincides with the mouse position. The rotation function obtains the current position of the mouse according to the event object of the mouse event, knows the left side of the intersection point, determines a straight line according to the intersection point and the point where the mouse is positioned, and calculates the corresponding rotation according to the slope of the straight lineTurning angle, finally using CSS transfer: rotate (x deg) for rotation.

The other two view components are updated after the translational rotation of the shaft on the view component. The principle of this is mainly: and wrapping the three view components in a parent container component, storing the displayed data on states by the parent component, respectively transmitting the data to the view components through tips attributes, calling a render life cycle when the states of the parent component are changed, and re-rendering the view components if the data to be displayed by the view components are changed. It is desirable to have a module that updates the data in the parent component after the translational rotation operation of the view component, which is a three-view linkage module. The three-view linkage module mainly receives the rotation angle transmitted by the view component and the intersection point of the two shafts, and can calculate the linear equation of the two shafts in a point inclined mode according to the rotation angle and the intersection point. The two axes correspond to the projections of the other two views on the view respectively, a corresponding projection plane can be obtained according to an equation of a projection straight line, pixel information is obtained from the plane, then data of a father component is updated, and finally re-rendering of the view component is triggered.

In this embodiment, optionally, the medical image display subsystem further includes a three-view module;

The embodiment of the invention provides a three-view function. What is essentially accomplished is to enable viewing of a three-dimensional DICOM file from the front, top, and side. That is, on one interface, the user can view the same DICOM file from three perspectives, which are all the same in display and function. According to the idea of exact componentization, a view component can be abstracted, a three-view module is composed of three view components, and the components can communicate with each other through a parent component.

In this embodiment, optionally, the medical image display subsystem further includes a multi-window module;

The embodiment of the invention provides a multi-window function. The loading and displaying of DICOM files, zooming in and out of the image, switching of slices, and adjusting of window width of the image have been achieved in the previous section, which are achieved in a single window. Based on the idea of exact componentization, these interfaces and the corresponding functions can be put together to form a single component, each such component having the same function. In the structure life cycle of the reaction, the component is initialized, the execution of the data loading module is mainly completed, and the display module is executed in the render life cycle after the execution of the data loading module. Later, when the image changes due to user interaction, the exact framework will call the shouldbcomponent update lifecycle to determine whether the page needs to be re-rendered, and if so, call the render lifecycle.

The construction of the window assembly is completed, and the window assembly is required to be displayed according to the requirements of users. Firstly, a selection function of gridding layout is required to be provided for a user, and the user can select a style needing layout according to own requirements. The user clicks the layout button to display the layout option board, when the mouse slides on the option board, the event of the mouse is monitored, the scratched area is marked as highlight, and the highlight area represents the layout form wanted by the user. After the user finishes clicking, the option board disappears firstly, then the number of rows and the number of columns selected by the user are returned, and finally a window instance corresponding to the number of rows and the number of columns is generated in the page. Because each window is mutually independent, the operation on the image can only be carried out on a specific window, and therefore, the scheme realizes the function of providing window selection for users. When the user clicks a window with the left mouse button, the frame of the window is turned red by using the CSS, and the user is prompted to currently focus on the window. The implementation of the operation module can be seen that the system defines a global mouse monitoring event. When a user focuses on a certain window, a callback function of a corresponding mouse monitoring event is pointed to an instance method of the focusing window, so that the function of operating the focusing window picture is achieved. When the window is operated, a certain window is sometimes required to be enlarged, so that the whole window is fully distributed on the whole screen, and the system realizes the function from a multi-window mode to a single-window mode. Double clicking the left mouse button is also a relatively conventional operation in use by the user, so double clicking the left mouse button will be the key for the window mode change operation of the present solution. According to the scheme, a mouse double-click event is registered on each window object, and when the double-click event occurs, the browser API, namely a document. To distinguish whether the current window is in multi-window mode or single window mode, the present scheme maintains a single layout variable. Currently, if in the multi-window mode, the CSS attribute "display" of these nodes is labeled "none", so that other windows are hidden, otherwise "display" is labeled "flex".

Fig. 3 is a schematic structural diagram of an interactive segmentation subsystem according to a first embodiment of the present invention. As shown in fig. 3, the interactive segmentation subsystem mainly comprises two parts, namely a server background and a Web front end. The Web front end mainly provides functions of file uploading, image marking and algorithm interaction on the basis of the medical image display subsystem, an nginx server controls access to a front end page, and finally the Web front end and the nginx server are packaged into a dock mirror image. The server background mainly adopts the flush framework of python. The server background mainly provides persistence of files and persistence of marked data for the system; re-presuming the doctor marked data and returning the segmentation result to the doctor; regular retraining of the algorithm; and packaging the realization of the server background into a docker mirror image. Finally, the Web front-end service runs on a server of the production environment as a docker container instance with the server background.

The embodiment of the invention provides a file uploading function. File uploads are required to be made to a specific user because it is necessary to record in the database which user has stored which files. Therefore, before uploading the file, the user is required to log in before the operation can be performed. When the user enters a webpage to upload a file, the browser detects that the user does not log in and jumps to a log-in page. The user fills in account numbers and passwords on the login page, and the browser sends a request to the server side to require verification. To ensure security, the password in the database is encrypted using SHA 256. When the server verifies the account number and the password, firstly, the stored password of the database is obtained according to the account number, then the password transmitted by the browser is subjected to SHA256 encryption, and then the two are compared, and if the two are the same, the password matching is proved to be successful. After the password is successfully matched, the server encrypts information (such as account id, user name and user role) of the account by using the PyJWT, generates an accessToken as a Cookie and returns the Cookie to the browser. Because of the Cookie mechanism, the next time the request is sent, the server will automatically access token, and the server will decrypt the access token to obtain the account information. After log-in authentication, the user may enter the system. The user enters the system and clicks the upload button, and the attribute "type" of the input label is set to be "file" by utilizing the < input > label of the browser, so that the upload file can be selected. After the user selects the file, the browser sends the file and the accessToken obtained by login to the server. After receiving the request, the server acquires the requested Cookie and takes out the accessToken from the Cookie. And (3) verifying by using the PyJWT Python library to verify timeliness and correctness of the accessToken. If the authentication fails, the user is redirected to the login page and is required to log in again. After verification is successful, the information of the user is analyzed from the accessToken, and the uploaded file is decompressed by using the zipfile and is decompressed into a file system of the server. And then writing the information of the uploading person and the information stored in the file into a DICOM file table of the database, and returning a URL path after file uploading to the Web front end by the server until the writing is completed. And finally, loading the DICOM file corresponding to the URL by using a remote loading data module of the medical image display subsystem. Through the above flow, the user can log in to upload the file, and finally successfully see the file on the page.

In this embodiment, optionally, the interactive segmentation subsystem further includes an image marking module;

The embodiment of the invention provides an image annotation function. The bottom layer uses a new label Canvas of HTML5, which is the basis for implementing drawing on the Web. Because the image marking function design is complex to draw, if the Canvas is used based on the Canvas, the Canvas is a Konva open source library, and Konva is a Canvas 2D drawing library realized by JavaScript, and various customized two-dimensional graphics can be easily drawn by using the Canvas. On the basis of Knova, labeling graphics of images frequently used by some doctors need to be realized. The ruler mainly realizes four functions of ruler function, dotting, drawing of rectangle and free painting brush.

In this scheme, specifically, the image tag uses data conversion to obtain the tag data capable of serialization, so as to construct error correction interaction information.

The purpose of the image marking function is not to draw a 2D pattern, but to convert the drawn data into serializable marking data, so that it is necessary to record information of marks under the marks on the basis of drawing the 2D pattern. The functions of redo and undo can be achieved with two stacks, above the recording of the marking information. And the functions of redoing and undoing are utilized, so that convenience is provided for doctors in the algorithm interaction process.

The embodiment of the invention provides an algorithm interaction function. The main aim of the algorithm interaction function is to provide a channel for doctors to interact with the interaction segmentation algorithm, and the interaction segmentation algorithm is the most core part of the interaction segmentation subsystem. The algorithm interaction function can liberate doctors from purely manual labeling, the doctors do not need to label the images at the pixel level any more, and the doctors can obtain the image segmentation result by providing a small amount of error correction interaction information on the basis of the segmentation result. In the implementation of the algorithm interaction prediction function, the algorithm interaction prediction method mainly comprises two modules: a front-end interaction module and a back-end algorithm prediction module.

In the front-end interaction module, an algorithm interaction mode is mainly realized, and in the algorithm interaction mode, a doctor can perform repeated iterative interaction with a background algorithm until satisfaction. And in the interaction mode, the doctor can check the result after any one or more interactions are completed for comparison. In the front-end interaction module, the front-end interaction module mainly comprises two stages, wherein the first stage is a rough segmentation stage, and the second stage is an interaction segmentation stage. In the rough segmentation stage, a user selects to enter an interactive mode in a toolbar, a request is sent to Nginx in data management, the Nginx serves as a reverse proxy server to forward the request to a background service, and the background service segments a current Dicom file by using a deep learning algorithm and returns a result. After the result is obtained, the data is required to be processed to be rendered into the renderable data, and the processed data is rendered by the segmentation result layer. The segmentation result layer is a transparent Canvas layer, and can be used for visually drawing data in a screen. After the drawing is completed, the system enters an interactive mode, i.e. an interactive segmentation stage. The doctor selects the corresponding marking tool to perform error correction by observing the segmentation result layer. The corresponding drawing is made on the graph by moving the mouse and the graph is displayed on the screen by the draglayer class. After the doctor's ' error correction ' is finished, selecting an ' interactive segmentation ' button in a toolbar, acquiring corresponding ' error correction information ' from a DrawController class in data management, sending the ' error correction information ' to an Nginx server, and acquiring an interactive segmentation result from a background service. And after data processing, the data is transmitted to a segmentation result layer for drawing. The doctor observes the latest segmentation result and makes a further labeling. In the interactive mode, the segmentation result layer can draw the results of multiple segmentations, and a doctor can compare according to the segmentation results of multiple selections. And obtaining a final segmentation result through several interactive iterations.

In the back-end algorithm prediction, DICOM segmentation is mainly performed by using a deep learning algorithm, and 'interactive error correction information' provided by doctors is recorded. The background server receives a request from the nginx server, the request is divided into rough segmentation and interactive segmentation, error correction interactive information exists in the interactive segmentation request, and the information is stored in a database in the background service for the purpose of carrying out model offline training subsequently. After receiving the request, an algorithm model is called, a file is acquired according to a DICOM file path in the algorithm model, model segmentation is performed, and the probability that each pixel is segmented out is output after the model segmentation. Such probability is subjected to data processing, and a threshold is usually set manually in the data processing, and the pixel is divided when the threshold is exceeded. And in order to increase the segmentation speed, the image is usually downsampled, so after the segmentation is completed, the downsampled pixels need to be mapped onto the original image. After mapping, a three-dimensional matrix with the same size as the original DICOM image is output, wherein only 1 and 0 (1 represents segmentation and 0 represents non-segmentation) are arranged in the matrix. And finally, carrying out contour segmentation on the three-dimensional array by using an openCV library to obtain segmented contours. And returning the segmented contour information to the nginx server, and returning the segmented contour information to the front end by the nginx for display.

In this embodiment, optionally, the interactive segmentation subsystem further includes an algorithm offline training module;

The embodiment of the invention provides an algorithm offline training function. The system sets a timer at the server to execute the task of timing training as soon as the time arrives. The server side can maintain a training file list, and the storage path of the file can be newly added in the training file list after the user uploads the file each time, so that the training sample of the scheme can be increased along with the continuous uploading of the file by the user. The algorithm model acquires the corresponding DICOM file from the file system from the file path in the file list, then continuously reads the marking information of the corresponding DIOCM file from the file system, checks whether the corresponding interactive error correction information exists in the database, and takes the information as information for supervised learning if the corresponding interactive error correction information exists in the database, so that model training is performed. After training is completed, the model is stored in a file system, and a stored path is returned to a server, and the server can obtain the latest model for inference when inferring next time.

In this embodiment, optionally, the medical image display subsystem and the interactive segmentation subsystem are implemented by adopting a containerized deployment.

The embodiment of the invention provides a containerized deployment function. The containerization technology shields the difference of the operating system level for the deployment of the system, can enable the development environment to be highly consistent with the production environment, and solves the uncertain factors caused by the environmental difference on the deployment. The system combines Jekins, docker, gitlab tools and the like to form a complete set of automatic flow from development to deployment. The Gitlab is a code management warehouse, and the developed codes are saved on a server; the Docker is a container generation tool, and can quickly generate and package running codes and environments into a Docker mirror image and run on various operating systems; jekins is a continuous inheritance tool whose main task is to monitor the code of the branches on the GitLab, automatically generate a new Docker image whenever a new change occurs to the branches, and run the image on a designated server.

For the entire deployment flow, the main point is how to generate the Docker mirror. To ensure that Docker performs the same product each time, it needs to be performed according to certain steps, which requires a script to do so, which becomes Dockerfile. Dockerfire is a tool for a developer to write a mirror image construction script for convenience, and the developer can generate a corresponding Docker mirror image only by correctly writing the script according to the grammar of the Dockerfire and executing the script. Dockerfile is a text file that is built up from a piece of Docker supplied instructions, each of which builds up a layer, and the Docker mirror is built up layer by layer. The system consists of two major parts, so each part corresponds to a Dockerfile file.

Fig. 4 is a schematic flow chart of interactive segmentation of a medical image according to an embodiment of the present invention. As shown in fig. 4, the interactive segmentation flow is:

the doctor may first view the DICOM image of the patient through the medical image presentation subsystem. When a doctor needs to divide the medical image, the system enters an algorithm division mode, firstly, a rough division model is called through a flash interface, the model predicts the division result, the division result is extracted, finally, the division boundary is returned, and a front-end page draws the division boundary. The doctor comprehensively observes the DICOM image and the rough segmentation result, and if the segmentation result is satisfied, a final segmentation result is obtained; if the segmentation result is unsatisfactory, the image is marked by using an image marking function provided by the previous section, and after marking is finished, the system transmits error correction interaction information to a background flash service through an Ajax request. After receiving the request, the flash service transmits error correction interaction information to the interaction segmentation model for prediction, and then stores the error correction interaction information into a database to accumulate training data for later model training. After the interactive segmentation model predicts the segmentation result, the frame of the segmentation result is extracted, and the segmentation result is returned to the front-end page for display through a flash service. And (3) confirming again by the doctor according to the segmentation result, and continuously providing error correction interaction information for the segmentation algorithm if the doctor is not satisfied, and completing the segmentation of the medical image after multiple iterations are performed.

Fig. 5 is an access schematic diagram of a medical image interaction segmentation platform according to an embodiment of the present invention. As shown in fig. 5, the access flow of the medical image interaction segmentation platform is as follows:

if the doctor interacts with the system through the browser, when the doctor accesses the browser to request the static resource, the browser sends an HTTP request to the Nginx server, the Nginx returns the static resource, and after the browser obtains the static resource, analysis and display are carried out. If doctor needs to call API of flash service (such as obtaining DICOM file, interacting with algorithm), HTTP request is sent to Nginx through browser, nginx is used as reverse proxy server to forward request to flash service, after flash service processes request (data processing and reading, algorithm model calling, etc.), result is returned to browser.

In an embodiment of the invention, the platform provides the capability to load and display images in DICOM files and to provide a range of functions that enable better viewing of medical images. Most importantly, the platform is capable of concatenating physician interaction information with the predicted outcome of the algorithm. The interactive medical image segmentation platform can relieve doctors from purely manual labeling, and solves the problems of insufficient robustness and precision of full-automatic algorithm segmentation results. The doctor does not need to label the image pixel by pixel, only provides corresponding error correction information on the basis of algorithm segmentation, and can obtain a final segmentation result through a few rounds of simple interaction. Meanwhile, the platform collects interaction information of doctors, and the algorithm model is periodically re-used by the information, so that the segmentation accuracy of the algorithm is improved.

Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.

Claims

1. An interactive medical image segmentation platform, the platform comprising: a medical image display subsystem and an interactive segmentation subsystem; wherein:

2. The platform of claim 1, wherein the interactive segmentation subsystem further comprises an algorithmic offline training module;

3. The platform of claim 1, wherein the interactive segmentation subsystem further comprises an image tagging module;

4. A platform according to claim 3, wherein the image markers are converted into serializable marker data by data conversion for error correction interactive information construction.

5. The platform of claim 1, wherein the medical image display subsystem further comprises an operations module;

6. The platform of claim 1, wherein the medical image display subsystem further comprises a three-view module;

7. The platform of claim 1, wherein the medical image display subsystem further comprises a multi-window module;

8. The platform of claim 1, wherein the medical image presentation subsystem is implemented using a reaction front end framework based on a B/S architecture.

9. The platform of claim 1, wherein the data transfer of the interactive segmentation subsystem is implemented based on a flash interface.

10. The platform of claim 1, wherein the medical image presentation subsystem and the interactive segmentation subsystem are implemented in a containerized deployment.