CN117437365A - Medical three-dimensional model generation method, device, electronic equipment and storage medium - Google Patents

Abstract

The application provides a method and a device for generating a medical three-dimensional model, electronic equipment and a storage medium, and relates to the technical field of computers. The method for generating the medical three-dimensional model comprises the following steps: responding to input operation in the interactive interface, and acquiring medical text; invoking a first generation network, and under the guidance of a medical text, learning a first tensor input into the first generation network to obtain a medical two-dimensional image conforming to the description of the medical text; the first generation network is a trained deep learning model with the capability of generating two-dimensional images from medical text to medical science; inputting the medical two-dimensional image into a second generation network to generate a medical three-dimensional model; the second generation network is a trained deep learning model with generation capabilities from a medical two-dimensional image to a medical three-dimensional model; and displaying the medical three-dimensional model generated by the second generation network in the interactive interface. The method and the device solve the problem of poor reality of the medical three-dimensional model in the related technology.

Description

Medical three-dimensional model generation method, device, electronic equipment and storage medium

Technical field

The present application relates to the field of computer technology. Specifically, the present application relates to a method, device, electronic equipment and storage medium for generating a medical three-dimensional model.

Background technique

At present, a large number of medical 3D models need to be used in both medical training and medical teaching. These 3D medical models are usually manually created by 3D model content companies. Not only is the virtual content realistic, but content editing also consumes a lot of time. And it cannot truly reflect the changes of patients in different situations and states.

It can be seen from the above that how to improve the realism of medical three-dimensional models has yet to be solved.

Contents of the invention

Each embodiment of the present application provides a method, device, electronic device, and storage medium for generating a medical three-dimensional model, which can solve the problem of poor authenticity of medical three-dimensional models existing in related technologies. The technical solutions are as follows:

According to one aspect of the present application, a method for generating a medical three-dimensional model includes: in response to an input operation in an interactive interface, obtaining medical text; calling the first generation network, and under the guidance of the medical text, to input the said The first tensor of the first generation network is learned to obtain a medical two-dimensional image that conforms to the description of the medical text; the first generation network is trained and has the depth to generate from medical text to medical two-dimensional images. Learning a model; inputting the medical two-dimensional image into a second generation network to generate a medical three-dimensional model; the second generation network is a deep learning model that has been trained and has the ability to generate from a medical two-dimensional image to a medical three-dimensional model. ; Display the medical three-dimensional model generated by the second generation network in the interactive interface.

According to one aspect of the present application, a device for generating a medical three-dimensional model includes: a text acquisition module, configured to acquire medical text in response to an input operation in an interactive interface; an image generation module, configured to call the first generation network, in Under the guidance of the medical text, the first tensor input to the first generation network is learned to obtain a medical two-dimensional image that conforms to the description of the medical text; the first generation network is trained and has the ability to A deep learning model with the ability to generate medical text into medical two-dimensional images; a model generation module for inputting the medical two-dimensional images into a second generation network to generate a medical three-dimensional model; the second generation network is trained, And a deep learning model with the ability to generate from medical two-dimensional images to medical three-dimensional models; a model display module used to display the medical three-dimensional model generated by the second generation network in the interactive interface.

According to one aspect of the present application, an electronic device includes at least one processor and at least one memory, wherein computer readable instructions are stored on the memory; the computer readable instructions are processed by one or more of the processors Execution enables the electronic device to implement the method for generating a medical three-dimensional model as described above.

According to one aspect of the present application, a storage medium has computer-readable instructions stored thereon, and the computer-readable instructions are executed by one or more processors to implement the method for generating a medical three-dimensional model as described above.

According to one aspect of the present application, a computer program product includes computer readable instructions, the computer readable instructions are stored in a storage medium, and one or more processors of an electronic device reads the computer readable instructions from the storage medium , loading and executing the computer-readable instructions, so that the electronic device implements the method for generating a medical three-dimensional model as described above.

The beneficial effects brought by the technical solution provided by this application are:

In the above technical solution, after obtaining the medical text, the first generation network with the ability to generate from medical text to medical two-dimensional images can be called to generate the first generation network input to the first generation network under the guidance of the medical text. A piece of data is learned to obtain a medical two-dimensional image that conforms to the medical text description, and then the medical two-dimensional image is input into the second generation network based on the second generation network with the ability to generate from medical two-dimensional images to medical three-dimensional models. The generation of medical three-dimensional models, and finally the medical three-dimensional models are generated and displayed in the interactive interface. It can be seen that on the one hand, two generation networks can be used to automatically and quickly generate medical three-dimensional models required for medical training or medical teaching. On the other hand, according to the actual needs of different medical training or medical teaching, input in the interactive interface Simple medical texts can guide the generation of medical three-dimensional models that can truly reflect changes in patients in different situations and states, thereby effectively solving the problem of poor authenticity of medical three-dimensional models existing in related technologies.

Description of the drawings

In order to explain the technical solutions in the embodiments of the present application more clearly, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of an implementation environment involved in this application;

Figure 2 is a flow chart of a method for generating a medical three-dimensional model according to an exemplary embodiment;

Figure 3 is a flow chart of the generation process of the first generation network from medical text to medical two-dimensional images according to an exemplary embodiment;

Figure 4 is a schematic diagram showing medical text to a medical three-dimensional model according to an exemplary embodiment;

Figure 5 is a flow chart of a process of constructing a first training set and a second training set according to an exemplary embodiment;

Figure 6 is a schematic diagram of the first training set and the second training set in the embodiment corresponding to Figure 5;

Figure 7 is a specific interaction diagram of a method for generating a medical three-dimensional model in an application scenario;

Figure 8 is a structural block diagram of a device for generating a medical three-dimensional model according to an exemplary embodiment;

Figure 9 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

FIG. 10 is a structural block diagram of an electronic device according to an exemplary embodiment.

Detailed ways

The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application and cannot be construed as limiting the present application.

Those skilled in the art will understand that, unless expressly stated otherwise, the singular forms "a", "an", "the" and "the" used herein may also include the plural form. It should be further understood that the word "comprising" as used in the description of the present disclosure refers to the presence of stated features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, Integers, steps, operations, elements, components and/or groups thereof. It will be understood that when we refer to an element being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Additionally, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. As used herein, the term "and/or" includes all or any unit and all combinations of one or more of the associated listed items.

The following is an introduction and explanation of several terms involved in this application:

AIGC, the full English name is AI-Generated Content, and the Chinese meaning is generative artificial intelligence.

Stable Diffusion means stable diffusion in Chinese.

VR, the full English name is Virtual Reality, and the Chinese meaning is virtual reality.

MR, the full English name is Mixed Reality, and the Chinese meaning is mixed reality.

As mentioned before, current medical 3D models are mainly created manually by 3D model content companies. They are usually produced using 3D software or generated after segmentation, reconstruction, and rendering of medical images. They still have the disadvantage of low realism and complex operation steps. , The production cost is high, there is no way to freely change and edit the suggested text description control content style, and it is difficult to adapt to the needs for different content styles in various scenarios such as medical training or medical teaching.

With the rapid development of deep learning models, generative artificial intelligence (AIGC) has begun to appear. The core idea of AIGC technology is to use artificial intelligence algorithms to generate content with certain creativity and quality. By training models and learning from large amounts of data, AIGC can generate relevant content based on input conditions or guidance. For example, by entering keywords, descriptions or samples, AIGC can generate matching articles, images, audio, etc.

The current AIGC performs well in natural image models and natural scenes. However, it is limited by the number of medical content data sets and cannot achieve training under large-scale medical content data sets. This makes it still unable to achieve the goal of generating medical-related content. Satisfactory results.

It can be seen from the above that there are still flaws in the related technology of poor realism of medical three-dimensional models, which makes it difficult to conduct accurate quantitative assessments in application scenarios such as medical training or medical teaching.

To this end, the method for generating a medical three-dimensional model provided by this application can effectively improve the authenticity of the medical three-dimensional model. Correspondingly, the method for generating a medical three-dimensional model is suitable for a medical three-dimensional model generating device. The generation of the medical three-dimensional model The device may be deployed on an electronic device, and the electronic device may be a computer device deployed with a von Neumann architecture. For example, the computer device includes a desktop computer, a notebook computer, a server, etc.

In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

Figure 1 is a schematic diagram of an implementation environment involved in a method for generating a medical three-dimensional model. It should be noted that this implementation environment is only an example adapted to the present application and cannot be considered to provide any restriction on the scope of use of the present application.

The implementation environment includes a client 110 and a server 130 .

Specifically, the user terminal 110 can be an electronic device that provides a display function. For example, the electronic device can be a desktop computer, a notebook computer, a server, etc. equipped with a display screen, or a smartphone, a tablet computer, etc. equipped with a touch screen.

Server 130. The server 130 can be an electronic device such as a desktop computer, a laptop computer, a server, etc., or it can be a computer cluster composed of multiple servers, or even a cloud computing center composed of multiple servers. The server 130 is used to provide background services. For example, the background services include but are not limited to medical three-dimensional model generation services and so on.

A communication connection is established in advance between the server 130 and the client 110 through wired or wireless means, and data transmission between the server 130 and the client 110 is implemented through this communication connection. The transmitted data includes but is not limited to: the first generation network, the second generation network, and so on.

In an application scenario, for the server 130, the generation service of the medical three-dimensional model is called so as to deploy the first generation network and the second generation network for generating the medical three-dimensional model to the client 110. Specifically, based on The first generation network is obtained by training with the pre-constructed first training set, and the second generation network is obtained by training based on the pre-constructed second training set, and the first generation network and the second generation network are sent to the user terminal 110 .

As the user end 110 interacts with the server end 130, the user end 110 receives the first generation network and the second generation network, and then completes the deployment of the first generation network and the second generation network. Then, the user uses the user end 110 to After inputting medical text into the interactive interface, the first generation network and the second generation network can be called to generate a medical three-dimensional model under the guidance of the medical text, so that the three-dimensional medical model can truly reflect the changes of patients in different situations and states, and Finally, the medical three-dimensional model is displayed in the interactive interface, thereby effectively solving the problem of poor authenticity of the medical three-dimensional model existing in related technologies.

Please refer to Figure 2. This embodiment of the present application provides a method for generating a medical three-dimensional model. The method is suitable for electronic equipment. The electronic equipment can be the user terminal 110 in the implementation environment shown in Figure 1.

In the following method embodiments, for convenience of description, the execution subject of each step of the method is an electronic device as an example, but this is not a specific limitation.

As shown in Figure 2, the method may include the following steps:

Step 310: Obtain medical text in response to the input operation in the interactive interface.

First of all, the interactive interface essentially refers to the page that interacts with the user during the generation of the medical 3D model. In some embodiments, the interactive interface is provided by a client running on the electronic device. It can be understood that as the client runs on the electronic device, the interactive interface can be displayed on the screen configured on the electronic device. The client can be in the form of an application program or a web page. Correspondingly, the page can be a program. It can be in the form of a window or a web page, which is not limited here.

Secondly, medical text is used to describe the content style, especially the content style of the two-dimensional medical images or three-dimensional medical models expected by the user. In other words, the medical text essentially reflects the different situations and states of patients. changes below. For example, a medical text can be described as "a lung with a tumor" or more accurately as "a tumor with a diameter of 20 mm in the right lung."

In order to understand the content style of the two-dimensional medical image or the three-dimensional medical model desired by the user, in some embodiments, an input portal is provided in the interactive interface so that the user can input medical text through the input portal.

For example, an input box is displayed in the interactive interface, and the user can enter medical text in the input box. For electronic devices, the input operation in the input box can be detected, and the user can understand the medical text expected by the user. 3D image or medical 3D model content style. Among them, the input box is regarded as the input entrance provided in the interactive interface, and the input operation is regarded as the input operation in the interactive interface.

Of course, in other embodiments, the input operations may differ according to different input components configured by the electronic device. For example, the electronic device is a desktop computer equipped with a keyboard. Correspondingly, the input operation may refer to mechanical operations such as clicking on the keyboard; the electronic device is a tablet computer equipped with a touch screen. Correspondingly, the input operation may refer to clicking, clicking on the touch screen, etc. Gesture operations such as sliding are not specifically limited here.

Step 330: Call the first generation network, and under the guidance of the medical text, learn the first tensor input to the first generation network to obtain a medical two-dimensional image that conforms to the description of the medical text.

Among them, the first generation network is a deep learning model that has been trained and has the ability to generate from medical text to medical two-dimensional images. That is to say, by training the deep learning model through the first training set, the first generation network with the ability to generate from medical text to medical two-dimensional images can be obtained. The first training set is composed of a large number of medical two-dimensional images. Constructed from images and corresponding medical text. Based on this, the first generation network essentially reflects the mathematical mapping relationship between medical texts and medical two-dimensional images. Then, through the call of the first generation network, the medical text and medical two-dimensional images reflected by the first generation network can be The mathematical mapping relationship between images is based on medical text mapping to obtain corresponding medical two-dimensional images.

In some embodiments, the first generative network includes a text encoder, a diffusion model, and an image decoder. Among them, the initial diffusion model may be a pre-trained Stable Diffusion model.

Figure 3 shows a schematic diagram of the generation process of the first generation network from medical text to medical two-dimensional images. In Figure 3, first, use the text encoder to convert the medical text into a second tensor (tensor), and obtain the randomly generated first tensor; then, control the second tensor to guide the diffusion model according to the description of the medical text. Content style, perform diffusion learning on the first tensor to obtain the third tensor; finally, use the image decoder to convert the third tensor into a medical two-dimensional image.

Among them, the process of diffusion learning specifically refers to: as shown in Figure 3, the first tensor is input into the diffusion model, and denoising is performed through the reverse process of the diffusion model; the second tensor is used as the guiding condition, and the guiding condition is introduced The denoising process of the first tensor results in the third tensor.

In the above process, since the medical text essentially describes the content style of the two-dimensional medical image expected by the user, the second tensor converted from the medical text can also reflect the content style of the two-dimensional medical image expected by the user. In the first Introducing the second tensor in the denoising process of the tensor is to guide the diffusion model to diffuse the first tensor in the content style expected by the user, and finally obtain a medical two-dimensional image that conforms to the medical text description, that is, the Medical 2D images conform to the content style expected by users. For example, if the medical text is "Lung with Tumor", the resulting two-dimensional medical image not only contains the lungs, but also shows tumors in the lungs; or, if the medical text is "Tumor with a diameter of 20 mm in the right lung" , the resulting medical two-dimensional image not only contains the left lung and the right lung, but also presents a tumor with a diameter of 20mm in the right lung. In this way, under the guidance of medical text, the medical two-dimensional image can truly reflect the Changes in patients in different situations and states.

It is worth mentioning that the first tensor refers to an image tensor of a set size. Correspondingly, the third tensor is also an image tensor of a set size. It can be understood that the size of the first tensor controls the size of the third tensor. The size of the tensor, and controls the image size of medical two-dimensional images. In other words, the set size can be flexibly set according to the actual needs for the image size of the medical two-dimensional image in the application scenario, which is not limited here.

Step 350: Input the two-dimensional medical image into the second generation network to generate a three-dimensional medical model.

Among them, the second generation network is a deep learning model that has been trained and has the ability to generate from medical two-dimensional images to medical three-dimensional models. In some embodiments, the deep learning model may be a pre-trained end-to-end deep learning model Pixel2Mesh.

That is to say, by training the deep learning model through the second training set, a second generation network with the ability to generate from medical two-dimensional images to medical three-dimensional models can be obtained, where the second training set is composed of a large number of the same or different The medical two-dimensional image and the corresponding medical three-dimensional model are constructed from the perspective of the patient. What is explained here is that the two-dimensional medical image corresponds to the three-dimensional medical model, which means that the two-dimensional medical image and the three-dimensional medical model at the same or different viewing angles have matching medical texts. It can also be understood as, the medical two-dimensional image at the same or different viewing angles has matching medical text. The two-dimensional medical image and the three-dimensional medical model conform to the content style described by the matching medical text.

Based on this, the first generation network essentially reflects the mathematical mapping relationship between the medical two-dimensional image and the medical three-dimensional model. Then, by inputting the medical two-dimensional image to the second generation network, the medical data reflected by the second generation network can be generated. The mathematical mapping relationship between the two-dimensional image and the medical three-dimensional model, and the corresponding medical three-dimensional model is obtained by mapping the medical two-dimensional image.

Figure 4 shows a schematic diagram of medical text to medical three-dimensional model. In Figure 4, the medical text is "a lung with tumors." Through the first generation network, a picture of a lung with tumors is obtained, that is, a two-dimensional medical image. On the basis of the two-dimensional medical image, through the second generation network, a three-dimensional mesh model of the lung with tumors, that is, a three-dimensional medical model, is obtained.

Step 370: Display the medical three-dimensional model generated by the second generation network in the interactive interface.

After obtaining the medical three-dimensional model, the medical three-dimensional model can be displayed to the user in the interactive interface. It should be noted that with the differences in display functions provided by different electronic devices, for example, the resolutions of display screens configured with different electronic devices are different, before displaying the medical 3D model in the interactive interface, it is necessary to compare it according to the data format compatible with the electronic device. The generated medical three-dimensional model is encoded to output a medical three-dimensional model adapted to the electronic device.

Through the above process, on the one hand, two generation networks can be used to automatically and quickly generate medical three-dimensional models required for medical training or medical teaching. On the other hand, according to the actual needs of different medical training or medical teaching, the interactive interface can be By inputting simple medical text into the system, the generated medical 3D model can truly reflect the changes of patients in different situations and states, thereby effectively solving the problem of poor authenticity of medical 3D models existing in related technologies.

Referring to Figure 5, in an exemplary embodiment, the above method may further include the following steps:

Step 410: Obtain the original medical image and annotate the original medical image with the medical text to obtain the medical annotation image.

Among them, medical annotation images refer to medical original images carrying medical text.

Regarding the acquisition of original medical images, it can be obtained from medical images published on the Internet, private medical image data from major hospitals/medical schools and other organizations, or from various public medical competition data; further, Based on the original medical image obtained above, it can also be segmented according to different organs, tissues, diseased parts, etc. For example, an original medical image containing the left lung and the right lung can be segmented into two original medical images according to the organs to form A large-scale medical original image data set with various organs, tissues, and diseased parts.

Secondly, annotation refers to adding medical text to medical original images. In some embodiments, the medical text can be added to the original medical image as a text label, or can also be added to the original medical image in the form of a file name, which is not limited here.

Step 430: Perform three-dimensional image reconstruction calculation on the medical annotation image to obtain a three-dimensional annotation model.

Among them, the three-dimensional annotation model carries medical text corresponding to the medical annotation image.

Specifically, based on the medical annotation image, determine the original medical image and the medical text it carries; use three-dimensional image reconstruction technology to perform three-dimensional image reconstruction calculations on the original medical image to obtain the original three-dimensional medical model; use the original medical image to carry the medical text The medical text annotates the original medical three-dimensional model to obtain a three-dimensional annotation model.

That is to say, using three-dimensional image reconstruction technology, each medical annotation image can obtain a corresponding three-dimensional annotation model. It should be noted that correspondence means that the medical annotation image and the three-dimensional annotation model have matching medical texts, which can also be understood as , both the original medical image and the original three-dimensional medical model conform to the content style described by the matching medical text.

Step 450: Decompose the three-dimensional annotation model into multiple two-dimensional annotation images according to different viewing angles.

Among them, each two-dimensional annotation image corresponds to a different perspective and carries medical text corresponding to the three-dimensional annotation model.

Specifically, based on the three-dimensional annotation model, determine the original medical three-dimensional model and the medical text it carries; decompose the original medical three-dimensional model into multiple original medical two-dimensional images from different perspectives; use the medical text carried by the original medical three-dimensional model Label multiple original medical two-dimensional images from different viewing angles to obtain multiple two-dimensional annotated images from different viewing angles.

What is explained here is that for each three-dimensional annotation model, there are corresponding multiple two-dimensional annotation images from different perspectives. The three-dimensional annotation model and its corresponding multiple two-dimensional annotation images have matching medical texts. , it can also be understood that the original medical three-dimensional model and its corresponding multiple original medical two-dimensional images conform to the content style described by the matching medical text.

Step 470: Construct a first training set based on each two-dimensional annotated image and the medical text it carries, and construct a second training set based on the three-dimensional annotation model and each two-dimensional annotated image carrying the medical text corresponding to the three-dimensional annotation model. set.

Among them, the two-dimensional annotated image refers to the original medical two-dimensional image carrying medical text; the three-dimensional annotated model refers to the original medical three-dimensional model carrying medical text.

As shown in Figure 6, the first training set is composed of a large number of original medical two-dimensional images and the medical texts they carry, that is, the first training set is a data set from medical texts to medical two-dimensional images; and the second training set It is composed of a large number of original medical three-dimensional models, corresponding multiple original medical two-dimensional images from different perspectives, and the medical texts they carry. That is, the second training set is a data set from medical two-dimensional images to medical three-dimensional models.

After obtaining the first training set, the first generation network can be trained based on the first training set, which may include the following steps: obtaining an initial diffusion model; the initial diffusion model is a pre-trained stable diffusion StableDiffusion model; based on the first A training set, perform parameter tuning training on the initial diffusion model. If the parameter tuning training of the initial diffusion model meets the set conditions, the diffusion model that has completed the training will be obtained; based on the text encoder, the diffusion model that has completed the training, and the image Decoder, builds the first generative network.

After obtaining the second training set, the second generation network can be trained based on the second training set, which may include the following steps: obtaining a deep learning model pre-trained using the natural image training set; based on the second training set, the depth The learning model performs parameter tuning training; if the parameter tuning training of the deep learning model meets the set conditions, the second generation network is obtained.

Among them, the setting conditions can be flexibly set according to the actual needs of the application scenario. For example, in one application scenario, the setting conditions can mean that the parameters are optimized to improve the training accuracy of the model; in another application scenario, the setting conditions can be The certain condition may mean that the number of iterations reaches a threshold to improve the training efficiency of the model, which is not limited here. Among them, the parameters are optimized, which can be achieved through loss functions, etc., and are not limited here.

Under the influence of the above embodiments, by constructing a medical text-medical three-dimensional model data set and applying it to the pre-trained model for training, an end-to-end medical text to medical three-dimensional model network is realized, which can be applied to Multi-clinical scenarios and categories in medicine improve the performance of the original text prompt-based three-dimensional AIGC generation model in medical content generation, so as to provide new technologies and tools for subsequent VR/MR clinical practice training and assessment.

The current traditional medical education system, which mainly focuses on animal specimens, human specimens and teaching aids, has problems such as insufficient resources and a high risk of harm to patients in the training process of medical talents. Virtual reality technology (VR) uses computers to generate virtual three-dimensional scenes, bringing users visual, auditory and tactile immersion. With the development of virtual reality and mixed reality (MR) technology in recent years, applying VR and MR technology to medical education, training and assessment has become a new trend. Compared with the traditional teaching model, VR and MR technology have very significant advantages. By collecting multi-dimensional data from the real human body and constructing a digital human body or target tissue model through simulation modeling, low-cost, repeatable, and quantifiable evaluation can be carried out. Digital teaching allows students to learn and grow in a repeatable practice environment, and provides rich case studies and scientific and standardized simulation materials, which can effectively improve the problems of insufficient teaching resources and difficulty in quantitative assessment. At the same time, in clinical teaching and preoperative planning, the use of mixed reality technology can achieve better results.

Current mixed reality medical training and teaching tasks require the use of a large number of medical 3D models. On the one hand, the mainstream method of existing medical 3D virtual models is to use geometric modeling software to directly create them. There are also some solutions that use medical images through segmentation, reconstruction, rendering, etc. Steps are taken to obtain a 3D model, but the operation steps are complicated and the production cost is high. There is no way to control the free change and editing of the content style through simple text descriptions. On the other hand, the current AIGC 3D generated large model is not suitable for natural image models and natural scenes. The effect is okay, but it cannot achieve satisfactory results in generating medical-related content, and it is difficult to adapt to the needs of different content styles in various medical training scenarios.

Figure 7 shows a schematic diagram of a medical three-dimensional model in an application scenario. As shown in Figure 7, in this application scenario, the server can be a server, etc., and both the first client and the second client can be electronic devices capable of interacting with users. For example, the first client can be a desktop computer. , the second user terminal can be a laptop, etc. Then, the first user terminal can generate a medical three-dimensional model from the medical text input by the user through user interaction, and the second user terminal can perform medical treatment on the user through user interaction. Assessments such as training or medical teaching. It is worth mentioning that the first client and the second client can also be deployed on the same electronic device, and this does not constitute a specific limitation.

Specifically, the server constructs a first training set and a second training set so as to obtain a first generation network based on training on the first training set and a second generation network based on training on the second training set, and deploy the two to the second generation network. A client.

After the first user terminal completes the deployment of the first generation network and the second generation network, the user can input the corresponding medical text with the help of the first user terminal according to the desired content style of the medical three-dimensional model, so that the first user terminal can pass The first generation network is called to generate a medical two-dimensional image from the medical text, and the second generation network is called to generate a medical three-dimensional model from the medical two-dimensional image. The medical three-dimensional model thus obtained conforms to the content style expected by the user, and then the medical three-dimensional model is generated. The medical three-dimensional model is transmitted to the second user terminal.

As the client in the second user terminal runs, the second user terminal will display to the user a virtual scene with the medical three-dimensional model imported. The virtual scene is a digital scene constructed using computer technology for assessments such as medical training or medical teaching. Simulate the environment required for medical training or medical teaching (such as a medical surgery laboratory), and then assess the user's medical training or medical teaching by capturing the user's simulated operations in the simulated environment.

For example, when a user wishes to participate in a medical training assessment, the client can be started to enter the corresponding virtual scene. For example, the virtual scene can be a simulated medical surgery laboratory, and the medical three-dimensional model imported into the virtual scene can be a certain The lungs of a patient with tumors. At this time, the simulated operations performed by the user on the patient's tumor-bearing lung include but are not limited to: the user's simulated operations on surgical instruments, the user's reply operations on the assessment content, etc. As the user simulates the operation process, the second user terminal can also collect corresponding operation videos based on the image acquisition device and obtain corresponding sensor data based on the mixed reality device to assess the user's medical training.

In this application scenario, by applying AIGC technology to the medical field, AIGC technology is applied to the generation of medical mixed reality models, filling the application gap in this area; it can also make up for the lack of medical content learning in AIGC. The shortcomings in medical content model generation, while using this generation solution, can solve the needs of 3D models in different scenarios in mixed reality medical simulation training and teaching; in addition, through this method of text generation, multiple conditions can be quickly obtained 3D medical content, reducing the cost of generating 3D models for mixed reality medical development.

The following are device embodiments of the present application, which can be used to execute the method for generating a medical three-dimensional model involved in the present application. For details not disclosed in the device embodiments of this application, please refer to the embodiments of the medical three-dimensional model generation method involved in this application.

Referring to Figure 8, an embodiment of the present application provides a device 900 for generating a medical three-dimensional model, including but not limited to: a text acquisition module 910, an image generation module 930, a model generation module 950, and a model display module 970.

Among them, the text acquisition module 910 is used to acquire medical text in response to input operations in the interactive interface.

The image generation module 930 is used to call the first generation network, and under the guidance of the medical text, learn the first tensor input to the first generation network to obtain a medical two-dimensional image that conforms to the description of the medical text. The first generative network is a deep learning model that has been trained and has the ability to generate from medical text to medical two-dimensional images.

The model generation module 950 is used to input the medical two-dimensional image into the second generation network to generate the medical three-dimensional model. The second generation network is a deep learning model that has been trained and has the ability to generate from medical two-dimensional images to medical three-dimensional models.

The model display module 970 is used to display the medical three-dimensional model generated by the second generation network in the interactive interface.

In an exemplary embodiment, the first generative network includes a text encoder, a diffusion model, and an image decoder.

Wherein, the image generation module 930 is also used to use the text encoder to convert the medical text into a second tensor, and obtain the randomly generated first tensor; control the second tensor Guide the diffusion model to perform diffusion learning on the first tensor according to the content style described in the medical text to obtain a third tensor; use the image decoder to convert the third tensor into the Medical 2D images.

In an exemplary embodiment, the image generation module 930 is further configured to input the first tensor into the diffusion model, and perform denoising through the reverse process of the diffusion model; using the second tensor As a guiding condition, and introducing the guiding condition into the denoising process of the first tensor, the third tensor is obtained.

In an exemplary embodiment, the apparatus 900 further includes: a training set construction module.

Wherein, the training set construction module is used to obtain original medical images, annotate the original medical images with medical texts, and obtain medical annotated images; the medical annotated images refer to original medical images carrying medical texts; Three-dimensional image reconstruction calculation is performed on the medical annotation image to obtain a three-dimensional annotation model; the three-dimensional annotation model carries medical text corresponding to the medical annotation image; the three-dimensional annotation model is decomposed into a plurality of two-dimensional annotation models according to different viewing angles. 2D annotated images; each two-dimensional annotated image corresponds to a different perspective and carries medical text corresponding to the three-dimensional annotated model; based on each two-dimensional annotated image and the medical text it carries, a first training set is constructed, And based on the three-dimensional annotation model and each of the two-dimensional annotation images carrying the medical text corresponding to the three-dimensional annotation model, a second training set is constructed; wherein the first training set is used for training to obtain the The first generating network, the second training set is used to train to obtain the second generating network.

In an exemplary embodiment, the device 900 further includes: a first training module.

Wherein, the first training module is used to obtain an initial diffusion model; the initial diffusion model is a pre-trained stable diffusion model; based on the first training set, the initial diffusion model is Parameter tuning and training are performed to obtain the diffusion model that has been trained; and the first generation network is constructed based on the text encoder, the diffusion model that has been trained, and the image decoder.

In an exemplary embodiment, the device 900 further includes: a second training module.

Wherein, the second training module is used to obtain a deep learning model pre-trained using a natural image training set; based on the second training set, perform parameter tuning training on the deep learning model; if the deep learning If the parameter tuning training of the model meets the set conditions, the second generating network is obtained.

In an exemplary embodiment, the device 900 further includes: an assessment module.

Wherein, the assessment module is used to import the medical three-dimensional model into the constructed virtual scene; the virtual scene is constructed for medical training or medical teaching; based on the target object in the virtual scene, based on the medical The simulation operation of the three-dimensional model is used to assess the medical training or medical teaching of the target object.

It should be noted that when the device for generating a medical three-dimensional model provided in the above embodiments generates a medical three-dimensional model, only the division of the above functional modules is used as an example. In practical applications, the above functions can be allocated from different modules as needed. The functional modules are completed, that is, the internal structure of the medical three-dimensional model generation device will be divided into different functional modules to complete all or part of the functions described above.

In addition, the device for generating a medical three-dimensional model and the method for generating a three-dimensional medical model provided in the above embodiments belong to the same concept. The specific manner in which each module performs operations has been described in detail in the method embodiment and will not be used here. Again.

Figure 9 shows a schematic structural diagram of an electronic device according to an exemplary embodiment. The electronic device is suitable for the client 110 in the implementation environment shown in FIG. 1 .

It should be noted that this electronic device is only an example adapted to the present application and cannot be considered to provide any limitation on the scope of use of the present application. The electronic device is also not to be construed as being dependent on or required to have one or more components of the exemplary electronic device 2000 shown in FIG. 9 .

The hardware structure of the electronic device 2000 may vary greatly due to different configurations or performance. As shown in FIG. 9 , the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU). ,Central Processing Units) 270.

Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 2000 .

The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted to this application, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc., as shown in Figure 9, which is not intended to be used here. This constitutes a specific limitation.

As a carrier for resource storage, the memory 250 can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc. The resources stored thereon include the operating system 251, application programs 253, data 255, etc., and the storage method can be short-term storage or permanent storage. .

Among them, the operating system 251 is used to manage and control each hardware device and application program 253 on the electronic device 2000 to realize the operation and processing of the massive data 255 in the memory 250 by the central processor 270. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

The application program 253 is a computer-readable instruction based on the operating system 251 to complete at least one specific work. It may include at least one module (not shown in FIG. 9 ), and each module may include a computer program for the electronic device 2000 . Readable instructions. For example, the device for generating a medical three-dimensional model can be regarded as an application program 253 deployed on the electronic device 2000 .

The data 255 may be photos, pictures, etc. stored in a disk, or may be a first generated network, a second generated network, etc., stored in the memory 250 .

The central processing unit 270 may include one or more processors, and is configured to communicate with the memory 250 through at least one communication bus to read the computer readable instructions stored in the memory 250, and thereby implement processing of the massive data 255 in the memory 250. operation and processing. For example, the method of generating a medical three-dimensional model is completed by the central processor 270 reading a series of computer-readable instructions stored in the memory 250 .

In addition, the present application can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, implementation of the present application is not limited to any specific hardware circuit, software, or combination of the two.

Referring to FIG. 10 , an electronic device 4000 is provided in an embodiment of the present application. The electronic device 400 may include: a desktop computer, a notebook computer, a server, etc.

In FIG. 10 , the electronic device 4000 includes at least one processor 4001 and at least one memory 4003 .

Among them, data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect, Peripheral Component Interconnect Standard) bus or an EISA (Extended Industry Standard Architecture, Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of presentation, only one thick line is used in Figure 10, but it does not mean that there is only one bus or one type of bus.

Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and/or data reception. It should be noted that in practical applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

The processor 4001 can be a CPU (Central Processing Unit, central processing unit), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), ASIC (Application Specific Integrated Circuit, application specific integrated circuit), FPGA (Field Programmable Gate Array, field programmable gate array) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various illustrative logical blocks, modules, and circuits described in connection with this disclosure. The processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types that can store information and instructions. Dynamic storage devices can also be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact DiscRead Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, Laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program instructions or codes in the form of instructions or data structures and can be stored by the electronic device 400 any other medium, but not limited to this.

Computer-readable instructions are stored in the memory 4003, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.

The computer readable instructions are executed by one or more processors 4001 to implement the method for generating a medical three-dimensional model in the above embodiments.

In addition, embodiments of the present application provide a storage medium with computer-readable instructions stored on the storage medium. The computer-readable instructions are executed by one or more processors to achieve the generation of the medical three-dimensional model as described above. method.

An embodiment of the present application provides a computer program product. The computer program product includes computer readable instructions. The computer readable instructions are stored in a storage medium. One or more processors of the electronic device read the computer readable instructions from the storage medium. , loading and executing the computer-readable instructions, so that the electronic device implements the method for generating a medical three-dimensional model as described above.

It should be understood that although various steps in the flowchart of the accompanying drawings are shown in sequence as indicated by arrows, these steps are not necessarily performed in the order indicated by arrows. Unless explicitly stated in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders. Moreover, at least some of the steps in the flow chart of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and their execution order is also It does not necessarily need to be performed sequentially, but may be performed in turn or alternately with other steps or sub-steps of other steps or at least part of the stages.

The above are only some of the embodiments of the present application. It should be pointed out that those of ordinary skill in the technical field can also make several improvements and modifications without departing from the principles of the present application. These improvements and modifications can also be made. should be regarded as the scope of protection of this application.

Claims (10)

1. A method for generating a medical three-dimensional model, characterized in that the method includes: Obtain medical text in response to input operations in the interactive interface; Call the first generation network, and under the guidance of the medical text, learn the first tensor input to the first generation network to obtain a medical two-dimensional image that conforms to the description of the medical text; the first generation network It is a deep learning model that has been trained and has the ability to generate from medical text to medical two-dimensional images; Input the medical two-dimensional image into a second generation network to generate a medical three-dimensional model; the second generation network is a deep learning model that has been trained and has the ability to generate a medical three-dimensional model from a medical two-dimensional image; The medical three-dimensional model generated by the second generation network is displayed in the interactive interface. 2. The method of claim 1, wherein the first generation network includes a text encoder, a diffusion model, and an image decoder; The first generation network is called, and under the guidance of the medical text, the first tensor input to the first generation network is learned to obtain a medical two-dimensional image that conforms to the description of the medical text, including: Using the text encoder, convert the medical text into a second tensor, and obtain the randomly generated first tensor; Control the second tensor to guide the diffusion model to perform diffusion learning on the first tensor according to the content style described in the medical text to obtain a third tensor; Using the image decoder, the third tensor is converted into the medical two-dimensional image. 3. The method of claim 2, wherein the controlling the second tensor guides the diffusion model to perform diffusion learning on the first tensor according to the content style described in the medical text, Get the third tensor, including: Input the first tensor into the diffusion model, and perform denoising through the reverse process of the diffusion model; Using the second tensor as a guiding condition and introducing the guiding condition into the denoising process of the first tensor, the third tensor is obtained. 4. The method of claim 1, further comprising: Obtain the original medical image, and annotate the original medical image with the medical text to obtain the medical annotated image; the medical annotated image refers to the original medical image carrying the medical text; Perform three-dimensional image reconstruction calculations on the medical annotation image to obtain a three-dimensional annotation model; the three-dimensional annotation model carries medical text corresponding to the medical annotation image; Decompose the three-dimensional annotation model into multiple two-dimensional annotation images according to different viewing angles; each two-dimensional annotation image corresponds to a different viewing angle and carries medical text corresponding to the three-dimensional annotation model; A first training set is constructed based on each of the two-dimensional annotated images and the medical text they carry, and based on the three-dimensional annotation model and each of the two-dimensional annotated images carrying the medical text corresponding to the three-dimensional annotation model. , construct the second training set; Wherein, the first training set is used to train the first generating network, and the second training set is used to train the second generating network. 5. The method of claim 4, wherein the first generating network is called, and under the guidance of the medical text, the first tensor input to the first generating network is learned to obtain a conforming result. Before the medical text describes the two-dimensional medical image, the method further includes: Obtain an initial diffusion model; the initial diffusion model is a pre-trained stable diffusion StableDiffusion model; Based on the first training set, perform parameter tuning training on the initial diffusion model to obtain the diffusion model that has completed training; The first generative network is constructed based on a text encoder, the trained diffusion model, and an image decoder. 6. The method of claim 4, wherein before inputting the medical two-dimensional image into the second generation network to generate the medical three-dimensional model, the method further includes: Obtain the deep learning model pre-trained using the natural image training set; Based on the second training set, perform parameter tuning training on the deep learning model; If the parameter tuning training of the deep learning model meets the set conditions, the second generation network is obtained. 7. The method according to any one of claims 1 to 6, wherein after displaying the medical three-dimensional model generated by the second generation network in the interactive interface, the method includes: Import the medical three-dimensional model into the constructed virtual scene; the virtual scene is constructed for medical training or medical teaching; Based on the target object's simulated operation on the medical three-dimensional model in the virtual scene, the medical training or medical teaching of the target object is assessed. 8. A device for generating a medical three-dimensional model, characterized in that the device includes: A text acquisition module, used to acquire medical text in response to input operations in the interactive interface; The image generation module is used to call the first generation network, and under the guidance of the medical text, learn the first tensor input to the first generation network to obtain a medical two-dimensional image that conforms to the description of the medical text; The first generation network is a deep learning model that has been trained and has the ability to generate from medical text to medical two-dimensional images; A model generation module for inputting the medical two-dimensional image into a second generation network to generate a medical three-dimensional model; the second generation network is trained and has the ability to generate from a medical two-dimensional image to a medical three-dimensional model. deep learning model; A model display module is used to display the medical three-dimensional model generated by the second generation network in the interactive interface. 9. An electronic device, characterized by comprising: at least one processor and at least one memory, wherein, Computer readable instructions are stored on the memory; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the method for generating a medical three-dimensional model according to any one of claims 1 to 7. 10. A storage medium with computer-readable instructions stored thereon, characterized in that the computer-readable instructions are executed by one or more processors to implement the method of any one of claims 1 to 7 Generating method of medical three-dimensional model.