JP2024172049A - IMAGE GENERATION DEVICE, IMAGE GENERATION METHOD, PROGRAM, AND IMAGE - Google Patents

Abstract

To provide a technique for reducing the effort required for creating a highly accurate medical illustration.SOLUTION: An image generation apparatus 10 generates a medical image using a 3D model. The image generation apparatus 10 has: a selection unit 11 which selects a mode on the basis of an instruction from a user; a designation unit 12 which receives a designation of a 3D model from the user; an acquisition unit 13 which acquires the at least one designaged 3D model from a database 30 which stores a 3D model group; a modifying unit 14 which modifies a state of the 3D model on the basis of a request to modify the state of the 3D model, in accordance with the selected mode; and a display control unit 15 which displays an image including the 3D model in the modified state, on a display screen.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image generating device, an image generating method, a program, and an image generated using the image generating device.

Medical papers usually use illustrations such as anatomical diagrams to make the content clearer. Illustrations used in medical papers need to be created based on proper knowledge of anatomy, and also need to meet the specific requests of the paper's author. For this reason, traditionally, illustrators with specialized knowledge would spend time meeting with the paper's author to understand the specific requests, and then create illustrations that meet those requests.

In other words, a huge amount of effort is required to create medical illustrations. Furthermore, there is a high demand for medical illustrations from the perspective of medical education.

The present invention has been made in consideration of the above problems, and has as an illustrative object to provide a technology that reduces the effort required to create highly accurate medical illustrations.

In order to achieve the above object, an image generating device according to one aspect of the present invention is an image generating device that generates medical images using a 3D model, and is characterized by having a selection unit that selects a mode based on an instruction from a user, a designation unit that accepts the designation of the 3D model from the user, an acquisition unit that acquires at least one of the designated 3D models from a database in which a group of 3D models is stored, a change unit that changes the state of the 3D model based on a request to change the state of the 3D model according to the selected mode, and a display control unit that displays an image including the 3D model in the changed state on a display screen.

According to one aspect of the present invention, it is possible to provide a technology that reduces the effort required to create highly accurate medical illustrations.

1 is a configuration diagram showing a configuration of an image generating system according to an embodiment of the present invention. FIG. 2 is a block diagram showing a hardware configuration of the image generating apparatus. FIG. 13 is a flowchart showing an image generation process in a model selection mode. FIG. 11 is a diagram illustrating a first example of an image generated in the model selection mode. FIG. 13 is a diagram illustrating a second example of an image generated in the model selection mode. FIG. 13 is a diagram illustrating a third example of an image generated in the model selection mode. FIG. 13 is a diagram illustrating a fourth example of an image generated in the model selection mode. FIG. 13 is a diagram illustrating a fifth example of an image generated in the model selection mode. FIG. 11 is a flow diagram showing an image generation process in a model placement mode. FIG. 13 is a diagram illustrating an example of an image generated in the model placement mode. 11A and 11B are diagrams illustrating an example of an image generated when a gaze selection is performed. FIG. 13 is a flowchart showing an image generation process in a pose change mode. 11A and 11B are diagrams illustrating examples of images generated in a pose change mode. FIG. 11 is a flow diagram showing an image generation process in an echo mode. 1A and 1B are diagrams illustrating examples of images generated in echo mode. FIG. 11 is a flowchart showing image generation processing in an endoscope mode. 11A and 11B are diagrams illustrating an example of an image generated in an endoscope mode.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

<System Configuration>
FIG. 1 is a configuration diagram showing the configuration of an image generating system 1 according to the present embodiment. The image generating system 1 includes an image generating device 10, a client device 20, a database 30, and an administrator device 40. The image generating system 1 is a system that generates a medical image using a 3D model based on an input operation by a user. In the image generating system 1, the 3D model can be freely changed to a state that the user envisions, and an image of that state can be acquired. The image generating device 10 and the client device 20, the image generating device 10 and the database 30, and the image generating device 10 and the administrator device 40 are each connected to be able to communicate with each other via a network 50. The network 50 includes, for example, the Internet, a wide area network (WAN), a local area network (LAN), a wireless base station such as WiFi, a provider device, a dedicated line, and the like. Note that in this embodiment, an example will be described in which the image generating device 10 and the database 30 are separate devices, but the database 30 may have the functions of the image generating device 10. Moreover, the image generating device 10 may have the functions of the administrator device 40 .

First, the client device 20 will be described. The client device 20 is a computer (information processing device) that displays an operation screen for performing image generation processing in the image generating device 10 and an image generated by the image generating device 10 based on instructions from a user. The client device 20 is, for example, a PC (Personal Computer), a tablet PC, a smartphone, etc. A user uses the client device 20 to input an instruction for image generation by the image generating device 10. The client device 20 includes an input device such as a touch panel, a keyboard, or a mouse that accepts operation input from the user. The client device 20 also includes a display device such as a liquid crystal display panel, a plasma display panel, or an organic EL display panel. The display device displays an operation screen for performing image generation processing in the image generating device 10 and an image generated by the image generating device 10. The client device 20 also includes a CPU as a control unit for executing programs, etc.

Next, the image generating device 10 will be described. The image generating device 10 is a computer, e.g., a server device, that has multiple modes and generates medical images using 3D models based on user operations input using a client device 20. The image generating device 10 may be realized, for example, by a cloud server that uses cloud computing technology. The image generating device 10 includes a selection unit 11, a designation unit 12, an acquisition unit 13, a change unit 14, a display control unit 15, and an output unit 16.

The image generating device 10 has multiple modes, such as a model selection mode (first mode), a model placement mode (second mode), a pose change mode (third mode), an echo mode (fourth mode), and an endoscope mode (fifth mode). The selection unit 11 selects a mode based on an instruction input by the user via the client device 20.

The designation unit 12 accepts the designation of a 3D model input by the user via the client device 20. Specifically, the designation unit 12 causes the client device 20 to display selectable 3D model categories according to the selected mode. The user designates a desired category from the displayed categories. The designation unit 12 accepts the designated category and causes the client device 20 to display 3D model items included in the designated category. The user designates a 3D model from which an image is desired to be obtained from the displayed items. In other words, the user designates a 3D model required to create an image of the state the user envisions from the displayed items. The designation unit 12 then accepts the designation.

The acquisition unit 13 acquires at least one specified 3D model from a database 30 in which multiple 3D models, i.e., a group of 3D models, are stored.

Here, the database 30 will be described. The database 30 stores a group of 3D models, including medical models including human bodies, each part of the human body (including blood vessels, muscles, bones, etc.), human organs, and diseases such as inflammation and tumors that occur in the human body, veterinary models including animal bodies, each part of the human body (including blood vessels, muscles, bones, etc.), animal organs, and diseases such as inflammation and tumors that occur in the animal body, as well as models of medical equipment and instruments. In addition, the database 30 also stores special models that combine 3D models with illustrations, and animation models that add movement to the human body or animal body. Here, the animation models are 3D models with preset animation functions. The 3D models stored in the database 30 are 3D models created based on appropriate knowledge based on anatomy, and are general 3D models necessary for learning physiology, histology, embryology, and other medical sciences.

The change unit 14 changes the state of the 3D model based on a request to change the state of the 3D model according to the mode selected by the selection unit 11. That is, the change unit 14 changes the state of the 3D model based on a request to change the state of the 3D model input by the user via the client device 20.

The display control unit 15 displays the 3D model in the changed state on the display screen of the client device 20.

The output unit 16 outputs to the client device 20 information for acquiring an image including a 3D model displayed on the display screen of the client device 20. Here, the information for acquiring the image is, for example, a URL and a code for downloading the image. Note that the image itself may be output as an image file in a downloadable format such as JPEG, GIF, TIFF, PNG, BMP, or RAW.

Next, the administrator device 40 will be described. The administrator device 40 is a computer used by an administrator who manages the image generation system 1. The administrator device 40 is, for example, a PC, a tablet PC, a smartphone, etc. The administrator device 40 accepts, for example, custom orders, which will be described later. The administrator device 40 may have a function of performing maintenance of the database 30, such as adding and changing 3D models in the database 30, and maintenance of the image generation device 10.

Next, the hardware configuration of the image generation system 1 will be described with reference to FIG. 2. FIG. 2 is a block diagram showing the hardware configuration of the image generation device 10. The image generation device 10 includes a storage unit 61, a RAM 62, a ROM 63, a CPU 64, and a communication unit 65.

The storage unit 61 stores programs executed by the CPU 64 (described below) and data used by such programs. The storage unit 61 can also store various data used in processing, created prediction models, etc.

The CPU 64 operates based on a program stored in the ROM 63 or the storage unit 61, and controls each unit of the image generating device 10. The ROM 63 stores a boot program executed by the CPU 64 when the image generating device 10 is started up, programs that depend on the hardware of the image generating device 10, and the like. The CPU 64 realizes a flow described below, for example, by executing a program loaded onto the RAM 62. Note that the CPU 64 may obtain these programs from another device, for example via a network, and execute them.

The communication unit 65 can receive data or instructions from an external device, such as another device, via a network and send them to the CPU 64, and can transmit data or instructions generated by the CPU 64 to other devices. Note that the image generating device 10 can be connected to an input device and a display device.

The client device 20, database 30, and administrator device 40 each have a similar configuration to that shown in FIG. 2, but detailed description will be omitted. In addition to the configuration shown in FIG. 2, the client device 20 and administrator device 40 are connected to an input device and a display device. The input device is, for example, a touch panel, keyboard, mouse, etc. that accepts operational input from the user. The display device is, for example, a liquid crystal display panel, a plasma display panel, an organic EL display panel, etc.

<Model selection mode>
The model selection mode will now be described. The model selection mode is a mode in which an image envisioned by a user is generated using a 3D model stored in the database 30. Fig. 3 is a flow diagram showing an image generation process in the model selection mode. Each operation (step) shown in this flow chart can be executed under the control of the CPU 64 of the image generation device 10.

In S101, the user uses the client device 20 to input an instruction to select the model selection mode. In response, the selection unit 11 selects the model selection mode and starts the model selection mode.

In S102, categories of 3D models available in the model selection mode are displayed on the client device 20. These categories can be customized according to the user. Specifically, for example, if the user is a doctor, customization is possible so that categories according to the user's specialty are displayed at the top. In addition, for example, machine learning may be used to display categories based on usage history. The user selects and specifies the category to which the image they wish to obtain belongs from the displayed categories. The specification unit 12 then accepts the specification and causes the client device 20 to display 3D model items included in the specified category.

In S103, 3D model items included in the category specified by the user are displayed. The user selects a 3D model required to create an image of the state the user envisions from the displayed items. The designation unit 12 then accepts the designation. Note that the method of designating a 3D model is not limited to this, and any configuration that allows the user to designate a desired 3D model may be used. For example, the item may be designated directly from a list of items, or may be designated by searching by character input or voice input. The acquisition unit 13 then acquires the 3D model designated by the designation unit 12 from the database 30, and displays it on the client device 20.

In S104, the designation unit 12 determines whether the designation and acquisition of items is complete. When the designation and acquisition of all items required for the image of the state envisioned by the user is complete, the designation and acquisition of items is determined to be complete. Specifically, for example, when the user inputs that the designation of items is complete, or when the user performs an operation to proceed to the next process, the designation unit 12 determines that the designation and acquisition of items is complete. If it is determined that the designation and acquisition is complete (Yes), the process proceeds to S105. On the other hand, if it is not determined that the designation and acquisition is complete (No), S103 is repeated.

In S105, the change unit 14 changes the state of the 3D model based on a request for changing the state of the 3D model input by the user via the client device 20. Then, the display control unit 15 displays the 3D model in the changed state on the display screen of the client device 20.

In S106, the output unit 16 determines whether or not to place a custom order. Here, a custom order means ordering an illustrator to create an illustration based on the 3D model displayed on the display screen of the client device 20. If a custom order is selected on the client device 20, it is determined that a custom order is to be placed (Yes), and the process proceeds to S107. On the other hand, if output is selected on the client device 20, it is determined that a custom order is not to be placed (No), and the process proceeds to S108.

In S107, for example, a URL is output to the client device 20 to obtain an image including the 3D model displayed on the display screen of the client device 20, i.e., an image including the 3D model in a state modified based on the user's modification request.

Meanwhile, in S108, the output unit 16 transmits the 3D model information to the administrator device 40. Here, the 3D model information is information for displaying on the administrator device 40 an image including the 3D model in a state changed based on the user's change request, including which 3D model was selected and the coordinate information of the 3D model. By the output unit 16 transmitting the 3D model information to the administrator device 40, it becomes possible to confirm the image in the state envisioned by the user on the administrator device 40, and even when an illustrator creates an illustration individually, the consultation time required to create the illustration can be significantly shortened.

Next, specific examples of images generated in the model selection mode will be described with reference to Figures 4 to 8. Figure 4 is a diagram for explaining a first example of an image generated in the model selection mode. This diagram shows an example in which a 3D model of a human leg 111 and a 3D model of an external fixation device 112 are specified in S103. Figure 4 (A) is a diagram showing a 3D model of a human leg 111. Figure 4 (B) is a diagram showing a 3D model of an external fixation device 112. The user uses the client device 20 to combine the 3D model of the leg 111 with the 3D model of the external fixation device 112 to change it to the state he or she envisions. Figure 4 (C) is a diagram showing a 3D model in a state changed by the user. The user can use the client device 20 to change the combination, position, posture, viewpoint for the 3D model, and color of the 3D model. That is, the change unit 14 can change the combination, position, posture, viewpoint for the 3D model, and color of the 3D model based on a change request from the user. Note that the viewpoint of a 3D model here refers to the direction from which the 3D model is viewed. In this way, users can, for example, freely combine 3D models to easily obtain an image that shows the state they envision.

FIG. 5 is a diagram illustrating a second example of an image generated in the model selection mode. In this diagram, an example is shown in which a 3D model of a model 121 showing a cross section of a human body is specified in S103. This diagram includes an example of a user interface screen displayed on the client device 20. The user can use the client device 20 to change the position of the 3D model, the viewpoint of the 3D model, and the color and texture of the 3D model. The color and texture of the 3D model can be changed, for example, by operating the operation panel 124. Note that the color and texture can be changed for each part divided based on the anatomical structure. Specifically, for example, the color of only the second cervical vertebra can be changed. In this way, the user can easily obtain an image showing the state he or she envisions, such as changing the color to highlight only the desired part. In addition, the viewpoint of the 3D model can be executed, for example, by operating the viewpoint change unit 125. In addition, by clicking the "Export" icon 122 displayed on the screen, it is possible to obtain, for example, a URL for obtaining an image including the 3D model displayed on the display screen. On the other hand, you can place a custom order by clicking on the "Custom Order" icon 123 displayed on the screen.

Figure 6 is a diagram illustrating a third example of an image generated in the model selection mode. This diagram shows an example in which a 3D model of a leg 131 with its internal structure exposed is specified in S103. The user can use the client device 20 to change the position, posture, viewpoint with respect to the 3D model, color, and display/hide of each part of the 3D model. Note that the display/hide of each part can be changed for each part divided based on the anatomical structure. Specifically, as shown in this diagram, for example, the triceps surae 132, tibialis anterior 133, peroneus longus 136, and extensor digitorum longus 137 can be hidden, and the tibia 134 and fibula 135 can be displayed. In this way, the display/hide of a desired part can be easily switched, and the user can easily obtain an image showing the state he or she envisions.

Figure 7 is a diagram illustrating a fourth example of an image generated in the model selection mode. This diagram shows an example in which a 3D model of a human leg 141 including edema is specified in S103. The user can change the edema state of the leg 141 using a slider 142 displayed on the display screen of the client device 20. The leg 141 displayed on the left side of this diagram shows a relatively mild (asymptomatic) state of edema. On the other hand, the leg 141 displayed on the right side of this diagram shows a relatively severe state of edema. In this way, the user can easily obtain an image showing the state of symptoms that he or she imagines. Note that the method of changing the onset state is not limited to this, and it may be possible to change it by other methods, such as inputting a numerical value.

Figure 8 is a diagram illustrating a fifth example of an image generated in the model selection mode. This figure shows an example in which a 3D model of a retractor 151, which is a type of medical instrument, is specified in S103. This figure includes an example of a user interface screen displayed on the client device 20. The user can change the opening angle of the retractor 151, i.e., the operating state of the retractor 151, by, for example, dragging the retractor 151 displayed on the display screen of the client device 20. In this figure, the opening angle of the retractor 151 is changed to be larger, and the state in which the retractor 151 is open is shown by a broken line. Note that the method of changing the operating state of the 3D model is not limited to this, and it may be possible to change it by, for example, inputting a numerical value. In this way, the user can easily obtain an image showing the state he or she envisions by freely changing the operating state of the 3D model.

The image generated by the image generating device 10 may be not only a still image but also an animated image. Specifically, when an animation model with an animation function is specified in S103, an animation image including the animation is generated.

Alternatively, a user can input text describing the image they have in mind, and the image can be automatically generated using machine learning.

As described above, the model selection mode allows users to easily obtain images that represent the state they envision.

<Model placement mode>
The model placement mode will now be described. The model placement mode is a mode in which a 3D model is placed in a virtual space such as a virtual operating room, and an image envisioned by the user is generated. Fig. 9 is a flow diagram showing image generation processing in the model placement mode. Each operation (step) shown in this flow chart can be executed under the control of the CPU 64 of the image generating device 10. Note that in this figure, steps similar to those in Fig. 3 are given the same numbers and explanations are omitted, and only differences will be explained in detail.

In S201, the user inputs an instruction to select the model placement mode using the client device 20. The selection unit 11 then selects the model placement mode and starts the model placement mode.

After S102, in S202, 3D model items included in the category specified by the user are displayed on the display screen of the client device 20. The user specifies the 3D model required to create an image of the state he or she envisions from the displayed items. In the model placement mode, at least one virtual space is specified in S202 to place a 3D model in a virtual space. In the model placement mode, when performing line-of-sight selection, which will be described later, at least one 3D model of a human body is specified in S202. The specification unit 12 then accepts the specification. The acquisition unit 13 then acquires the 3D model specified by the specification unit 12 from the database 30, and displays it on the client device 20.

In S203, the modification unit 14 modifies the state of the 3D model based on a request for changing the state of the 3D model input by the user via the client device 20. Specifically, in the model placement mode, the modification unit 14 places the specified 3D model in the specified virtual space. Then, the display control unit 15 displays the 3D model placed in the virtual space on the display screen of the client device 20.

In S204, the modification unit 14 determines whether item placement is complete. Here, it is determined that item placement is complete when all items required for the image of the state envisioned by the user have been placed in the virtual space. Specifically, for example, the modification unit 14 determines that item placement is complete when the user inputs that item placement is complete, or when the user performs an operation to proceed to the next process. Here, if it is determined that it is complete (Yes), the process proceeds to S205. On the other hand, if it is not determined that it is complete (No), S202 to S203 are repeated.

In S205, the change unit 14 determines whether or not to perform gaze selection. Here, gaze selection refers to displaying the virtual space viewed from the viewpoint of at least one 3D model of a human body placed in the virtual space on the display screen. For example, when one of the 3D models of a human body placed in the virtual space is selected on the client device 20, the change unit 14 determines that gaze selection will be performed (Yes) and proceeds to processing at S206. On the other hand, when it is determined that gaze selection will not be performed (No), the process proceeds to S106.

In S206, the change unit 14 changes the line of sight to that of the selected 3D model of the human body, and the display control unit 15 displays the virtual space seen from the viewpoint of the selected 3D model of the human body on the display screen.

Next, specific examples of images generated in the model placement mode will be described with reference to Figures 10 and 11. Figure 10 is a diagram for explaining an example of an image generated in the model placement mode. This figure shows an example in which a virtual operating room 211 is specified as the virtual space, and a 3D model of an operating table 212, a 3D model of a doctor 213, and a 3D model of a patient 214 are specified in S202. The user uses the client device 20 to place the specified 3D models of the operating table 212, doctor 213, and patient 214 in the virtual operating room 211, changing it to the state the user envisions. This allows the user to easily obtain an image showing the space the user envisions.

Figure 11 is a diagram illustrating an example of an image generated when a line of sight is selected. Specifically, Figure 11 is an example of an image generated when changing to the line of sight of doctor 213 is selected in Figure 10. Here, because the line of sight of doctor 213 is selected, doctor 213 is not displayed on the display screen, and only the 3D models of operating table 212 and patient 214 are displayed. Note that the direction of the selected line of sight may be freely changed. By being able to generate such an image, it is possible to easily show a third party (e.g., a trainee) the line of sight from which the doctor is performing the treatment. Also, when the line of sight of patient 214 is selected, it is possible to understand the field of view from which the patient is receiving the treatment.

<Pose change mode>
The pose change mode will be described. The pose change mode is a mode in which the posture of a 3D model of a human body is changed within the structural movable range of the human body, and an image of the human body in a posture envisioned by the user is generated. Specifically, for example, by combining multiple animation models, more complex poses are possible compared to the model selection mode. FIG. 12 is a flow diagram showing an image generation process in the pose change mode. Each operation (step) shown in this flow chart can be executed under the control of the CPU 64 of the image generation device 10. Note that in this figure, steps similar to those in FIG. 3 are assigned the same numbers and explanations are omitted, and only differences are explained in detail.

In S301, the user uses the client device 20 to input an instruction to select the pose change mode. In response, the selection unit 11 selects the pose change mode and activates the pose change mode.

After S102, in S302, 3D model items included in the category specified by the user are displayed. The user selects the 3D model required to create an image of the state he or she envisions from the displayed items. In the pose change mode, at least one 3D model of a human body is specified in order to generate an image in which the posture of the 3D model of the human body is freely changed within the structural movable range of the human body. The specification unit 12 then accepts the specification. The acquisition unit 13 acquires the 3D model specified by the specification unit 12 from the database 30, and displays it on the client device 20.

After S104, in S303, the modification unit 14 modifies the state of the 3D model based on a request for modifying the state of the 3D model input by the user via the client device 20. Here, it is possible to freely modify the posture of the 3D model of the human body within the structural movable range of the human body. Then, the display control unit 15 displays the modified 3D model on the display screen of the client device 20.

Next, a specific example of an image generated in the pose change mode will be described with reference to FIG. 13. FIG. 13 is a diagram illustrating an example of an image generated in the pose change mode. In the pose change mode, at least one 3D model of a human body 311 is specified. Then, using the client device 20, the user can change the viewpoint relative to the 3D model and the posture of the 3D human body model within the structural movable range of the human body. Note that a 3D model of an animal's body may be specified instead of a human body.

As described above, the pose change mode allows users to easily obtain images of the human body in the pose they envision.

<Echo mode>
The echo mode will be described. The echo mode is a mode in which an image is generated by changing the position of a virtual ultrasound probe (hereinafter, simply referred to as a virtual probe) on an operation screen. Fig. 14 is a flow diagram showing an image generation process in the echo mode. Each operation (step) shown in this flow chart can be executed under the control of the CPU 64 of the image generating device 10. In this figure, steps similar to those in Fig. 3 are given the same numbers and their explanations are omitted, and only differences will be described in detail.

In S401, the user inputs an instruction to select echo mode using the client device 20. In response, the selection unit 11 selects echo mode and activates the echo mode.

After S102, in S402, 3D model items included in the category specified by the user are displayed. The user selects a 3D model required to create an image of the state he or she envisions from the displayed items. In echo mode, a 3D model including at least a part of the human body is specified in order to generate a virtual ultrasound image by changing the virtual probe position. In addition, since the imaging range and appearance generally differ depending on the type of ultrasound probe, the type or model of ultrasound probe may be specified here. The specification unit 12 then accepts the specification. The acquisition unit 13 then acquires the 3D model specified by the specification unit 12 from the database 30 and displays it on the client device 20.

After S104, in S403, the modification unit 14 modifies the state of the 3D model and generates an image based on a request to change the state of the 3D model input by the user via the client device 20. Specifically, in echo mode, the modification unit 14 generates a virtual ultrasound image, which is an image that would be captured if the probe were to be placed in contact with or close to that position, by changing the virtual probe position,. The display control unit 15 then displays the generated virtual ultrasound image, i.e., a virtual ultrasound image corresponding to the movement of the virtual probe, on the display screen of the client device 20.

Next, a specific example of an image generated in echo mode will be described with reference to FIG. 15. FIG. 15 is a diagram for explaining an example of an image generated in echo mode. FIG. 15(A) is a diagram for explaining a virtual probe position. In echo mode, a 3D model of at least a part of a human body (here, a human hand 411) is specified. In this diagram, for simplification, the display of the muscles and skin of the human hand 411 is omitted, and only the bones are displayed. The user uses the client device 20 to change the virtual probe position 412 on the human hand 411 to a desired position. Then, an image that will be captured at the virtual probe position 412 is generated as a virtual ultrasound image. At this time, it is also possible to display a virtual imaging range 413 at the virtual probe position 412 on the screen. FIG. 15(B) is a diagram showing the virtual imaging range 413 of the virtual probe.

In addition, an animation image that matches the operation of the virtual probe, such as changing the virtual probe position 412, may be generated as a virtual ultrasound image.

As described above, echo mode allows you to use a virtual ultrasound probe to easily obtain ultrasound images that would be captured if you placed the ultrasound probe at a position you envision.

<Endoscope mode>
The endoscope mode will be described. The endoscope mode is a mode in which an image is generated by operating a virtual endoscope on an operation screen. In the endoscope mode, the state of the 3D model can be changed so that the virtual endoscope is inserted into at least a part of the 3D model of the human body. FIG. 16 is a flow diagram showing an image generation process in the endoscope mode. Each operation (step) shown in this flow chart can be executed under the control of the CPU 64 of the image generating device 10. In this figure, the same steps as those in FIG. 3 are assigned the same numbers and their explanations are omitted, and only the differences will be described in detail.

In S501, the user inputs an instruction to select the endoscope mode using the client device 20. The selection unit 11 then selects the endoscope mode and starts the endoscope mode.

After S102, in S502, 3D model items included in the category specified by the user are displayed. The user selects a 3D model required to create an image of the state he or she envisions from the displayed items. In the endoscope mode, a 3D model including at least a part of the human body is selected in order to generate a virtual endoscope image by operating the virtual endoscope via the client device 20. In addition, since the imaging range, the part of the human body to be imaged, and the appearance generally differ depending on the type of endoscope, the type and model of endoscope may be selected here. The designation unit 12 then accepts the designation. The acquisition unit 13 then acquires the 3D model specified by the designation unit 12 from the database 30 and displays it on the client device 20.

After S104, the modification unit 14 modifies the state of the 3D model and generates an image based on a request to change the state of the 3D model input by the user via the client device 20. In the endoscope mode, first, in S503, the modification unit 14 determines the insertion position of the virtual endoscope. Specifically, the user selects the position on the 3D model of the human body where the virtual endoscope is to be inserted via the client device 20. The modification unit 14 sets the selected position as the insertion position of the virtual endoscope.

In S504, the change unit 14 changes the position of the virtual endoscope. Specifically, the user performs an operation to change the position of the virtual endoscope inserted in the 3D model of the human body via the client device 20. The change unit 14 changes the position of the virtual endoscope in response to the operation, and generates an image that the endoscope would capture at that position as a virtual endoscopic image. The display control unit 15 then displays the generated virtual endoscopic image on the display screen of the client device 20.

Next, a specific example of an image generated in the endoscope mode will be described with reference to FIG. 17. FIG. 17 is a diagram illustrating an example of an image generated in the endoscope mode. In the endoscope mode, a 3D model of at least a part of a human body (here, human body 511) is specified. Then, the user uses the client device 20 to select an insertion position 515 of the virtual endoscope 512. Then, the user uses the slider 514 to change, i.e., move, the position of the virtual endoscope 512. Then, an image that would be captured by the virtual endoscope 512 at the changed position is generated as a virtual endoscopic image 513.

In addition, an animation image that matches the operation of the virtual endoscope 512, such as changing the position of the virtual endoscope 512, may be generated as a virtual endoscope image.

As described above, according to the endoscope mode, it is possible to easily obtain, using a virtual endoscope, an endoscopic image that would be captured if the endoscope were inserted into a position that the user imagines.
<Other embodiments>
Although the embodiments of the present application have been described in detail above with reference to several drawings, the above-mentioned embodiments are merely examples for explaining the present invention, and the present invention is not limited to these embodiments. Various modifications and changes are possible within the scope of the present invention.

When the processing in the image generating device 10 is realized by a computer, the processing contents of the functions that each part of these devices should have are executed based on a program. The program describing the above-mentioned processing contents can be recorded on a computer-readable recording medium. The computer-readable recording medium may be any type, such as a magnetic recording device, an optical disk, a magneto-optical recording medium, or a semiconductor memory. The processing of each part may also be configured by executing a predetermined program on a computer. In other words, the client device 20 may execute the functions of the image generating device 10 by installing a program for executing the functions of the image generating device 10 in the client device 20.

Reference Signs List 1 Image generation system 10 Image generation device 11 Selection unit 12 Designation unit 13 Acquisition unit 14 Change unit 15 Display control unit 16 Output unit 20 Client device 30 Database 40 Administrator device 50 Network

Claims (11)

An image generating device that generates a medical image using a 3D model,
A selection unit that selects a mode based on an instruction from a user;
A designation unit that accepts designation of the 3D model from the user;
an acquisition unit that acquires at least one of the designated 3D models from a database in which a group of 3D models is stored;
a change unit that changes a state of the 3D model based on a change request for the state of the 3D model according to the selected mode;
and a display control unit that displays an image including the 3D model in a changed state on a display screen. When the first mode is selected in the selection unit,
The image generating device according to claim 1, characterized in that the modification unit is capable of modifying at least one of the color, texture, and display/non-display of each part divided based on the anatomical structure of the 3D model, the posture of the 3D model, the viewpoint relative to the 3D model, the combination of the 3D models, the motion state of the 3D model, and the onset state of a symptom. When the second mode is selected in the selection unit,
the at least one designated 3D model includes a 3D model of a human body;
the change unit is capable of placing the 3D model in a virtual space and changing a state of the 3D model to the virtual space as seen from a viewpoint of the 3D model of the human body placed in the virtual space;
The image generating device according to claim 1 , wherein the display control unit causes an image of the virtual space seen from the viewpoint to be displayed on the display screen. When the third mode is selected in the selection unit,
the at least one designated 3D model includes a 3D model of at least a portion of a human body;
The image generating device according to claim 1 , wherein the modification unit is capable of modifying a posture of the 3D model of at least a part of the human body within a structural movable range of the human body. When the fourth mode is selected in the selection unit,
the at least one designated 3D model includes a 3D model of at least a portion of a human body;
The change unit is capable of changing a position on the 3D model of at least a part of the human body to which a virtual ultrasound probe is applied,
The image generating device according to claim 1 , wherein the display control unit causes a virtual ultrasound image corresponding to the movement of the virtual ultrasound probe to be displayed on the display screen. When the fifth mode is selected in the selection unit,
the at least one designated 3D model includes a 3D model of at least a portion of a human body;
The change unit is capable of changing a state of the 3D model so as to insert a virtual endoscope into at least a part of the 3D model of the human body,
The image generating apparatus according to claim 1 , wherein the display control unit causes a virtual endoscopic image corresponding to a movement of the virtual endoscope to be displayed on the display screen. The image generating device according to claim 1, characterized in that the group of 3D models stored in the database includes medical models, veterinary models, and medical equipment and instrument models. The image generating device according to claim 1, further comprising an output unit that generates and outputs information or an image file for acquiring an image including the 3D model displayed on the display screen. A method for generating medical images using a 3D model, comprising:
Select a mode based on user instructions,
Accepting a designation of the 3D model from the user;
Obtaining at least one of the specified 3D models from a database in which a group of 3D models is stored;
changing a state of the 3D model based on a state change request for the 3D model according to the selected mode;
A method of generating an image, comprising: displaying an image including the 3D model in a modified state on a display screen. 10. A program for causing a computer to execute the image generating method according to claim 9. An image generated using the image generating device according to claim 1.