KR20230012820A - Electronic device for realization of virtual reality of medical environment - Google Patents

Abstract

According to various embodiments, an electronic device comprises a processor which generates a virtual medical image with respect to a virtual model including a stack of a plurality of medical images between a virtual generator and the virtual director by using a virtual medical image device including the virtual generator and the virtual direction in a virtual space. The processor may be configured to: generate a first virtual medical image based on an array of a plurality of cells of a first size for each of the plurality of medical images; and generate a second virtual medical image based on an array of a plurality of cells of a second size, smaller than the first size, for each of the plurality of medical images. According to the present invention, the electronic device can improve user convenience through an integrated system.

Description

Electronic device for realizing virtual reality in a medical environment

Various embodiments below relate to an electronic device for realizing virtual reality in a medical environment, for example, an electronic device for surgery/procedure education and/or simulation for pre-surgery/procedure.

In order to perform surgery/procedure under a radiation-guided environment in a virtual reality space, a matching technology for implementing a virtual X-ray device similar to an actual X-ray device and using the virtual X-ray device to support surgery/procedure is being developed. For example, US Patent Application Publication No. US 2018/0049622 A1 discloses systems and methods for sensory enhancement in medical procedures.

According to various embodiments, it is possible to provide surgery/procedure education and/or simulation for pre-surgery/procedure according to various cases.

An electronic device according to various embodiments may use a virtual medical imaging device including a virtual generator and a virtual detector in a virtual space to generate virtual images for a virtual model including a stack of a plurality of medical images between the virtual generator and the virtual detector. A processor generating a medical image, wherein the processor generates a first virtual medical image based on an array of a plurality of cells having a first size, respectively, of the plurality of medical images; It may be configured to generate a second virtual medical image based on an array of a plurality of cells having a second size smaller than the first size of .

In an embodiment, the processor may be configured to simultaneously generate the first virtual medical image and the second virtual medical image.

In an embodiment, the processor may be configured to overlap the second virtual medical image with the first virtual medical image.

In one embodiment, the processor may be configured to determine the first size based on a user's input.

In an embodiment, the processor may include a first vector between the virtual generator and the detector, a second vector between the virtual generator and first corner points of each of the plurality of medical images, and the plurality of medical images. The virtual medical image may be generated using each normal vector.

In an embodiment, the processor may include a third vector between a second corner point different from a first corner point of each of the plurality of medical images and an arbitrary point excluding corner points of each of the plurality of medical images; and The normal vector may be calculated using a fourth vector between a third corner point different from the first corner point and the second corner point of each of the plurality of medical images and the arbitrary point.

In an embodiment, the processor is configured to calculate a ratio of a dot product of the first vector and the normal vector, and a dot product of the second vector and the normal vector, wherein the processor: It may be configured to multiply a direction toward a point by the ratio, and calculate a contact point where each of the plurality of medical images and a virtual line along the direction meet.

In one embodiment, the processor connects a first intersection point of the contact point on a first line connecting the first corner point and the second corner point, and the second corner point and the third corner point. and calculate a second intersection point of the contact point on a second line that comprises: a first ratio of a length of the first line to a length between the first corner point and the first intersection point; and Generation of the virtual medical image may be terminated when a second ratio of a length of a second line to a length between the third corner point and the second intersection point is greater than 1.

In an embodiment, the processor multiplies the first ratio and the second ratio by sizes in a first direction and sizes in a second direction of the plurality of medical images, respectively, and absorbance of each of the plurality of medical images. It can be configured to calculate

In an embodiment, the processor stacks a plurality of deformed medical images by integrating data for a virtual marker configured to be positioned on the virtual model and data for an anatomical structure included in each of the plurality of medical images. and generate the virtual medical image based on the plurality of modified medical images.

In one embodiment, the processor may determine at least one of a positional relationship between the virtual marker and the virtual model, a coupling structure between the virtual marker and the virtual model, and a size relationship between the virtual marker and the virtual model. A stack of the plurality of deformed medical images may be generated by using the image.

An electronic device according to various embodiments may provide surgery/procedure education and/or simulation for pre-surgery/procedure according to various cases.

An electronic device according to various embodiments may improve user convenience through an integrated system.

An electronic device according to various embodiments may improve matching accuracy of objects implemented in virtual/augmented/mixed reality.

Effects of the electronic device according to various embodiments are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a block diagram of an electronic device according to various embodiments.
2 is a flowchart illustrating a routine for generating a virtual medical image in a virtual space of an electronic device according to an exemplary embodiment.
3 is a diagram of a virtual space implemented by an electronic device according to an embodiment.
4A to 4D are diagrams for explaining a method of generating a virtual medical image in a virtual space according to an exemplary embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes can be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents or substitutes to the embodiments are included within the scope of rights.

Terms used in the examples are used for descriptive purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

In addition, in the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the subject matter of the embodiment, the detailed description will be omitted.

In addition, in describing the components of the embodiment, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, order, or order of the corresponding component is not limited by the term. When an element is described as being “connected,” “coupled to,” or “connected” to another element, that element may be directly connected or connected to the other element, but there may be another element between the elements. It should be understood that may be "connected", "coupled" or "connected".

Components included in one embodiment and components having common functions will be described using the same names in other embodiments. Unless stated to the contrary, descriptions described in one embodiment may be applied to other embodiments, and detailed descriptions will be omitted to the extent of overlap.

Referring to FIG. 1 , an electronic device 10 according to various embodiments may include a processor 110, a memory 120, an input module 130, a display 140, and a communication module 150. In the electronic device 10 according to some embodiments, at least one of the above-listed components may be omitted or one or more other components may be added.

The processor 110 may execute a program to control at least one other component of the electronic device 10 connected to the processor 110 and perform various data processing or calculations. The processor 110 may store commands or data received from at least one other component in the memory 120, process the commands or data stored in the memory 120, and store resultant data in the memory 120. there is. In an embodiment, the processor 110 may be configured to read a medical image received from the outside of the electronic device 10 (eg, a user). In an embodiment, the processor 110 implements a virtual imaging device (eg, the imaging device 330) in a virtual space (eg, the virtual space 311), and uses the virtual imaging device to generate a plurality of medical images. A virtual medical image of a stack of (eg, medical images 421) or a virtual model thereof (eg, the model 321) may be generated.

The memory 120 may store various data used by at least one component (eg, the processor 110) of the electronic device 10. For example, the memory 120 may store at least one medical image received from the outside (eg, a user) and may store intermediate or result data of the medical image processed by the processor 110 .

The input module 130 may receive commands or data to be used by the processor 110 from outside the electronic device 10 . For example, the input module 130 may receive at least one medical image or a stack of a plurality of medical images received from a user or an external electronic device.

The display 140 may visually provide information to the outside of the electronic device 10 . For example, the display 140 may display a 2D or 3D virtual model generated by the processor 110 based on result data of a medical image read by the processor 110 .

The communication module 150 may secure a communication channel between the electronic device 10 and an external electronic device and support communication through the secured communication channel. For example, the communication module 150 may relay communication between an external electronic device and the memory 120 and/or the processor 110 .

Referring to FIG. 2 , in an electronic device (eg, electronic device 10) according to an embodiment, a plurality of stacks of a plurality of medical images (eg, medical images 421) are processed by different routines. It may include a processor (eg, the processor 110) generating two (eg, two) virtual medical images. For example, the processor performs an operation 210 of loading a stack of a plurality of medical images, and loads a plurality of cells of a first size (eg, 128 x 128 pixels or voxels) for the stack of the plurality of medical images. s) generating a first virtual medical image of the array (221) and displaying the first virtual medical image on a virtual display (eg, the virtual display 340) in the virtual space (222). Routine 220, and operation 231 of generating a second virtual medical image of an array of a plurality of cells (eg, 512 x 512 pixels or voxels) of a second size smaller than the first size, and A second routine 230 including an operation 232 of displaying a second virtual medical image on the display may be performed.

In one embodiment, the processor may perform the first routine 220 and the second routine 230 substantially simultaneously. For example, while the processor performs the first routine 220 to generate a first virtual medical image from a medical image having an array of 128 x 128 pixels or voxels, a plurality of pixels or While calculation of a parameter (eg, absorbance) of the array of voxels is repeated at least 16,384 times, the processor performs a second routine 230 to generate a second virtual medical treatment from a medical image having an array of 512 x 512 pixels or voxels. While generating an image, calculation of a parameter (eg, absorbance) of an array of a plurality of pixels or voxels of the second virtual medical image may be repeated at least 262,144 times. If each routine for a stack of at least 300 medical images is performed by the processor, the difference between the processing time by the first routine 220 and the processing time by the second routine 230 is proportional to the number of medical images. can increase exponentially. Simultaneous execution of the first routine 220 and the second routine 230 by the processor relatively quickly provides a first virtual medical image with accuracy suitable for provision to the user, while performing any arbitrary action of the user (e.g. : virtual procedure/surgery simulation), the accuracy of the first virtual medical image may be supplemented by providing a second virtual medical image with relatively high accuracy.

In an embodiment, the processor may perform an operation 240 of overlapping the first virtual medical image generated by the first routine 220 and the second virtual medical image generated by the second routine 230. there is. In some embodiments, the processor may overlap the first virtual medical image generated by the first routine 220 with the second virtual medical image generated by the second routine 230 .

Referring to FIG. 2 , an electronic device (eg, electronic device 10) according to an embodiment displays a virtual imaging device 330 for generating a virtual medical image for a virtual model 321 and the virtual medical image. It may include a processor (eg, the processor 110) that implements the virtual space 311 including the virtual display 340 to be displayed.

In one embodiment, the virtual imaging device 330 includes, for example, a virtual generator 331, a virtual detector 332, and a C-shaped virtual arm connecting the virtual generator 331 and the virtual detector 332 ( 333) may be included.

In an embodiment, the virtual model 321 may be a stack of a plurality of medical images or may be generated based on a plurality of medical images. In one embodiment, the virtual model 321 may be combined with the virtual marker 322. In some embodiments, data of a plurality of medical images (eg, DICOM-type medical images) constituting the virtual model 321 and data of the virtual marker 322 are integrated into one data (eg, modified medical image). can be implemented as For example, a coupling relationship between an anatomical structure constituting the model 321 and a structure of the marker 322, at least some of the anatomical structures constituting the model 321 (eg, a portion where the marker 322 is installed) ) and the marker 322, the size relationship between the size of the anatomical structure constituting the model 321 and the size of the marker 322, etc. may be implemented in a consistent manner within one data. Implementing the virtual model 321 and the virtual marker 322 in the virtual space 311 using one integrated data is the data of a plurality of medical images constituting the virtual model 321 and the virtual marker 322 It is possible to provide a more consistent result than a result in which the data of is individually implemented in the virtual space 311 . In one embodiment, the virtual model 321 may be combined with the virtual medical tool 323. In some embodiments, data of a plurality of medical images constituting the virtual model 321 and data of the virtual medical tool 323 may be integrated and implemented as one data (eg, modified medical image). In some embodiments, data of a plurality of medical images constituting the virtual model 321, data of the virtual marker 322, and data of the virtual medical tool 323 may be implemented as one integrated data. In one embodiment, virtual model 321 , virtual marker 322 and/or virtual medical tool 323 may be positioned between virtual generator 331 and virtual detector 332 .

Referring to FIGS. 4A to 4D , an electronic device (eg, the electronic device 10) according to an exemplary embodiment includes a virtual generator 431 (eg, the virtual space 311) in a virtual space 411 (eg, the virtual space 311). A virtual model (eg, the virtual model 321) between the virtual generator 331) and the virtual detector 432 (eg, the virtual detector 332) or a virtual medical image of a stack of a plurality of medical images 421 is generated. It may include a processor (eg, the processor 110) that generates it. The position of the virtual generator 431, the position of the virtual detector 432, and the position of an arbitrary point (or cell) on the array of a plurality of cells of the plurality of medical images 421 are determined in advance within the virtual space 411. It can be expressed as a vector according to the determined absolute coordinate system.

Referring to FIG. 4A , first, the processor may set targets 4321 having a predetermined size (eg, targets having a resolution size set by a user) on the virtual detector 432 . For example, the processor may set an array of 128 x 128 targets 4321 on the virtual detector 432 to generate a virtual medical image of 128 x 128 resolution.

Referring to FIG. 4B , the processor obtains a position vector A obtained by subtracting the position vector v1 of the virtual generator 431 from the position vector v2 of the target 4321 of the virtual detector 432, and a stack of a plurality of medical images 421. The position vector B obtained by subtracting the position vector v1 of the virtual generator 431 from the position vector v4 of the first corner point 4212 of one medical image 421 and the normal vector C of the medical image 421 may be calculated. In an embodiment, the processor uses the position vector v5 of the second corner point 4213 of the medical image 421 and the position vector v6 of the third corner point 4214 of the medical image 421 (eg, v5 and cross product of v6) to calculate the normal vector C.

Referring to FIGS. 4B and 4C , the processor may calculate a first inner product ratio according to the dot product of the position vector A and the position vector C and a second inner product ratio according to the inner product of the position vector B and the position vector C. can Thereafter, the processor may calculate the point 4211 where the plane of the medical image 421 and the line according to the position vector A intersect by multiplying the first dot product ratio and the second dot product ratio by the position vector A, respectively. Meanwhile, when the first dot product ratio and the second dot product ratio calculated by the processor are negative numbers, since the direction of the virtual generator 431 is in the opposite direction, the first dot product ratio and the second dot product ratio are obtained only when the first dot product ratio and the second dot product ratio are positive numbers. 2 dot product ratios can be used in the above calculations.

Referring to FIG. 4D , the processor calculates the position vector v7 of the first intersection point 4215 of the projection of the position vector v3 on the line of the position vector v4 by subtracting the position vector v4 from the position vector v5, and the position at the position vector v6. The position vector v8 of the second intersection point 4216 of the projection of the position vector v3 on the line of the position vector obtained by subtracting the vector v4 can be calculated respectively. In one embodiment, the processor, when a value obtained by dividing the size of the position vector (v5 - v4) by the size of the position vector (v7 - v4) is greater than 1 and the size of the position vector (v6 - v4) is the position vector (v8 - When the value divided by the size of v4) is greater than 1, the image parameter (eg: absorbance), and the calculation of imaging parameters for that target 4321 can be finished. In an embodiment, the processor may calculate a value of one pixel of the target 4321 by dividing the sum of image parameters (eg, absorbance) calculated as above by the number of medical images 421 . In an embodiment, the processor may generate a virtual medical image by repeating the above operation as many times as the number of targets 4321 for the plurality of targets 4321 on the detector 432 .

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may store program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (11)

A processor generating a virtual medical image for a virtual model including a stack of a plurality of medical images between the virtual generator and the virtual detector using a virtual medical imaging device including a virtual generator and a virtual detector in a virtual space; ,
The processor generates a first virtual medical image based on an array of cells having a first size of each of the plurality of medical images, and generates a second virtual medical image of a second size smaller than the first size of each of the plurality of medical images. An electronic device configured to generate a second virtual medical image based on an array of a plurality of cells. According to claim 1,
The processor is configured to simultaneously generate the first virtual medical image and the second virtual medical image. According to claim 1,
The electronic device configured to overlap the second virtual medical image with the first virtual medical image. According to claim 1,
The processor is configured to determine the first size based on a user's input. According to claim 1,
The processor generates a first vector between the virtual generator and the detector, a second vector between the virtual generator and first corner points of each of the plurality of medical images, and a normal vector of each of the plurality of medical images. An electronic device configured to generate the virtual medical image using According to claim 5,
The processor determines a second corner point different from the first corner point of each of the plurality of medical images and a third vector between arbitrary points excluding corner points of each of the plurality of medical images, and the plurality of medical images. An electronic device configured to calculate the normal vector using a fourth vector between a first corner point and a third corner point different from each of the second corner points and the arbitrary point. According to claim 5,
the processor is configured to calculate a ratio of a dot product of the first vector and the normal vector, and a dot product of the second vector and the normal vector;
The processor is configured to multiply a direction from the virtual generator toward the arbitrary point by the ratio, and calculate a contact point where each of the plurality of medical images and a virtual line along the direction meet. According to claim 7,
The processor determines a first intersection point of the contact point on a first line connecting the first corner point and the second corner point, and a second line connecting the second corner point and the third corner point. configured to calculate a second intersection point of the contact point;
The processor determines a first ratio of the length of the first line between the first corner point and the first intersection point and the length of the second line between the third corner point and the second intersection point. An electronic device configured to terminate generation of the virtual medical image when the second ratio of the length is greater than 1. According to claim 8,
The processor is configured to multiply the first ratio and the second ratio by sizes in a first direction and sizes in a second direction of the plurality of medical images, respectively, and calculate absorbance of each of the plurality of medical images. Device. According to claim 1,
The processor generates a stack of a plurality of deformed medical images by integrating data on a virtual marker configured to be positioned on the virtual model and data on an anatomical structure included in each of the plurality of medical images, and the plurality of modified medical images are stacked. An electronic device configured to generate the virtual medical image based on two modified medical images. According to claim 10,
The processor may use at least one of a positional relationship between the virtual marker and the virtual model, a coupling structure between the virtual marker and the virtual model, and a size relationship between the virtual marker and the virtual model to determine the plurality of locations. An electronic device that creates a stack of deformed medical images.

Images