KR20240011560A - Device and method to provide medical sliced image - Google Patents

Abstract

The electronic device according to one embodiment visually outputs a target area model determined based on a medical image of the patient's target area through a display, and captures a manipulation part for screen control on the user's body using a vision sensor. , recognize the position and posture of the manipulation part with respect to the target region based on the data captured by the vision sensor, and create an indicator graphic representation indicating the position of the manipulation region in at least one of the area including the target region model and the surrounding area. A medical cross-sectional image of a plane corresponding to the posture of the manipulation part and the target point corresponding to the position of the manipulation part in the target region can be output together with the target region model and indicator graphic representation.

Description

Method and device for providing medical cross-sectional images {DEVICE AND METHOD TO PROVIDE MEDICAL SLICED IMAGE}

Hereinafter, a technology for providing medical cross-sectional images is provided.

Medical devices that provide various information during surgery to operators (e.g., doctors and nurses) participating in the surgery may be placed in the operating room. For example, in an operating room, devices capable of providing medical images, such as an Depending on the purpose of surgery, different types of main imaging devices may be installed in each operating room, and one main imaging device may be installed in one operating room. An auxiliary imaging information device that provides auxiliary information related to surgery may also be deployed along with the corresponding imaging device.

Among the medical images provided by the above-described imaging device, images containing information necessary for the operator must be provided quickly and accurately during surgery. Complex tactile manipulation may be used to select a desired medical image.

The background technology described above is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and cannot necessarily be said to be known technology disclosed to the general public before this application.

An electronic device according to one embodiment can control a pointer on a large screen using a non-contact gesture.

The electronic device according to one embodiment may control the moving speed or moving distance of the pointer based on the user's gesture.

The electronic device according to one embodiment may provide a medical cross-sectional image corresponding to a target point selected based on a hand gesture along with 3D modeling generated based on a large amount of medical image images.

The electronic device according to one embodiment may provide both selection of a cross-section corresponding to a medical cross-sectional image and spatial movement of a target point at which the medical cross-sectional image will be generated based on a gesture.

The electronic device according to one embodiment may determine the cross-sectional direction based on the direction of the gesture.

An electronic device according to one embodiment may enable a medical staff to control the screen based on a gesture at a location adjacent to the patient's surgical site.

However, technical challenges are not limited to the above-mentioned technical challenges, and other technical challenges may exist.

An electronic device according to an embodiment includes a display that visually outputs a target area model determined based on a medical image of a patient's target area, and a manipulation part for screen control on the user's body. A vision sensor that captures; Memory in which instructions executable by a computer are stored; And a processor that accesses the memory and executes the instructions, wherein the instructions determine the position and posture of the manipulation part with respect to the target part based on data captured by the vision sensor ( posture), and output an indicator graphical representation indicating the position of the manipulation part in at least one of an area including the target part model and a surrounding area, and the position of the manipulation part in the target part. It can be set to output a medical cross-sectional image of a plane corresponding to the posture of the manipulation part for the point corresponding to the target part model and the indicator graphical representation.

The instructions move the indicator graphical representation based on detecting movement of the manipulation part using the captured data, and display a medical cross-sectional image output on the display corresponding to the moved position of the manipulation part. It can be set to change to a medical cross-sectional image of the target point.

The manipulation part includes a hand, and the commands recognize the position of the hand and the palm direction of the hand, and select a target plane corresponding to the palm direction at a target point corresponding to the position of the hand among the target parts. It may be set to determine and output a medical cross-sectional image of the determined target plane through the display.

The commands determine a virtual plane created by expanding the surface of the palm perpendicular to the palm direction as the target plane, and the target plane is a medical cross-sectional image corresponding to a cross-section passing through the target area of the patient. can be set to be output through the display.

The instructions rotate the target plane based on detecting the rotation of the palm based on the captured data, and display a medical cross-sectional image output on the display as a cross-section through which the rotated target plane passes the target area. It can be set to change to a medical cross-sectional image.

The instructions can be set to determine the target plane as one of the axial plane, coronal plane, and sagittal plane, based on the palm orientation.

The commands, based on the fact that the target plane is an axial plane, suggest a possible direction and movable range of the cross-section along an axis perpendicular to the axial plane, and based on the target plane being a coronal plane, perpendicular to the coronal plane. It can be set to present the movable direction and movable range of the cross-section on the axis, and based on the fact that the target plane is the sagittal plane, the movable direction and movable range of the cross-section can be presented on the axis perpendicular to the sagittal plane. there is.

The commands output a medical cross-sectional image corresponding to the axial plane based on the palm direction being perpendicular to the axial plane with respect to the target patient, and the palm direction being perpendicular to the coronal plane with respect to the target patient. Based on this, outputting a medical cross-sectional image corresponding to the coronal plane, and based on the palm direction being perpendicular to the sagittal plane based on the target patient, can be set to output a medical cross-sectional image corresponding to the sagittal plane. there is.

The instructions determine the target plane as the axial plane based on the angle difference between the plane perpendicular to the palm direction and the axial plane being less than a critical angle, and the angle difference between the plane perpendicular to the palm direction and the coronal plane is Based on being less than a critical angle, determine the target plane as the coronal plane, and based on the angle difference between the plane perpendicular to the palm direction and the sagittal plane being less than the critical angle, determine the target plane as the sagittal plane. can be set.

The commands may be set to fix at least one of a target point or a target plane on the target area based on the manipulation part being determined to perform a predetermined gesture based on the captured data.

The commands may be set to change the medical cross-sectional image output on the display to a medical cross-sectional image from another viewpoint based on detecting movement of the manipulation part while the target point and the target plane are fixed.

The manipulation part is a hand, and the commands can be set to identify the number of fingers spread out on the hand and to change the time of the medical cross-sectional image output on the display at a speed determined based on the number of identified fingers. .

The commands may be set to increase the time change rate of the medical cross-sectional image when the number of spread fingers increases, and to decrease the time change speed of the medical cross-sectional image when the number of spread fingers decreases. .

The commands may be set to increase the spatial movement speed of the medical cross-sectional image when the number of spread fingers increases, and to decrease the spatial movement speed of the medical cross-sectional image when the number of spread fingers decreases. .

The vision sensor may include at least one of a camera sensor, a radar sensor, a LiDAR sensor, an ultrasonic sensor, or an infrared sensor.

A method implemented with a processor according to an embodiment includes visually outputting a target region model determined based on a medical image of a target region of a patient through a display; Capturing a manipulation part for screen control on the user's body using a vision sensor; Recognizing the position and posture of the manipulation part relative to the target part based on data captured by the vision sensor; outputting an indicator graphic representation indicating the location of the manipulation part in at least one of an area including the target part model and a surrounding area; And outputting a planar, medical cross-sectional image corresponding to the posture of the manipulation region for a target point corresponding to the position of the manipulation region in the target region together with the target region model and the indicator graphical representation. You can.

An electronic device according to one embodiment can prevent contamination in an operating room and provide hygienic screen control by controlling a pointer on a large screen using a non-contact gesture.

The electronic device according to one embodiment can provide convenience in pointer control on a large screen by controlling the moving speed or moving distance of the pointer based on the user's gesture.

The electronic device according to one embodiment can intuitively provide, through a display device, a medical cross-sectional image corresponding to a target point selected based on a hand gesture along with 3D modeling generated based on a large amount of medical image images.

The electronic device according to one embodiment increases cognitive efficiency in the search of medical images during surgery by providing both selection of a cross section corresponding to the medical cross-sectional image and spatial movement of the target point where the medical cross-sectional image will be generated based on a gesture. You can do it.

The electronic device according to one embodiment can conveniently provide a medical cross-sectional image of a cross-section desired by the user during surgery by determining the cross-sectional direction based on the direction of the gesture.

The electronic device according to one embodiment allows medical staff to control the screen based on gestures at a location adjacent to the patient's surgical site, thereby reducing unnecessary movement of medical staff and time wasted in searching for images.

1 shows an apparatus for providing a medical cross-sectional image according to one embodiment.
Figure 2 is a flowchart illustrating a method of providing a medical cross-sectional image according to an embodiment.
Figure 3 shows the output of a target region model based on a medical image according to one embodiment.
Figure 4 is a diagram illustrating detection of a user's manipulation part according to an embodiment.
Figure 5 shows the output of a medical cross-sectional image based on detection of a manipulation site according to one embodiment.
FIG. 6 illustrates an operation of determining a target plane representing a cross section corresponding to a medical cross-sectional image according to an embodiment.
Figure 7 illustrates an operation of moving a target point where a medical cross-sectional image is generated among target areas according to an embodiment.
Figure 8 shows movement of a target point for each target plane according to one embodiment.
FIG. 9 illustrates an operation of changing the time of a medical cross-sectional image output on a display according to an embodiment of the present invention based on a gesture.
FIG. 10 illustrates an operation of changing the magnification of spatial movement of a medical cross-sectional image output on a display according to an embodiment of the present invention based on a gesture.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific disclosed embodiments, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Terms such as first or second may be used to describe various components, but these terms should be interpreted only for the purpose of distinguishing one component from another component. For example, a first component may be named a second component, and similarly, the second component may also be named a first component.

When a component is referred to as being “connected” to another component, it should be understood that it may be directly connected or connected to the other component, but that other components may exist in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of the described features, numbers, steps, operations, components, parts, or combinations thereof, and are intended to indicate the presence of one or more other features or numbers, It should be understood that this does not exclude in advance the possibility of the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. No.

As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1 shows an apparatus for providing a medical cross-sectional image according to one embodiment.

A device for providing a medical cross-sectional image (eg, the electronic device 100) according to an embodiment may provide an image related to a patient's surgery based on the gesture of a user (eg, an operator or medical staff). A user's gesture is an input (e.g., gesture input) identified by at least one of the movement, rotation, shape, posture, or action of the user's body part (e.g., manipulation part) used for manipulation. can indicate. For example, the electronic device 100 may identify a user's gesture and control a visualized screen based on the identified gesture.

The electronic device 100 may include a vision sensor 110, a display 120, a processor 130, a memory 140, and a communication unit 150.

The vision sensor 110 can capture a manipulation part for screen control on the user's body. In this specification, the manipulation part mainly describes the user's hand, but it is not limited thereto, and other parts such as the foot may be used as the manipulation part depending on the design. For example, the vision sensor 110 may photograph scenes and/or environments within an operating room. The vision sensor 110 may be arranged such that a field of view (FOV) of the vision sensor 110 includes the operating table on which the patient is placed and the movement range of the user (eg, operator) within the operating room. The data captured by the vision sensor 110 recognizes the patient's surgical site, the location of the user's (e.g., medical staff) manipulation part (e.g., hand position), and the orientation of the manipulation part (e.g., palm direction). It can be used to

The vision sensor 110 may include at least one of a camera sensor, a radar sensor, a LiDAR sensor, an ultrasonic sensor, or an infrared sensor. For example, the camera sensor may be implemented as a stereo camera sensor and configured to measure depth. In this specification, an example in which the vision sensor 110 includes a combination of a camera sensor and a lidar sensor is mainly described, but is not limited thereto.

The display 120 may output information and images related to the patient's surgery. For example, the display 120 may visually output a target region model determined based on a medical image of the patient's target region. Medical images are images taken of a patient for medical purposes and may include X-ray images, CT (Computed Tomography) images, and MRI (Magnetic Resonance Imaging) images. The target area may represent the area of the patient's body that is the target of the procedure. In this specification, the patient's head is mainly described as an example of the target area, but it is not limited thereto. The target area model is a model created for the target area, and may be, for example, a two-dimensional model and/or a three-dimensional model corresponding to the external appearance (eg, geometric shape) of the target area. The target region model may be rendered and/or generated based on a plurality of medical images. The target region model may be generated before and/or during surgery by electronic devices and/or other devices (e.g., servers). It is not limited to this.

Additionally, the display 120 can output a medical cross-sectional image along with a target area model. As will be described later, a medical cross-sectional image may be an image representing a cross-section at a point selected by the user among the target area. The display 120 may also output an indicator graphic representation that indicates or suggests a point selected for generating a medical cross-sectional image. Accordingly, the electronic device 100 can intuitively guide to which point of the target region model the medical cross-sectional image is generated.

For reference, in FIG. 1, the vision sensor 110 is shown attached to the display 120, but it is not limited thereto. The vision sensor 110 and the display 120 may be arranged to be spaced apart from each other. The display 120 may be placed in a location that can be observed by the user during surgery.

The processor 130 may access the memory 140 to execute instructions for generating and displaying a target area model and a medical cross-sectional image. The processor 130 may acquire a plurality of medical images related to the patient's surgery. The processor 130 may generate a target region model corresponding to the plurality of medical images described above. The processor 130 is a two-dimensional medical image of a surgical patient stored in an existing hospital information system (HIS (171), Hospital Information System) and/or a medical image storage and transmission system (PACS (172), Picture Archiving and Communication System). By converting (e.g. CT image, MRI image) into 3D, a target area model can be created and saved. The processor 130 may load a 3D target area model and output it using the display 120. However, it is not limited to this, and the processor 130 may load a pre-built target region model. The processor 130 may identify the user's gesture based on data captured by the vision sensor 110. The processor 130 may determine the target point and target plane in the patient's target area (eg, body part to be operated on) based on the user's gesture. The electronic device 100 may generate and output a medical cross-sectional image in which the target area is sliced into the target plane at the determined target point. The operation of the processor 130 is described in detail in FIGS. 2 to 9 below.

The memory 140 can store instructions that can be executed by a computer. Memory 140 may temporarily and/or permanently store medical images, medical cross-sectional images, and data required for generating medical cross-sectional images.

The communication unit 150 may receive information necessary to provide medical images from the server 170. For example, the communication unit 150 may establish communication with the server 170 including the HIS 171 and/or the PACS 172. The electronic device 100 may request data that needs to be searched from the HIS 171 and/or the PACS 172 through the communication unit 150 during surgery. The electronic device 100 may request and receive data expected to be needed during the surgery from the HIS 171 and/or the PACS 172 before surgery and store it in the memory 140.

The electronic device 100 according to one embodiment may provide high-speed spatial scrolling of sections in large-capacity medical images stored in the PACS 172. The electronic device 100 may provide a spatial and/or temporal point and/or section necessary for the user based on the user's non-contact gesture. The electronic device 100 acquires a medical cross-sectional image (e.g., load) corresponding to the position of the user's (e.g., medical staff) manipulation part (e.g., hand position) on the target area (e.g., affected area) of the patient lying down. ) or creation) can be done. In other words, when the relative position of the medical staff's hand changes with respect to the affected area, for example, a reference point on the affected area, the medical cross-sectional image output from the display 120 is converted to a medical cross-sectional image corresponding to the changed position. It can be. The electronic device 100 may synchronize the spatial frame of the 3D target area model displayed on the display 120 with the position of the hand. The above-described 3D target area model can also be interpreted as a set of images (e.g., spatial frame images) corresponding to a plurality of spatial frames arranged along an arbitrary axis, and each spatial frame image is associated with the corresponding spatial frame. It may be a two-dimensional medical image of a corresponding cross-section. For reference, FIGS. 2 to 8 mainly illustrate the search for spatial frames according to movement of the manipulation part, but the present invention is not limited thereto, and temporal frames can also be searched as described in FIG. 9 .

The electronic device 100 can immediately provide a medical cross-sectional image of the immediate location of the affected area to the user by matching the positional movement of the above-described manipulation area and the change in the spatial frame of the target area model. The electronic device 100 can improve user convenience by providing intuitive navigation of medical cross-sectional images according to the movement axis along which the medical staff's palm moves. Accordingly, the electronic device 100 can intuitively search for and visualize the PACS image desired by the user in response to the user's gesture.

Figure 2 is a flowchart illustrating a method of providing a medical cross-sectional image according to an embodiment.

First, in step 210, the electronic device (e.g., the electronic device 100 of FIG. 1) may visually output a target region model determined based on a medical image of the patient's target region through a display. For example, the electronic device may acquire a plurality of medical images of a target area for a patient's surgery from a database (eg, HIS and/or PACS). The plurality of medical images regarding the target region may be medical images captured toward the target region at multiple timings and/or multiple imaging directions (eg, imaging angles). The electronic device may render and/or generate a target region model from a plurality of medical images.

And in step 220, the electronic device can capture the manipulation part for screen control on the user's body using a vision sensor. For example, a vision sensor may be placed in a location where the operating area can be captured. For example, a vision sensor can generate sensing data by capturing the user's hand. However, in this specification, examples in which the position and viewing angle of the vision sensor are static are mainly described, but are not limited thereto. As another example, the vision sensor may track the movement of the user and perform rotation and/or movement so that the user's manipulation part (eg, hand) is included within the viewing angle. Therefore, the vision sensor can continuously capture the user's manipulation area during surgery. As described above, the vision sensor can acquire depth data using a depth sensor such as a lidar sensor. Depth data may include information indicating the distance to the user's manipulation area and the patient's target area. The vision sensor can acquire image data (e.g., color image) of a scene in the operating room using a camera sensor. Image data about the scene within the operating room may include information about the shape of the user's manipulation area and the patient's target area.

Next, in step 230, the electronic device may recognize the position and posture of the manipulation part with respect to the target part based on the data captured by the vision sensor. The electronic device may identify the user's gesture based on a combination of at least two of the position, position movement, or posture of the manipulation part. The electronic device may perform an operation intended by the user (e.g., selecting a cross-section, outputting a medical cross-sectional image, moving the spatial and/or temporal frame of the medical cross-sectional image) corresponding to the identified gesture.

For example, the electronic device may determine the target area's location in the operating room (e.g., coordinates indicating the location) and The location of the manipulation site in the operating room (e.g., coordinates indicating the location) can be determined. The electronic device can determine the relative position of the manipulation area based on the target area. The electronic device may calculate relative position coordinates of the manipulation part based on a reference point among the target parts (e.g., a point corresponding to the crown of the head if the target part is the head).

Additionally, the electronic device may determine the posture of the manipulation part based on image data acquired by a camera sensor among the vision sensors. For example, the electronic device may detect the shape (eg, outline) of the target area based on image data. The electronic device may classify the posture of the target area based on the detected shape of the target area. For example, the electronic device may determine which of the anatomical planes of the target area the direction (e.g., the direction the palm is facing) of the target area is perpendicular. Anatomical planes are planes set based on the anatomical structure of the patient's body located in the operating room, and may include, for example, the axial plane, coronal plane, and sagittal plane. You can. However, the anatomical planes are not limited to this. The direction of the target area may be defined as a direction perpendicular to a plane corresponding to the target area (eg, a plane corresponding to the palm). However, it is not limited to this, and the electronic device may determine whether the plane of the target area (eg, palm plane) is parallel to which of the anatomical planes.

In the above-described example, position determination based on depth data and posture determination based on image data are shown as being performed separately for the target area, but the present invention is not limited to this. The electronic device uses at least one or a combination of two or more of the sensing data of a plurality of sensors included in the vision sensor (e.g., a camera sensor, a radar sensor, a lidar sensor, or an ultrasonic sensor, etc.) to determine the location of the target area described above. and posture can also be determined.

Furthermore, the electronic device may identify body action of the target area based on sensing data acquired by a vision sensor. For example, the electronic device may identify a body motion in which the user's hand assumes one of the following configurations: an open hand, a fist, or some fingers spread. As will be described later, when the user's hands are spread out around the patient, the electronic device can generate a medical cross-sectional image. If the user's hand assumes a fist shape, the electronic device may pause spatial and/or temporal scrolling of the medical cross-sectional image. When some of the user's fingers are spread, the electronic device may set a scroll change amount (eg, a temporal scroll change amount) in scrolling of the medical cross-sectional image.

And in step 240, the electronic device may output an indicator graphical representation indicating the location of the manipulation part in at least one of the area including the target part model and the surrounding area. For example, the electronic device can explicitly display the point corresponding to the cross section from which the medical cross-sectional image was created by overlaying and outputting an indicator graphical representation on the target region model. As another example, the electronic device may output an indicator graphical representation around the area where the target region model is output on the display. The electronic device may output a bar object corresponding to an axis perpendicular to the cross section of the medical cross-sectional image around the target area model. The bar object may represent a spatial range in which an anatomical plane can be created in the target area. The bar object is described below. The electronic device may determine the location of the graphical representation of the indicator on the bar object based on the relative position of the manipulation area relative to the uniaxial length of the target area (e.g., a length along an axis perpendicular to the anatomical plane). Accordingly, the electronic device can intuitively guide the user to the location where the medical cross-sectional image was created.

Subsequently, in step 250, the electronic device may output a medical cross-sectional image of a plane corresponding to the posture of the manipulation region for a point corresponding to the position of the manipulation region in the target region together with a target region model and an indicator graphical representation. . The electronic device may jointly visualize the medical cross-sectional image, an indicator graphical representation representing the physical location of the cross-section corresponding to the medical cross-sectional image (e.g., location within the target region), and a target region model. Therefore, the electronic device can conveniently provide the user with information necessary during surgery based on intuitive and non-contact gestures.

Figure 3 shows the output of a target region model based on a medical image according to one embodiment.

An electronic device (e.g., the electronic device 100 of FIG. 1) according to an embodiment may load (e.g., call) information (e.g., medical images 351) related to the surgery of the patient 390. The electronic device may request medical images 351 predicted to be needed in surgery from the server and store them in the internal memory in advance. If additional medical images are needed in addition to the medical images 351 stored in the internal memory, the electronic device may request additional medical images from the server. For example, the electronic device may request an additional medical image from the server based on the fact that an additional medical image called based on a user's gesture during surgery is not retrieved from the internal memory. However, it is not limited to this, and regardless of the generation of the target area model 352, which will be described later, the electronic device is connected to the corresponding point and/or cross section whenever the point and/or cross section designated by the user's gesture is changed. Information (e.g., one or more medical images) required to provide a corresponding medical cross-sectional image may be requested and received from the server each time.

Medical images 351 predicted to be necessary for surgery may be medical images 351 that capture a target area subject to surgery. The electronic device can determine the target area. For example, the electronic device may determine the target area for surgery based on the user's input and/or gesture. The user's input may be a manual input indicating a body part of the patient 390 that is the subject of surgery, and may be, for example, a query input indicating a search for images related to the body part in a database. The electronic device may receive the above-described manual input through an input device including a keyboard and mouse. The user's gesture for determining the target area may represent a gesture indicating a part of the body of the patient 390 located in the operating room. For example, the electronic device may identify a pair of manipulation parts (eg, both hands) of the user based on capture through the vision sensor 310. The electronic device can determine whether the identified pair of manipulation parts is in a shape for determining the target area (e.g., both palms are spread out facing each other). If the pair of manipulation parts matches the shape for determining the target part, the electronic device may determine the space defined by the pair of manipulation parts (for example, the space between the two hands spaced apart). The electronic device may determine the area included in the space indicated by the pair of manipulation parts as the target area. In addition to vision recognition of the inputs and gestures described above, the electronic device identifies the location of the patient 390 and the surgical site (e.g., target site) through recognition of markers placed on the patient 390 lying on the surgical bed. You may. In this specification, an example in which there is only one target area is mainly described, but the present invention is not limited to this, and multiple target areas may be selected for surgery.

The electronic device can convert the above-described medical images 351 into a target area model 352. As an example, the electronic device may generate a 3D target area model 352 based on medical images 351 stored in the internal memory. However, it is not limited to this, and the target area model 352 may be built in advance for each target area separately from the medical image of a database system such as HIS and/or PACS. The pre-built target area model 352 may be stored in a server or in the internal memory of the electronic device. Based on the target area being determined, the electronic device may load a pre-built target area model 352 corresponding to the determined target area.

The electronic device may visually output the target area model 352 through the display 320. As will be described later, by outputting an indicator graphic representation corresponding to the target point at a location determined based on the target area model 352, the electronic device can intuitively guide the user to the point representing the cross section from which the medical cross-sectional image was created.

For reference, in this specification, for convenience of explanation, an example in which the target area is one head is mainly described, but it is not limited to this. For example, the target part may be another body part of the patient 390 other than the head. When a plurality of target parts are subject to surgery, the electronic device may request and receive medical images 351 for the plurality of target parts from the server. The electronic device may generate and/or obtain a target area model 352 for each target area. When the electronic device acquires a plurality of target region models, the electronic device may switch the target region and target region model 352 to be output on the display 320 in response to the user's control (e.g., input and/or gesture). .

Figure 4 is a diagram illustrating detection of a user's manipulation part according to an embodiment.

An electronic device (e.g., the electronic device 100 of FIG. 1) according to an embodiment may detect at least one of the position, shape, or posture of the user's manipulation part 480 using the vision sensor 410. .

The electronic device can detect the user's manipulation part 480 using the vision sensor 410. For example, the electronic device may acquire sensing data through capture of the vision sensor 410. The electronic device can detect the manipulation part 480 based on sensing data. The electronic device provides location information (e.g., coordinates where the target area of the patient 490 is located) of the patient 490 in the operating room and medical staff (e.g., : The user's hand position (e.g., coordinates where the manipulation part 480 in the operating room is located) can be recognized. The electronic device determines the posture of the manipulation part 480 based on analysis of sensing data (eg, image analysis), the direction of the manipulation part 480 (eg, palm direction) and the plane of the manipulation part 480 (eg, palm). plane) can be identified. The electronic device can also identify the shape (e.g., fist shape, open shape) of the manipulation part 480 based on analysis of sensing data.

The electronic device may identify the user's gesture based on at least one or a combination of two or more of the position, shape, or posture of the manipulation part 480 described above. The electronic device may illustratively determine an output position within the display of a graphical representation of an indicator (e.g., a pointer indicating the position of a cross-section on the screen) based on the position of the medical staff's hand. Additionally, the electronic device may determine a plane on which to slice the target area at the corresponding hand position according to the palm direction. The display of the indicator graphical representation and determination of the target plane are described in Figures 5 and 6 below.

Figure 5 shows the output of a medical cross-sectional image based on detection of a manipulation site according to one embodiment.

For example, the manipulation area may include the hand 580 of a user (eg, medical staff). The electronic device can recognize the position 581 of the hand 580 and the direction of the palm of the hand 580. As described above in FIG. 4 , the electronic device may identify the position 581 and palm direction of the hand 580 relative to the target area 585 based on the sensing data of the vision sensor.

The electronic device may retrieve a medical image corresponding to the position of the user's hand 580 among the medical images of the patient 590. The electronic device displays medical information (e.g., a medical cross-sectional image 559 as a medical image) of the point where the user's hand 580 is located (e.g., target point) among the surgical areas of the patient 590 (e.g., target area 585). ) can be received by requesting it from a server (e.g., information system of a medical institution).

The electronic device may include information requested by the user's gesture among the medical information at the target point (e.g., the medical cross-sectional image 559) and/or information required to generate the requested information (e.g., the medical cross-sectional image 559). You can also select the medical images needed for creation. According to one embodiment, the electronic device may determine a target plane corresponding to the palm direction at a target point corresponding to the position 581 of the hand 580 among the target areas 585. The target point may be a point within the target area 585 (eg, affected area) of the patient 590. For example, the electronic device may retrieve medical cross-sectional images 559 in a target plane (e.g., one of the axial plane, coronal plane, and sagittal plane) from the server and/or memory. It can be collected. The electronic device may select the medical cross-sectional image 559 corresponding to the target point among the medical cross-sectional images 559 of the target plane. For another example, the electronic device may collect medical cross-sectional images 559 corresponding to the target point and select the medical cross-sectional image 559 corresponding to the target plane from among the collected medical cross-sectional images 559. The electronic device can slice the target area 585. The electronic device may load a medical image (e.g., medical cross-sectional image 559) corresponding to a sliced cross-section of the target region 585 from the server's database and/or internal memory. However, it is not limited to this, and the electronic device may generate a medical cross-sectional image 559 corresponding to a sliced cross-section of the target region 585 based on existing medical images. Based on the target plane being determined, the electronic device may retrieve or generate a medical cross-sectional image 559 corresponding to the target plane for the target point and other remaining points of the target area 585.

The electronic device may output a medical cross-sectional image 559 of the determined target plane through a display. As will be described later in FIG. 7 , when the target point is moved, the electronic device may display a medical cross-sectional image 559 corresponding to the moved target point among the collected medical cross-sectional images 559. For example, when the plane of the medical staff's palm is parallel to the axial plane, the electronic device may extract images corresponding to the axial plane. Medical cross-sectional images 559 extracted for the target plane may be divided by point within the target area 585. The medical cross-sectional image 559 for each point can also be represented as a spatial frame image. The electronic device can synchronize the spatial frame of the medical cross-sectional images 559 to points on the target area 585 . When the target point changes as the medical staff's hand 580 moves, the electronic device can immediately output a medical cross-sectional image 559 in a spatial frame corresponding to the changed target point on the display.

Additionally, the electronic device may output a bar object corresponding to the determined target plane around the target area model. The bar object may represent a spatial range in which an anatomical plane can be created in the target region 585. The bar object may be an object representing an axis corresponding to a direction perpendicular to the target plane. Accordingly, the electronic device can guide the user to spatially move the manipulation part based on the target part 585 through the bar object. The indicator graphic representation 553 may be placed at a position corresponding to the target point among the target areas in the bar object. FIG. 5 shows an example where the target plane is an axial plane, and the bar object may represent an axis (eg, x-axis) corresponding to a direction perpendicular to the axial plane.

Accordingly, the electronic device can recognize the position 581 of the hand 580 and the direction of the palm relative to the body of the patient 590 (e.g., the target area 585) at the surgical site. The electronic device maps and outputs a pointer (e.g., indicator graphic representation 553) indicating the position 581 of the hand 580 described above on the three-dimensional modeling of the display (e.g., target area model 552), It can be controlled in response to the user's gestures. The electronic device may provide an intuitive medical cross-sectional image 559 to the user by displaying the medical cross-sectional image 559 corresponding to the position of the indicator graphic representation 553.

FIG. 6 illustrates an operation of determining a target plane representing a cross section corresponding to a medical cross-sectional image according to an embodiment.

The electronic device according to one embodiment may determine a virtual plane created by expanding the palm surface perpendicular to the palm direction 682 of the user's hand 680 as the target plane 681. Palm direction 682 may indicate the direction the user's palm faces. The electronic device can determine the cross section through which the target plane 681 passes in the target area. The electronic device may acquire a medical cross-sectional image 620 corresponding to the determined cross-section. For example, if a medical cross-sectional image 620 corresponding to the cross-section determined at the target point already exists, the electronic device may load it from a server and/or memory. The electronic device may retrieve or generate a set of medical cross-sectional images (eg, a bundle of files) corresponding to the target plane 681 determined based on the posture of the manipulation part. For another example, if the electronic device cannot access the medical cross-sectional image 620 corresponding to the cross-section determined at the target point, the medical cross-sectional image 620 of the corresponding cross-section is synthesized using one or more medical images and/ Or you can create it.

The electronic device may output a medical cross-sectional image 620 corresponding to a cross-section through which the target plane 681 passes through the patient's target area through a display. The electronic device may output an indicator graphical representation 652 of a planar shape that slices the target region model together with the target region model. Accordingly, the electronic device can intuitively guide the user to the cross section sliced from the target area.

The electronic device may rotate the target plane 681 based on detecting the rotation of the palm based on the captured data. The electronic device can acquire a medical cross-sectional image 620 by slicing the target area with the rotated target plane 681. The electronic device may change the medical cross-sectional image 620 output on the display into a medical cross-sectional image 620 about a cross-section through which the rotated target plane 681 passes through the target area. Accordingly, the electronic device may also slice the target region in the axial plane, coronal plane, and sagittal plane, and further customize it by the user's gesture (e.g., palm plane). A medical cross-sectional image 620 in which the target area is sliced in a flat plane can be freely provided.

Electronic devices can intuitively control the display of medical images that can be expressed in three or more axes or planes, such as CT and MRI, in response to user gestures. Electronic devices can prevent and/or reduce contamination through contact by providing non-contact controls to the user when checking patient status prior to surgery.

Figure 7 illustrates an operation of moving a target point where a medical cross-sectional image is generated among target areas according to an embodiment.

An electronic device (eg, the electronic device 100) according to an embodiment may move the graphical representation of the indicator based on detecting the movement of the manipulation part 780 using captured data. The electronic device may change the medical cross-sectional image output on the display into a medical cross-sectional image of the target point corresponding to the moved position of the manipulation part 780. For example, as shown in FIG. 7 , the electronic device may output the first medical cross-sectional image 741 at the first target point. When the manipulation part 780 (e.g., hand) moves in the first direction (e.g., left direction in FIG. 7), the electronic device corresponds to a second target point located in the first direction with respect to the first target point. A second medical cross-sectional image 742 may be output. When the manipulation part 780 moves in a second direction opposite to the first direction (e.g., the right direction in FIG. 7), the electronic device moves to a third target point located in the second direction with respect to the first target point. The corresponding third medical cross-sectional image 743 may be output. The movement axis of the manipulation part 780 may vary depending on the determined target plane, which is explained in FIG. 8 below.

Accordingly, the electronic device can move the control pointer according to the movement of the hand and change the medical cross-sectional image corresponding to the moved position to display on the screen. Electronic devices can intuitively navigate medical short-lived images by moving the spatial frame using hand gestures.

Figure 8 shows movement of a target point for each target plane according to one embodiment.

The electronic device according to one embodiment may determine the target plane as one of the axial plane, coronal plane, and sagittal plane, based on the palm direction. The axial plane can also be referred to as a horizontal plane or transverse plane. The electronic device may determine the target plane based on the angle difference between the palm plane and each anatomical plane.

For example, the electronic device may determine the target plane as the axial plane based on the angle difference between the plane perpendicular to the palm direction and the axial plane being less than a critical angle. Based on the fact that the target plane is the axial plane, the electronic device can suggest the movable direction and movable range of the cross-section along an axis perpendicular to the axial plane. The electronic device may output a medical cross-sectional image corresponding to the axial plane based on the direction of the palm being perpendicular to the axial plane based on the target patient.

For another example, the electronic autonomy may determine the target plane to be the coronal plane based on the angle difference between the plane perpendicular to the palm direction and the coronal plane being less than a critical angle. Based on the fact that the target plane is the coronal plane, the electronic device can suggest the movable direction and movable range of the cross-section along an axis perpendicular to the coronal plane. The electronic device may output a medical cross-sectional image corresponding to the coronal plane based on the direction of the palm being perpendicular to the coronal plane with respect to the target patient.

As another example, the electronic device may determine the target plane to be the sagittal plane based on the angle difference between the plane perpendicular to the palm direction and the sagittal plane being less than a critical angle. Based on the fact that the target plane is the sagittal plane, the electronic device can present the movable direction and movable range of the cross-section along an axis perpendicular to the sagittal plane. The electronic device may output a medical cross-sectional image corresponding to the sagittal plane based on the direction of the palm being perpendicular to the sagittal plane with respect to the target patient.

Accordingly, the electronic device can display an image of a corresponding cross-section of anatomical planes on the screen of the display according to the direction of the palm. Electronic devices can output medical images such as MRI and CT on a display that displays medical information. The electronic device can select the cross-sectional direction (axial, coronal, sagittal) of medical image information according to the direction of the medical staff's palm. The electronic device can move the spatial front and back frames of the image cross-section of the medical image according to the medical staff's palm movements (e.g., left and right, forward and backward, and up and down movements) based on the selected axis and cross section.

In Figure 8, the cranio-caudal axis is shown as the y-axis, the antero-posterior axis is shown as the z-axis, and the left-right axis is shown as the x-axis. The axial plane can be expressed as the xz plane, the sagittal plane as the yz plane, and the coronal plane as the xy plane. When the target plane is the axial plane, the electronic device may output an indicator graphical representation 851 on the bar object corresponding to the y-axis perpendicular to the axial plane. The electronic device may change the target point in response to hand movement 881 (e.g., left and right movement) along the y-axis and provide a medical cross-sectional image 841 corresponding to the changed target point. When the target plane is the sagittal plane, the electronic device may output an indicator graphical representation 852 on the bar object corresponding to the x-axis perpendicular to the sagittal plane. The electronic device may change the target point in response to hand movement 882 (back and forth movement) along the x-axis and provide a medical cross-sectional image corresponding to the changed target point. When the target plane is the coronal plane, the electronic device may output a graphical representation 853 on the bar object corresponding to the z-axis perpendicular to the coronal plane. The electronic device may change the target point in response to hand movement along the z-axis (e.g., up and down) and provide a medical cross-sectional image corresponding to the changed target point.

electronic devices It is possible to compare a 3D modeling image (e.g. target area model) with the actual patient's affected area (e.g. target area) recognized through a vision sensor. The electronic device can calculate where the hand is located on the 3D modeling image. As an example, the electronic device may compare the length of the 3D modeling image with the length of the target area of the patient actually lying in the bed. The electronic device may calculate where the hand is located on the length axis of the patient's affected part based on the above-described length comparison result. The electronic device can determine the length ratio corresponding to the hand position point on the 3D modeling image. Based on the determined position, the electronic device can display on the screen a medical cross-sectional image of the point where the pointer is located in 3D modeling. Additionally, the electronic device can map the patient's surgical site, the medical staff's hand position, and the direction of the palm to 3D modeling and output it.

FIG. 9 illustrates an operation of changing the time of a medical cross-sectional image output on a display according to an embodiment of the present invention based on a gesture.

The electronic device according to one embodiment may fix at least one of a target point or a target plane on the target area based on the manipulation part being determined to perform a predetermined gesture based on the captured data. For example, as shown in FIG. 9 , the electronic device may detect a fist-clenching gesture 991 at the target point. The electronic device may output a medical cross-sectional image 941 of the target point. The electronic device may fix the spatial frame of the target point (eg, a frame corresponding to the location of the target area). The navigation bar 960 corresponding to the spatial frame may be fixed. When the electronic device detects a predetermined gesture, it may provide a search bar for the temporal frame. In Figure 9, a temporal point (eg, first time point 951) is displayed.

The electronic device may change the medical cross-sectional image output on the display to a medical cross-sectional image from a different viewpoint based on detecting movement of the manipulation part while the target point and target plane are fixed. The manipulation part may be the hand. The electronic device can identify the number of fingers spread out on the hand. The electronic device may change the time of the medical cross-sectional image output on the display at a rate determined based on the number of identified fingers. The electronic device can change the time frame from that target point to a medical cross-sectional image from another viewpoint. For example, as shown in FIG. 9, the electronic device can fix the temporal frame in the clenched fist gesture 991 even if the hand moves. The electronic device may detect finger extension gestures 992 and 993 and move the temporal frame when the hand moves.

The electronic device can increase the time change rate of the medical cross-sectional image as the number of spread fingers increases. The electronic device may reduce the time change rate of the medical cross-sectional image as the number of spread fingers decreases. As shown in FIG. 9 , when the electronic device performs a gesture 992 in which the number of spread fingers is a first number (eg, 1), the electronic device may change the temporal frame at a first speed. The electronic device may provide a medical cross-sectional image from a second viewpoint 952 that is different from the first viewpoint (e.g., a later viewpoint). If the gesture 993 includes a second number of spread fingers (e.g., 2), the electronic device may change the temporal frame at a second speed. The electronic device may provide a medical cross-sectional image at a third viewpoint 953 that is farther from the first viewpoint 951 than the second viewpoint 952 .

Electronic devices can conveniently control the moving speed or distance of a pointer using non-contact gestures on a large screen. Electronic devices can scroll large volumes of images at high speed when searching medical image data stored in a database. Electronic devices can conveniently search the image by controlling the image using non-contact gestures to find the required section and controlling the desired movement speed and distance. The electronic device may begin recognizing gestures at the user's request to activate scrolling. Electronic devices can recognize the shape of a finger through a camera. The electronic device can give weight to the displacement value for existing location information according to the shape of the hand gesture (number of fists and spread fingers). The electronic device can determine the pointer movement speed according to the determined number. As the number of fingers increases, movement speed can increase. The electronic device can recognize finger shapes and gesture movements.

The electronic device can control the movement of the pointer. The electronic device can obtain medical image information from the information system of the medical institution. For example, in the case of CT or MRI, an electronic device can configure 2D images into frames and reconstruct them into movie clips. Electronic devices can retrieve, store, and analyze medical image information on an integrated medical information screen through a network. The electronic device can control the moving speed of the video frame.

When controlling a medical image on the screen using a non-contact gesture, an electronic device can determine the playback speed according to the number of fingers spread out. When moving an electronic device left or right with an open finger, the forward and backward movement speeds may vary. The electronic device may initiate gesture recognition in response to a user's request (such as holding a mouse over or clenching and unclenching a fist). Electronic devices can recognize the shape of a finger through a camera. The electronic device may analyze the acquired hand image and determine the weight of the moving speed of the pointer. The electronic device may control the moving speed of the pointer by applying weight according to the moving distance determined by the motion gesture (e.g., gesture by the manipulation part). The electronic device can adjust the moving speed of the pointer according to the number of spread fingers and the speed of motion of the user.

Accordingly, an electronic device can quickly move to a desired section by searching large amounts of medical image data at various speeds in a temporal frame based on finger gestures. Electronic devices can provide users with cognitively intuitive and efficient convenience in controlling medical images. Electronic devices can help medical staff perform desired tasks on medical display devices with fewer movements by improving the inconvenient pointer control method when remotely controlling the pointer for content selection and control on a large medical information screen. When restoring pre-stored video data, an electronic device can quickly skip unnecessary parts or adjust the playback speed. Electronic devices can reduce video search time and provide convenience when moving to a desired video section. After skipping unnecessary images, the user can clench and unclench his/her fist to exit the image control mode when the required spatial and/or temporal section is reached. The electronic device can maintain information necessary for surgery on the screen by moving the image to a desired section and then deactivating gesture control of the movement speed of the temporal frame.

However, it is not limited to changing only the moving speed of the temporal frame depending on the number of fingers, and changing the moving speed of the spatial frame depending on the number of fingers will be described below.

FIG. 10 illustrates an operation of changing the magnification of spatial movement of a medical cross-sectional image output on a display according to an embodiment of the present invention based on a gesture.

When the electronic device according to one embodiment detects a specific gesture, it may initiate an operation to adjust the moving speed of the spatial frame. For example, if the gesture of clenching a fist is detected for less than a threshold time, the electronic device may change the movement speed in the spatial frame, which will be described later. When skipping the fist state 1091 from the palm state 1099 and proceeding to the finger state, the time for which the fist clenching gesture was maintained may be regarded as 0 seconds. For reference, when the electronic device detects a fist clenching gesture for more than a threshold time (eg, 2 seconds), the electronic device may change the movement speed in the temporal frame described above in FIG. 9. However, it is not limited to this, and the electronic device determines the entry into the movement speed change operation in the temporal frame of FIG. 9 and the entry into the movement speed change operation in the spatial frame of FIG. 10 based on the type of gesture, the maintenance time of the gesture, etc. You can choose according to your needs.

First, when the palm is open, the electronic device can determine a target point as described above with reference to FIGS. 5 to 8 and determine a target plane for slicing a body part at the target point. The electronic device may stop the rotation of the target plane described above based on detecting a gesture that switches from the palm state 1099 to the finger state. The finger state may indicate a state in which one or more fingers are spread out and four or fewer fingers are spread out. The palm state 1099 may represent a state in which all five fingers are spread out.

Afterwards, the electronic device may initiate an operation to change the moving speed of the spatial frame. The electronic device may maintain the target plane determined at the time of transition from the palm state 1099 to the finger state and move only the target point according to the movement of the hand. For example, when the number of spread fingers increases, the electronic device may increase the spatial movement speed of the medical cross-sectional image. When the number of spread fingers decreases, the electronic device may reduce the spatial movement speed of the medical cross-sectional image. For example, when the electronic device detects a state in which one of the user's fingers is spread out (1092) while providing a cross-sectional image (1041) corresponding to the axial plane of the head, the electronic device may initiate movement of the spatial frame. The electronic device may move the target point at a moving speed (e.g., 1 times the speed) corresponding to the one-open state 1092 than the palm state 1099. When the electronic device detects that there are two fingers (1093), the electronic device may move the target point at a corresponding movement speed (e.g., twice the speed). In the spatial search bar, an indicator corresponding to the target point may be moved from the first point (1061) through the second point (1062) to the third point (1063). On the other hand, the indicator graphical representation of the temporal navigation bar 1050 may be fixed. The electronic device may provide a cross-sectional image 1041 corresponding to the axial plane at the moved target point. Accordingly, the electronic device can provide the user with the function of adjusting the spatial search speed for the patient's body part depending on the number of fingers.

The electronic device may dynamically adjust the speed of movement of the spatial frame. For example, the electronic device may perform spatial search at a first speed when the first number of fingers are spread out, and may perform spatial search at a second speed when detecting a second number of fingers. If the second number is greater than the first number, the second speed may be faster than the first speed. Conversely, if the second number is smaller than the first number, the second speed may be slower than the first speed. However, it is not limited to this, and depending on the design, the number of fingers and the movement speed of spatial search may be mapped in inverse proportion.

If the electronic device detects a gesture of opening the user's palm while operating in a mode for changing the moving speed of the spatial frame, it may end the operation of changing the moving speed of the spatial frame. The electronic device may resume the operation of changing the target point and target plane described in FIGS. 1 to 8. At this time, the electronic device may initialize and/or return the target point to the point physically pointed by the user's palm. In the example shown in Figure 10, the graphical representation of the indicator on the spatial search bar moves from the first point 1061 to the third point 1063 according to the spatial search, and then moves to the first point 1061 at the end of the spatial search. A return is shown.

Accordingly, the electronic device can provide more intuitive control by providing the function of changing the movement speed in the temporal frame and/or spatial frame, and by returning the target point to the physical point indicated by the user's gesture even if the function is terminated. there is.

The embodiments described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). It may be implemented using a general-purpose computer or a special-purpose computer, such as an array, programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. A computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination, and the program instructions recorded on the medium may be specially designed and constructed for the embodiment or may be known and available to those skilled in the art of computer software. It may be possible. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

The hardware devices described above may be configured to operate as one or multiple software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (17)

In electronic devices,
A display that visually outputs a target area model determined based on medical images of the patient's target area
A vision sensor that captures a manipulation part for screen control on the user's body;
Memory in which instructions executable by a computer are stored; and
A processor that accesses the memory and executes the instructions
Including,
The above commands are:
Recognize the position and posture of the manipulation part relative to the target area based on the data captured by the vision sensor,
Outputting an indicator graphical representation indicating the location of the manipulation part in at least one of an area including the target part model and a surrounding area,
Set to output a planar, medical cross-sectional image corresponding to the posture of the manipulation region for a point corresponding to the position of the manipulation region in the target region together with the target region model and the indicator graphical representation,
Electronic devices. According to paragraph 1,
The above commands are:
moving the indicator graphical representation based on detecting movement of the manipulation portion using the captured data;
Set to change the medical cross-sectional image output on the display to a medical cross-sectional image of a target point corresponding to the moved position of the manipulation part,
Electronic devices. According to paragraph 1,
The manipulation part includes the hand,
The above commands are:
Recognize the position of the hand and the direction of the palm of the hand,
Determining a target plane corresponding to the palm direction at a target point corresponding to the position of the hand among the target areas,
Set to output a medical cross-sectional image of the determined target plane through the display,
Electronic devices. According to paragraph 3,
The above commands are:
Determine a virtual plane created by expanding the surface of the palm perpendicular to the palm direction as the target plane,
The target plane is set to output a medical cross-sectional image corresponding to a cross-section passing through the target area of the patient through the display,
Electronic devices. According to clause 4,
The above commands are:
rotating the target plane based on detecting rotation of the palm based on the captured data;
Set to change the medical cross-sectional image output on the display to a medical cross-sectional image related to a cross-section through which the rotated target plane passes through the target area,
Electronic devices. According to paragraph 3,
The above commands are:
Based on the palm orientation, set to determine the target plane as one of an axial plane, a coronal plane, and a sagittal plane,
Electronic devices. According to clause 6,
The above commands are:
Based on the fact that the target plane is an axial plane, the movable direction and movable range of the cross-section are presented along an axis perpendicular to the axial plane,
Based on the fact that the target plane is a coronal plane, the movable direction and movable range of the cross-section are presented along an axis perpendicular to the coronal plane,
Based on the target plane being the sagittal plane, set to present a movable direction and movable range of the cross-section along an axis perpendicular to the sagittal plane,
Electronic devices. According to clause 6,
The above commands are:
Based on the palm direction being perpendicular to the axial plane based on the target patient, outputting a medical cross-sectional image corresponding to the axial plane,
Based on the palm direction being perpendicular to the coronal plane based on the target patient, outputting a medical cross-sectional image corresponding to the coronal plane,
Set to output a medical cross-sectional image corresponding to the sagittal plane based on the palm direction being perpendicular to the sagittal plane with respect to the target patient,
Electronic devices. According to clause 8,
The above commands are:
Based on the angle difference between the plane perpendicular to the palm direction and the axial plane being less than a critical angle, determining the target plane as the axial plane,
Based on the angle difference between the plane perpendicular to the palm direction and the coronal plane being less than a critical angle, determining the target plane to be the coronal plane,
Set to determine the target plane as the sagittal plane, based on the angle difference between the plane perpendicular to the palm direction and the sagittal plane being less than a critical angle,
Electronic devices. According to paragraph 1,
The above commands are:
Set to fix at least one of a target point or a target plane on the target area based on the manipulation part being determined to perform a predetermined gesture based on the captured data,
Electronic devices. According to clause 10,
The above commands are:
Set to change the medical cross-sectional image output on the display to a medical cross-sectional image at a different viewpoint based on detecting movement of the manipulation part while the target point and the target plane are fixed,
Electronic devices. According to clause 11,
The manipulation part is the hand,
The above commands are:
Identifying the number of fingers spread out on the hand,
set to change the time of the medical cross-sectional image output on the display at a rate determined based on the number of identified fingers,
Electronic devices. According to clause 12,
The above commands are:
As the number of spread fingers increases, the time change rate of the medical cross-sectional image increases,
set to reduce the time change rate of the medical cross-sectional image as the number of spread fingers decreases,
Electronic devices. According to paragraph 1,
The above commands are:
As the number of spread fingers increases, the spatial movement speed of the medical cross-sectional image increases,
set to reduce the spatial movement speed of the medical cross-sectional image when the number of spread fingers decreases,
Electronic devices. According to paragraph 1,
The vision sensor is,
Containing at least one of a camera sensor, radar sensor, lidar sensor, ultrasonic sensor, or infrared sensor,
Electronic devices. In a method implemented with a processor,
Visually outputting a target region model determined based on a medical image of the patient's target region through a display;
Capturing a manipulation part for screen control on the user's body using a vision sensor;
Recognizing the position and posture of the manipulation part relative to the target part based on data captured by the vision sensor;
outputting an indicator graphic representation indicating the location of the manipulation part in at least one of an area including the target part model and a surrounding area; and
Outputting a planar, medical cross-sectional image corresponding to the posture of the manipulation region for a target point corresponding to the position of the manipulation region in the target region together with the target region model and the indicator graphical representation.
How to include . A computer-readable recording medium storing one or more computer programs including instructions for performing the method of claim 16.

Images