TWM569887U - Augmented reality system - Google Patents

Abstract

An augmented reality system includes a photographic module, an object generation module, and an image processing module. The photography module generates a first image, and the first image includes a mark. The object generation module generates a second image, and the second image includes a virtual object. The image processing module signal is connected to the photographic module and the object generating module, and the image processing module receives the first image and the second image and synthesizes the first image and the second image to generate an augmented reality image, and augments the reality The markers in the image are set to correspond to the control of the virtual object.

Description

Augmented reality system

This creation is about a technique for augmenting reality, and in particular with respect to an augmented reality system.

Most of the existing telemedicine services establish a long-distance connection between the medical end and the patient through video. Such telemedicine services require the installation of photographic equipment and communication equipment in medical institutions, and the patient terminal also needs to be equipped with audio-visual playback devices and communication devices, and the communication devices of the medical institutions and the patient terminals are connected through the network. In the case of telemedicine services, the photographic equipment of the medical institution will immediately take care of the medical staff. At this time, the medical staff can explain the patient's condition or any medical condition, and the video taken by the photographic equipment (including image and sound signals) It will be transmitted to the patient through the communication device and played out instantly by the patient's audio-visual playback device, and the patient can instantly understand the condition by the medical staff's video. Of course, patients and medical staff can also interact through these devices, and medical staff can immediately consult the patient to understand the current condition of the patient and achieve the effect of remote diagnosis.

Generally speaking, in the course of video, medical staff can explain with medical images. For example, a medical professional can match an X-ray picture or an ultrasound picture previously taken by a patient to explain. The aforementioned medical images can be presented through a playback device (such as a computer screen) on the scene of the medical institution, or on a printed physical paper.

According to the prior art, when performing telemedicine services, medical personnel can be explained with medical images, but the medical images are presented through the playing equipment on the scene of the medical institution or presented on the printed physical paper, and the presentation manner is relatively rigid. It is difficult for the medical staff to explain or understand the patient.

In view of this, the purpose of this creation is to propose an augmented reality system, in order to present medical images in a flexible manner when performing telemedicine services, so that medical staff can be more clear in the explanation and also make patients more Easy to understand.

In an embodiment of the present invention, an augmented reality system includes a photography module, an object generation module, and an image processing module. The photography module generates a first image, and the first image includes a mark. The object generation module generates a second image, and the second image includes a virtual object. The image processing module signal is connected to the photographic module and the object generating module, and the image processing module receives the first image and the second image and synthesizes the first image and the second image to generate an augmented reality image, and augments the reality The markers in the image are set to correspond to the control of the virtual object.

In an embodiment of the present invention, the image processing module analyzes and obtains the coordinates of the mark and the virtual object in the augmented reality image to determine whether the mark and the virtual object in the augmented reality image are positioned with each other.

In an embodiment of the present invention, when the mark in the augmented reality image and the virtual object are positioned relative to each other, the image processing module moves, scales, or rotates the virtual object according to the dynamic of the mark.

In an embodiment of the present invention, the image processing module analyzes the dynamics of the mark to obtain a coordinate change value, and the image processing module moves, scales, or rotates the virtual object according to the coordinate change value.

In an embodiment of the present creation, the virtual object is a three-dimensional organ model or a two-dimensional organ image.

In an embodiment of the present invention, the augmented reality system further includes a display module, wherein the display module signal is connected to the image processing module, and the display module displays the augmented reality image.

In an embodiment of the present invention, the augmented reality system further includes a remote communication module, wherein the remote communication module signal is connected to the image processing module, and the remote communication module transmits the augmented reality image to the client. end.

In an embodiment of the present invention, the augmented reality system further includes a diagnostic database, wherein the diagnostic database signal connects the object generating module, and the object generating module generates the second image according to the human medical image in the diagnostic database.

In an embodiment of the present invention, the human medical image includes a computed tomography image file, an ultrasound image file, or an X-ray image file.

In an embodiment of the present creation, a plurality of marks are marked.

In summary, with the augmented reality system of the embodiments of the present invention, when the medical staff and the patient perform the telemedicine service, the medical staff can present the medical image in a flexible manner, and thus, the medical care The staff can be more clear in the explanation and make the patient easier to understand.

The detailed features and advantages of the present invention are described in detail below in the embodiments, which are sufficient to enable anyone skilled in the art to understand the technical contents of the present invention and to implement the present invention, and to disclose the contents, patent claims, and drawings according to the present specification. Anyone familiar with the relevant art can easily understand the purpose and advantages of this creation.

Please refer to FIG. 1. FIG. 1 is a block diagram of an augmented reality system 100 according to an embodiment of the present invention. In the present embodiment, the augmented reality system 100 is a medical device and system that utilizes Augmented Reality (AR) technology, which can be used for telemedicine services, but is not limited thereto. In different embodiments, the augmented reality system 100 can also be used for on-site outpatient services of a medical institution, that is, the physician can also explain the condition to the patient in conjunction with the augmented reality system 100 during the on-site clinic consultation. Moreover, whether it is a telemedicine service or an on-site outpatient service, the augmented reality image (as described later) of the augmented reality system 100 can make it easier for the medical staff to explain the patient's condition and make the patient easier to understand. Your own medical condition. In various embodiments, the augmented reality system 100 can also be used in many fields other than telemedicine or on-site clinics, such as surgical education training, instrument assembly education training, etc.; alternatively, the augmented reality system 100 can also be applied. The entertainment industry, such as the virtual three-dimensional object operation of the game and the interaction with virtual three-dimensional objects in the game.

As shown in FIG. 1 , in the embodiment, the augmented reality system 100 includes a camera module 110 , an object generating module 120 and an image processing module 130 , and the image processing module 130 is connected to the camera module 110 and the object. A module 120 is generated. The photography module 110 can be disposed at a suitable location in a medical institution. The camera module 110 is, for example, a combination of a camera and a microphone, which can be used to capture images of medical personnel and receive sounds from the scene. The object generation module 120 and the image processing module 130 can be integrated, for example, integrated into a computer device, and the camera module 110 is connected to the computer device through a wire.

Please refer to FIG. 2 at the same time. FIG. 2 is a schematic diagram showing the synthesis of the augmented reality image 330 of the augmented reality system 100 according to an embodiment of the present invention. As shown in FIG. 1 and FIG. 2, the photographic module 110 captures a medical staff located at a medical institution's site, and correspondingly generates a first image 310. The first image 310 presents an image of the photographed medical staff (hereinafter referred to as Narrator image 311). Moreover, the object generation module 120 generates a second image 320 according to a specific operation, and the second image 320 includes a virtual object 321 . For example, when the medical staff performs the telemedicine service, it is considered that the specific affected part of the patient needs to be explained. Therefore, it is necessary to generate the image of the specific affected part through the object generating module 120, so as to be explained with the virtual image of the specific affected part. . The medical staff can operate the interface of the computer device to select the relevant information of the patient to create the virtual object 321 and generate the second image 320.

As shown in FIG. 2, in the embodiment, the virtual object 321 in the second image 320 is a specific affected part of the patient, and the virtual object 321 is, for example, a three-dimensional model image of the heart of the patient. The three-dimensional modeling image of the patient's heart can be established according to the prior art. For example, during the previous medical treatment, the patient has performed a chest scan through the computerized tomography equipment of the medical institution, and the scanning process establishes a series of two-dimensional image files. Then, the heart part of the 2D image file is separated by the prior art, and then the 3D model image of the patient's heart is established by two-dimensional conversion to a three-dimensional image processing program.

As shown in FIG. 1 and FIG. 2, in this embodiment, the image processing module 130 receives the first image 310 and the second image 320, and further synthesizes the first image 310 and the second image 320 to generate an amplification. Reality image 330. In some embodiments, the augmented reality image 330 can be presented through the display device of the medical institution and the patient end, so that both the medical staff and the patient can see the augmented reality image 330 during the telemedicine process. . In the augmented reality image 330, the virtual object 321 is presented in front of the narrator image 311, so that the medical staff can intuitively and conveniently explain the virtual object 321 in the augmented reality image 330.

As shown in FIG. 1 and FIG. 2, in some embodiments, in order to increase the flexibility of the medical staff during the explanation, and the interaction between the medical staff and the virtual object 321 in the augmented reality image 330, the medical staff The virtual object 321 can be further manipulated or controlled in the augmented reality image 330.

As shown in FIG. 2, in the embodiment, the first image 310 may include the mark 313, and after the first image 310 and the second image 320 are combined into the augmented reality image 330, the mark 313 of the first image 310 is Correspondingly, it is presented in the augmented reality image 330, and the marker 313 in the augmented reality image 330 is set to correspond to the control virtual object 321 . For example, the virtual object 321 in the augmented reality image 330 can follow the movement of the marker 313.

As shown in FIG. 2, in the present embodiment, the mark 313 is plural. In the first image 310 and the augmented reality image 330, the marker 313 presents a circular image and is located at the fingertip of the narrator image 311, respectively. For example, the mark 313 can separate the fingertips of the narrator image 311 in the first image 310 by the image processing method of the prior art, and stack the circular image on the fingertip of the narrator image 311. . In the above embodiment, the medical staff of the explanation does not need to wear gloves or an additional marking device, and only needs to be photographed through the camera, and the soft and hard body of the image processing can find the fingertip of the medical staff in the narrator image 311 in real time. To get the marker 313 and analyze its gesture and hand trajectory. In some embodiments, the above embodiments may also be used with an infrared depth sensor to track the gestures and trajectories of medical personnel, increasing overall reliability. In some embodiments, the above embodiments may also be used with a miniature laser locator to detect the gestures and hand trajectories of the healthcare professional. In some embodiments, the above embodiments may also be used with artificial intelligence to track their gestures and hand trajectories. In some embodiments, the medical staff can wear a special glove that is provided with a light-emitting element at the fingertip of the finger, so that the first image 310 or the augmentation can be more easily recognized during image processing. The position of the fingertip of the narrator image 311 in the image 330. Alternatively, the glove may be provided with a retroreflective element at the position of the fingertip of the finger, and under the illumination of the infrared ray, the reflective component will reflect the infrared ray in the image, thereby easily recognizing the fingertip of the finger during image processing. position.

For example, the image processing module 130 continuously and immediately analyzes the augmented reality image 330, and obtains the coordinates of the mark 313 and the virtual object 321 in the augmented reality image 330 to determine the augmented reality image. Whether the mark 313 in the 330 and the virtual object 321 are positioned to each other. The coordinates of the mark 313 may be the coordinates of the center of the circular image, and the coordinates of the virtual object 321 are the complex coordinates of the outline of the virtual object 321 . For example, when the marker 313 in the augmented reality image 330 contacts the outline of the virtual object 321 , that is, the coordinates of the marker 313 in the augmented reality image 330 overlap with any of the coordinates of the outline of the virtual object 321 , which The indication mark 313 and the virtual object 321 are positioned to each other. When the mark 313 and the virtual object 321 in the augmented reality image 330 are positioned with each other, the virtual object 321 in the augmented reality image 330 performs a corresponding action according to the dynamics of the mark 313, for example, moving, Zoom or rotate, etc.

For example, in the augmented reality image 330, the medical staff can grasp the virtual object 321 with the fingers of both hands, and move the virtual object 321 through the up, down, left and right movement of the two hands, the opening and merging of the two hands, or the rotation of the two hands. The augmented reality image 330 is followed by movement, scaling or rotation. That is, as shown in FIG. 2, when the left and right groups of markers 313 of the narrator image 311 in the augmented reality image 330 contact the left and right sides of the virtual object 321 (ie, the marker 313 and the virtual object 321) Positioning each other), at this time, with the dynamics of the left and right groups of markers 313, the virtual object 321 will move, scale or rotate accordingly. In some embodiments, the image processing module 130 analyzes the dynamics of the marker 313 to obtain a coordinate change value, and the image processing module 130 moves, scales, or rotates the virtual object 321 according to the coordinate change value. For example, when the mark 313 is at the first coordinate at the first time point and at the second coordinate at the second time point, the virtual object 321 is at the first time point and the second time point, along with the mark The movement of the first coordinate to the second coordinate of 313 is followed by movement, that is, the movement of the virtual object 321 corresponds to the coordinate change value of the first coordinate and the second coordinate.

In the present embodiment, the virtual object 321 is a three-dimensional organ model (eg, a three-dimensional modeled image of the heart). In other embodiments, the virtual object 321 can also be a two-dimensional organ image (eg, a two-dimensional X-ray image of the chest). In the present embodiment, the virtual object 321 is a virtual human object such as a limb, a bone, or the aforementioned organ. In other embodiments, the virtual object 321 is not limited to a virtual human object, such as in a device assembly education training application, the virtual object 321 may be a device that requires training. Also for example, in an entertainment application, the virtual object 321 can be a character or item in a video game.

Please refer to FIG. 3. FIG. 3 is a block diagram of an augmented reality system 100' according to another embodiment of the present invention, wherein the elements of the same reference number of the augmented reality system 100' and their functions can be referred to the foregoing augmentation. Environment system 100. As shown in FIG. 3, in the embodiment, the augmented reality system 100' further includes a display module 140, wherein the display module 140 is connected to the image processing module 130, and the display module 140 can display the augmentation Horizon image 330. For example, the display module 140 is a display screen located near the photographic module 110, and the display screen is connected to the computer device having the object generation module 120 and the image processing module 130 by wires. When the medical staff is being photographed by the photography module 110 and performing the telemedicine service, the medical staff can immediately see the self (ie, the narrator image 311) and the virtual object in the augmented reality image 330 from the display module 140. The virtual object 321 of the augmented reality image 330 is operated and interpreted according to the visual feedback of the display module 140.

As shown in FIG. 3, in the embodiment, the augmented reality system 100' further includes a remote communication module 150, wherein the remote communication module 150 is connected to the image processing module 130, and the remote communication module The augmented reality image 330 can be transmitted to the client 200. The remote communication module 150 is, for example, a 3G/4G communication module or an internet communication module, and the image processing module 130 can synchronously transmit the generated augmented reality image 330 through the remote communication module 150. To the client 200. Moreover, the display device (not shown) of the client 200 can display the augmented reality image 330 so that the patient located at the client 200 can immediately see the augmented reality image 330.

As shown in FIG. 3, in the present embodiment, the augmented reality system 100' further includes a diagnostic database 160, wherein the diagnostic database 160 is connected to the object generating module 120. The diagnostic database 160 can store all diagnostic data for all patients in the hospital, including diagnostic medical images of the patient, including computed tomography images, ultrasound image files, or X-ray images. That is to say, any medical and medical images obtained by the medical staff using the computerized tomography device, the ultrasonic device or the X-ray device for the examination of the patient can be stored in the diagnostic database 160 when the patient is visiting the medical institution. The object generation module 120 can generate the second image 320 according to the human medical image in the diagnostic database 160. For example, the object generation module 120 can generate the second image according to the chest computed tomography image file of the patient in the diagnostic database 160. 320, and the second image 320 includes a three-dimensional modeled image of the patient's heart.

In some embodiments, the diagnostic database 160 may be in a server located in the cloud or in a computer device located at the local end such as a clinic. In some embodiments, the object generation module 120, the image processing module 130, and the diagnostic database 160 can be integrated into the same computer device.

Please refer to FIG. 4. FIG. 4 is a schematic diagram showing the state of use of the augmented reality system 100' of FIG. As shown in FIG. 3 and FIG. 4, in the embodiment, the object generation module 120, the image processing module 130, the remote communication module 150, and the diagnostic database 160 of the augmented reality system 100' are integrated in the computer device. 400. In this embodiment, there are two display modules 140, but are not limited thereto. Moreover, the computer device 400, one of the display modules 140 and the camera module 110 are disposed in the medical institution, and the display module 140 and the camera module 110 are connected to the computer device 400 in a wired manner, and the other display is The module 140 can be placed at the location of the client 200 that receives the telemedicine service. In the medical institution, the display module 140 and the camera module 110 are arranged to allow the medical staff performing the remote medical service to face the display module 140 and the camera module 110. For example, the medical staff (located in the upper part of FIG. 4) can be photographed by the camera module 110 to generate the narrator image 311 shown in FIG. 2, and the display module 140 can display the augmented reality image 330 and let the medical care The person views the augmented reality image 330 through the display module 140. Another display module 140 located at the client 200 can connect to the computer device 400 via the Internet. Moreover, through the display module 140 located at the client 200, the relevant person located at the client 200, such as the patient (located in the lower part of FIG. 4), can simultaneously view the augmented reality image 330.

In summary, with the augmented reality system of the embodiments of the present invention, when the medical staff and the patient perform the telemedicine service, the medical staff can display the augmented reality image in a flexible manner and utilize the same. Amplify the virtual objects in the real-life image to explain, so that the medical staff can be more clear in the explanation, and make it easier for the patient to understand their medical condition.

Although the technical content of the present invention has been disclosed in the above preferred embodiments, it is not intended to limit the present invention. Anyone who is familiar with the art, and some modifications and refinements that do not depart from the spirit of the present invention should be included in the creation. Therefore, the scope of protection of this creation is subject to the definition of the scope of the patent application.

100, 100'‧‧‧Augmented Reality System
110‧‧‧Photography module
120‧‧‧ Object Generation Module
130‧‧‧Image Processing Module
140‧‧‧ display module
150‧‧‧Remote communication module
160‧‧‧Diagnostic database
200‧‧‧Client
310‧‧‧ first image
311‧‧‧ narrator image
313‧‧‧ mark
320‧‧‧Second image
321‧‧‧Virtual objects
330‧‧‧Augmented Reality Image
400‧‧‧Computer equipment

1 is a block diagram showing an augmented reality system according to an embodiment of the present invention; [FIG. 2] is a schematic view showing a synthesis of an augmented reality image of an augmented reality system according to an embodiment of the present invention; [Fig. 3] is a block diagram showing an augmented reality system of another embodiment of the present invention; and [Fig. 4] is a view showing a state of use of the augmented reality system of Fig. 3.

Claims (10)

An augmented reality system, comprising: a camera module, generating a first image, the first image comprising a mark; an object generating module, generating a second image, the second image comprising a virtual object; An image processing module, the signal is connected to the camera module and the object generating module, and the image processing module receives the first image and the second image and synthesizes the first image and the second image to generate an image The real image is augmented, and the mark in the augmented reality image is set to correspondingly control the virtual object. The augmented reality system of claim 1, wherein the image processing module analyzes and obtains coordinates of the mark in the augmented reality image and coordinates of the virtual object to determine the amplification. The mark in the real image is positioned with the virtual object. The augmented reality system of claim 2, wherein when the mark in the augmented reality image and the virtual object are positioned relative to each other, the image processing module moves and scales according to the dynamic of the mark. Or rotate the virtual object. The augmented reality system of claim 3, wherein the image processing module analyzes the dynamics of the mark to obtain a target change value, and the image processing module moves, zooms, or according to the coordinate change value. Rotate the virtual object. The augmented reality system of claim 1, wherein the virtual object is a three-dimensional organ model or a two-dimensional organ image. The augmented reality system of claim 1, further comprising a display module, wherein the display module signal is connected to the image processing module, and the display module displays the augmented reality image. The augmented reality system of claim 1, further comprising a remote communication module, wherein the remote communication module signal is connected to the image processing module, and the remote communication module The augmented reality image is transmitted to a client. The augmented reality system of claim 1, further comprising a diagnostic database, wherein the diagnostic database signal is connected to the object generating module, and the object generating module is based on the diagnostic database A human medical image produces the second image. The augmented reality system of claim 8, wherein the human medical image comprises a computed tomography image file, an ultrasound image file or an X-ray image file. The augmented reality system of claim 1, wherein the plurality of markers are plural.