BRPI1101511A2 - METHOD AND COLLABORATIVE CONSCIOUS SPACE VOLUME DISPLAY SYSTEM - Google Patents

Abstract

METHOD AND COLLABORATIVE CONSCIOUS SPACE VOLUME DISPLAY SYSTEM. The present invention describes a collaborative space-conscious volume visualization method and system. A mobile display, preferably a tablet PC, or smartphone, or some type of digital pad is used, through which to view the patient's inner organs that are rendered from previously stored tomography data. Three main modules are part of the method and system implementation: volume viewer, augmented reality, and network communication. The three modules work together and allow anatomy exploration to take place directly over the real body.

Description

Patent Invention Descriptive Report

Space-Conscious Volume Visualization Method and System

Collaborative

Field of the Invention

The present invention describes a collaborative space-conscious volume visualization method and system. A mobile display device, preferably a tablet PC or smartphone, or some kind of digital pad is used to view the patient's internal organs which are rendered from previously stored tomography data. Three main modules are part of the method and system implementation: volume viewer, augmented reality, and network communication. The three modules work together and allow anatomy exploration to take place directly over the real body. The present invention is in the field 15 of Virtual Reality and Human-Computer Interaction.

Background of the Invention

Scientific Visualization

Visualization is the process that facilitates the observation of a set of 20 information. Allows the viewer to visually perceive characteristics hidden in the raw data, which may be critical for accurate analysis. A successful visualization can reduce the time required to understand underlying data, to understand their relationships, and to extract meaningful information.

“Visualization involves the human sense that has the greatest capacity

of information capture per unit of time. The sense of sight is fast and parallel, and it is even possible to pay attention to an object of special interest, without losing sight of what is happening around. ”

If the information to be viewed comes from data from scientific experiments, it is appropriate to call it J 2/30

Scientific visualization. Therefore, visualization is a tool that enables scientists to analyze, understand and communicate numerical data generated by scientific research. In recent years, humans have collected data at a rate beyond what can be studied and understood. Scientific visualization uses computer graphics to process numerical data in two and three dimensions of visual images. This visualization process includes data collection, processing, visualization, analysis and interpretation. It is revolutionizing the way scientists do science, as well as changing the way people handle large amounts of information.

Scientific three-dimensional visualization techniques are proving to be useful in a wide range of medical applications, including diagnosis, planning and simulation of surgeries, and medical training. In addition to its widespread use in medicine, volumetric visualization 15 is also used in other areas of science, for example meteorology, biology, architecture, geology, etc.

Viewing volumetric medical images is covered in this document. The basic concepts of volume visualization, as well as the most used techniques and their characteristics will be presented. The technique used at the end is presented.

Visualization of volumetric medical images

Objects created in computer graphics usually do not have information about their interior. They are developed using 3D modeling software and depend on the creativity of the modeler to construct the object 25. For the most part, these objects consist of meshes of triangles that represent only their surfaces. If it is necessary to show a character's internal structures in a game, for example, textures will need to be worked out to represent its interior.

On the other hand, the purpose of volumetric visualization is to show the interior of objects, as in many scientific applications, so traditional visualization models should be replaced by techniques developed for this purpose. The data used to allow such visualization should contain spatial information about its interior. In the case of a person, the volume must have information about the internal parts of the body, such as the organs, bones, and other relevant structures. From the information collected, specific software creates the three-dimensional model automatically.

Datasets obtained through medical imaging methods are the data source for three-dimensional reconstruction. Although medical professionals can perform diagnostics based on unprocessed data, 10 there is a strong interest in visualizing these data three-dimensionally, especially in complex organs such as the liver and its vascular system. The three-dimensional visualization method allows for faster learning for medical and anatomy students, as well as a reduction in medical error caused by the analysis and misinterpretation of two-dimensional data, which can in many cases lead to serious patient complications. This is mainly with less experienced doctors.

Over the years many techniques have been developed for rendering volumes. These techniques are divided into two classes 20 defined by Meibner and others as indirect volume rendering, which works with the extraction of surfaces represented by graphic primitives and direct volume rendering, which work by generating the image directly from the volume.

Indirect Volume Rendering Surface-based visualization techniques involve

the extraction and representation of an isosurface that is later visualized by computer graphics. Isosurface reconstruction can be done from Marching Cubes, contour traking, opaque cubes, among others. These algorithms adjust geometric shapes, such as polygons or cubes, to surface positions in volumetric datasets, the structure of interest must be identified (or segmented) in each slice for later composition of the polygon mesh. In general, these methods require decision making for each sample, and this can produce false positives (false surfaces) or false negatives (surface holes), which impair accuracy during object creation due to small samples. or low quality. This method is not suitable for surface representation of some structures (particularly in human body visualization for medical imaging).

Direct Volume Rendering Direct Volume Rendering is representing the volume.

voxels (pixel represented three-dimensionally) that are projected directly in the viewport and stored as an image, eliminating the use of geometric primitives. In this case, the volume is rendered based on a transfer function, giving the voxels 15 color and opacity values. This approach allows the interior of the volume to contain information, which can be viewed by changing values in the transfer function, allowing the display of materials with different densities. The transfer function usually has two modifiable values, which correspond to the width and center of the viewport. This functionality is very useful for exploring and analyzing information while viewing. Further details on how these functions work can be seen in the Real-time Volume Graphics book pages 81-102.

Algorithms for direct rendering of volumes have been optimized, 25 especially for graphics hardware rendering. These algorithms project the volume directly onto the image plane and are best suited for viewing volumes that represent inhomogeneous objects, such as parts of the human body. These techniques have a high computational cost because they usually involve a type of interpolation at points 30 along the viewing direction. On the other hand, they produce excellent quality images, since all voxels can be used in the construction of the images. The first model was developed in the early 80's by James Blinn. Since then, several other authors have explored modifications and optimizations from the original model.

There are four basic approaches to the direct volume rendering problem, according to Bruckner: image-order volume rendering, object-order volume rendering, hybrid-order volume rendering, and methods based on texture mapping techniques. However, they all try to solve the same problem of obtaining the (approximate) solution of the volumetric visualization integral for each pixel in the projection plane. The solution integrates attenuated colors along each radius of view between the viewer and each screen pixel.

Image Order

In this approach, viewing rays are cast from the viewer through the volume, gathering color and opacity information to apply to the painted pixel in the viewing window. This method allows you to view details inside the volume, has good transparency control by trivially removing parts hidden behind parts defined as opaque, you can view it from any direction. Ray Casting is the main order-based technique and is one of the most widely used for volumetric visualization. Ray Casting was originally proposed by Levoy. The quality of the volume created using this technique is usually very good, but as the generated medical dataset typically has a high resolution, it ends up having a big impact on performance.

Order of the obietos

Scroll through the volume and, for each voxel, find the pixels affected by your contribution. Splatting, proposed by Westover, unlike ray casting, determines how pixels are painted in the image plane, for each data sample, in other words, all the elements that make up the volume are thrown in the viewport like snowballs. , these elements have color and opacity information, which can be changed for the type of surface to be viewed. This method is faster than ray casting, but the quality of the generated volume is lower. Hybrid Order

It combines the advantages of the two previous approaches. The algorithm

The most common is Shear-warp. The purpose of Shear-warp is to simplify the projection step of the visualization pinline. This is accomplished by applying shear functions on the volume slices, transforming the data to an intermediate coordinate system. This simplifies the process of projecting slices because volumetric data is accessed in storage order. The composition and projection of slices in this new coordinate system generates a distorted intermediate image that needs to be corrected. The final image is obtained by applying a transformation called warp. This transformation, performed in 2D, restores the true 15 dimensions of the image that will be viewed by the user. In general, the performance gain of the algorithm compared to Ray casting can be 5 to 10 times.

Volumetric Rendering Based on Textures

The volumetric visualization by texture mapping comes down to the rendering of textured polygons. For visualization by this technique, the radius integral for each pixel is calculated by rendering slice-based rendering polygons. Each fragment of a polygon corresponds to the contribution of a ray segment to the volumetric rendering integral for each frame buffer pixel. This approach exploits the advancement of the processing power and flexibility of the Graphics Processing Unit (GPU). GPU-based solutions are capable of rendering volumes at interactive refresh rates (approx. 30 fps) for mid-sized datasets on standard hardware.

The pipeline for texture-based volume generation can be

It is divided into three stages: initialization, update and drawing. The first stage is to store the volumetric data set as a texture in the GPU. Data can be preprocessed, or can send information for processing at later stages. This decision influences the implementation of the design stage. 5 Pipeline updates are required when the system undergoes user intervention by changing the display parameters (change of viewpoint or projection plane, for example). The slice-based rendering approach with planes parallel to the projection plane requires that all sampling polygons be reconstructed. The drawing phase is responsible for generating the final image. At this stage, the color of each fragment of the sampling polygons is calculated and combined in the frame buffer. The final color is the texture color, corresponding to the data set space sampled by this fragment.

Methods that use GPU texture mapping are based on two approaches. The first explores the 2D texture mapping method. This method stores in 2D memory stacks of slices for each main axis of view. During rendering, the stack corresponding to the parallel axis closest to the current viewing direction is chosen and then the textured planes are rendered backwards using alpha and blending. In this approach the interpolation is bilinear in the slices and the sample rate is variable, depending on the viewing direction. The second approach uses 3D texture mapping. The texture is loaded into graphics hardware, and mapped to polygons parallel to the view plane and rendered backwards using alpha and blending. Three-dimensional textures allow the use of trilinear interpolation and achieve a constant sampling rate regardless of viewing angle. The problem with this approach is the limitation of video memory for graphics cards. This can cause a dramatic loss of performance because if the dataset does not "fit" in video memory it is subdivided, and then these blocks are rendered separately, congesting the bandwidth. One way to overcome this limitation is dataset compression. The final image quality in traditional texture mapping rendering methods is poor compared to previous methods. However, as GPU-based programmability has increased, a number of research studies have emerged with the goal of refining the final volume 5 result. One of the implementations that exploits graphics hardware and has several improvements in both the volume quality generated and rendering performance is the approach described by Engel, Kraus and Ertl in their High-Quality Pre-Integrated Volume Rendering Using Hardware-Accelerated Pixel article. Shading 2001. Here the authors develop an innovative form for texture-based volume rendering that can achieve the image quality of the Image Order, Object Order, and Hybrid Order methods even with a low-slice, low-resolution dataset. This method is suitable for graphics hardware intended for the general public. This is especially useful for volumetric effects in games and scientific volumetric visualization. The implementation uses multi-textures with advanced texture search and pixel shading operations.

Virtual and Augmented Reality Virtual Reality

The term Virtual Reality (VR) is defined by Ivan Sutherland as: "...

a simulated, computer-generated world in which the user can navigate in real time as if moving in real three-dimensional space. "The basis of RV technology emerged in the 1970s in flight simulators where an environment was created. synthetic simulator based on a real environment.

From the above, we can say that Virtual Reality is a simulation in which the computer is used to mimic (or not) the properties of a real environment. It is an environment created from code in programming language, computer graphics and three-dimensional modeling, which reacts to user actions, which are captured in countless ways. Since the goal of virtual reality is to create virtual environments with as much immersion as possible, forms of interaction that replace conventional input (mouse and keyboard) and output (monitors) forms should be adopted. To provide this sense of presence, the RV system can integrate sophisticated devices such as data gloves, stereoscopic vision goggles, biofeedback sensors, etc. Real-time interactivity is a key feature of virtual reality, it is a fundamental requirement for the user to have the feeling of immersion. This means that every action generated must have some noticeable reaction in the simulated environment almost instantly.

Paul Milgram created the continuum of virtuality, where the real world is

at the far left representing only real objects. On the right, defines environments consisting only of virtual objects, RV. The interior of the continuum is generically defined as Mixed Reality (MR), as an environment in which real world and virtual world objects 15 are presented together within the same context. Also, in the continuity of virtuality, mixed reality is subdivided into two: Augmented Reality and Augmented Virtuality.

Augmented Virtuality

A system is characterized by being in augmented virtuality (VA) when the virtual environment is the main or there is predominance of the virtual and elements of the real world are inserted in it. These inserted elements can be people, objects, and so on. Everything that is inserted into a VA system needs to be precaptured or captured in real time so that they can then be rebuilt in the virtual world where they can interact. A work conceived in augmented virtuality is Rapid Scene Modeling, Registration and Specification for Mixed Reality. The design allows real ambience scenes to be inserted into a simulated environment by video capture. In the simulated video environment, virtual avatars interact with the real scene. Augmented Reality It is a widespread term, often used in place of

Mixed reality. Augmented Reality (AR) can be defined in several ways:

According to Insley1 RA is a real-world enhancement with computer-generated text, images and virtual objects.

Milgram, 1995, says that it is the mix of real and virtual worlds at some point of continuous reality / virtuality that connects completely real environments to completely virtual environments.

Azuma's definition says that RA is a system that supplements the real world with computer-generated virtual objects, seeming to coexist in the same space and has the following properties:

• combines real and virtual objects in the real environment;

• runs interactively in real time;

• aligns real and virtual objects with each other;

• applies to all senses including hearing, touch and strength and smell.

Thus we can conclude that augmented reality allows a

enriching the real environment with virtual elements, these elements inserted in the scene are arranged in position and orientation defined by patterns captured from the real environment. This information can be anything from three-dimensional objects representing buildings to captions of an image in an art gallery.

Augmented reality requires capturing the real environment for a computerized system (computer vision), this allows the insertion of virtual objects. Usually this capture is performed by a camcorder or webcam.

The basic requirements for building an augmented reality system are tracking, registration and display.

In tracking, first of all it is necessary to capture the video frame. The frame is analyzed and patterns are detected, which can be tags or previously established information (such as contour and edge detection). If these patterns are found, the system records the position and calculates the transformation of the pattern contained in the frame relative to the virtual camera. Then, the three-dimensional object is associated with the pattern found according to the calculations of the corresponding virtual object in its correct orientation and arrangement in the viewport. Tracking, logging and display are performed 5 continuously during video capture, and are only terminated when the application is finished.

Types of RA Systems

AR can be classified in two ways, by direct and indirect vision, depending on how the user views the mixed world. Direct vision is when the user sees the mixed world pointing their eyes straight at the actual positions with the optical scene. Indirect view is when the user sees the mixed world on some device, such as a monitor or projector, not aligned with actual positions. Augmented reality with direct vision can be implemented using: Head Mounted Displays with associated handheld direct-pointing video capture camera and tablet pc, or virtual object projections in the real environment. Augmented reality with indirect vision can be achieved through the use of monitors and projections, the camera is positioned anywhere, and virtual objects are viewed by viewing on the monitor or projection.

RA tracking techniques

Two tracking techniques based on computer vision are the main ones, those based on auxiliary elements (fiducial markers / tags) for position capture and those that use elements of the actual scenario itself (markerless).

Tracing by fiducial markers (marker-based augmented realitv)

The use of fiducial tags has shown good practical results and acceptable lead times for real time applications. Tags are images that contain easy-to-extract visual characteristics. The visual characteristics of these tags must be fully visible for the successful tracking process to be carried out. If the tag is partially or totally blocked the tracking process may be compromised. These markers are images printed on plain paper, they do not require any associated electronic devices. All processing is done on the hardware where the camera is turned on, so that multiple devices running the same software can track the tag and render the related virtual object according to its position in space.

The most widely used marker-based resource for RA is the Artoolkit library. It is very popular because it provides low-cost, real-time 3D tracking solutions and because most of its features are open source, encouraging users to use, study and modify the library to suit their desired applications. An example of this is Artoolkitplus, an extended version that has many performance enhancements, modifications to tracking functions, and so on.

Artoolkit is implemented in C and C ++. It uses optical (camera) tracking, and implements computer vision techniques to identify and estimate in real time the position and orientation of a tag relative to the video capture device. Thus, the calculation of the correlation between the estimated real marker data 20 and its image makes it possible to position virtual objects aligned with the marker image.

• Markers

The tracking implemented in Artoolkit estimates the tag pose, making it possible to know the position and orientation of actual elements or actions, which are represented in the scene by markers. Pattern recognition is performed by identifying the four square region vertices contained in the video image, which is converted to a binary image (black and white), the symbol inside the vertices is compared to the markers templates registered by the user. If the information contained within the extracted rectangle is similar to any registered marker, the system will identify the marker. • Tracking

With the marker identified Artoolkit proceeds to estimate its position and orientation. The marker position and orientation are obtained through video image analysis, which establishes the relationship between the marker coordinates and the camera coordinates.

The relationship between marker coordinates and camera coordinates is accomplished through a 3x4 matrix, called a "transformation matrix".

Estimated the transformation matrix, the OpenGL API is used to adjust the virtual camera, position and draw the aligned virtual object in the actual marker view.

Without using markers (markerless auqmented realitv)

A markerless augmented reality tracking system differs from a marker-based system in that it uses a differentiated real-world method of inserting virtual objects. The approach used by this technique is not based on the use of markers, which are placed in the real world to support tracking for position and orientation calculations. In this type of tracking, any part of the actual environment can be used as a marker that can be tracked to place virtual objects. Therefore, there is no invasion of the environment caused by non-environment markers. Another advantage is the ability to extract environment features that can later be used to make tracking faster or for other purposes. The disadvantage is that tracking and recording are more complex, as they use only the captured images of the scene to retrieve the camera's pose and scene structure, without any help from previously acquired camera knowledge, requiring a high power of processing and directly affecting the frame rate of the application. Works developed in RA Several projects in RA have been and are being developed to

various areas of knowledge. Some of them are described below. Entertainment

The Black Magic Book uses augmented reality to tell the story of America's Cup in a different way. When people look through a handheld display on the pages of a normal book, animated virtual boat models jump out of the pages. The system also provides a shared experience. Each user can view the virtual model from their own point of view. The project was developed at the HIT NZ Lab using technology from Hiroshima City University, and with the support of Canterbury Development Corporation (CDC) and 10 Canterbury University.

Medicine

At the University of Northern Colorado (UNC) in Chapei Hill, uterus mapping was performed on pregnant women using ultrasound sensors, generating a 3D representation of the fetus. Three-dimensional images 15 were visualized directly on the patient's body through an HMD. Its goal is to help a doctor in the process of visualizing the fetus moving in the womb. Subsequently, an augmented reality system for tissue sewing after breast tumor removal was developed.

Construction and Maintenance Processes Columbia University researchers have developed a system

for maintaining a laser printer. This project used position trackers to obtain information to determine the position of the part of the printer where the operation is required, as well as whether or not there is any object obstructing the path of a particular action, or even whether the action has already been performed.

Collaborative Virtual Environments Overview

One of the classifications for virtual environments can be related to the number of users using a system, ranging from a single person to several people using a distributed system. Distributed virtual environments (AVDs) have high application potential. Its main features are the interaction, cooperation and sharing of computational resources in real time among users who may be geographically dispersed. Support for communication is from computer networks.

In this context, users can share the same virtual space, where they can count on each other's assistance to perform tasks based on the principles of Computer Supported Cooperative Work (CSCW). Thus the system is classified as a Collaborative Virtual Environment.

Collaborative systems are a way of sharing information and knowledge. Working collaboratively can produce better results than acting individually. In a group there may be the complementation of individual skills, knowledge and efforts, 15 and the interaction between people with complementary understandings, points of view and skills.

By collaborating, group members can early identify inconsistencies and flaws in their thinking and, together, develop ideas to help solve problems.

According to Barros 1994: “collaborating (co-labore) means working

together, it implies the concept of shared goals and an explicit intention to add something, to create something new or different through collaboration, as opposed to a simple exchange of information or instructions. ”

Collaborative work also motivates, because by observation and opinion

The participant actively works on his concepts, reasoning about them and refining them. Although advantageous, teamwork requires effort, as it requires coordination. Without it, communication will not be harnessed by being distorted or misinterpreted.

So collaboration comes down to doing a job, done by

a group of people who aim, through knowledge sharing, to reach a conclusion or an end product. Leadership is not centered on just one individual. An individual may lead a particular activity in which he is most competent, while other individuals will lead others (which require other skills).

Components of a Collaborative Virtual Environment

According to Sementille, there are four basic components for a system to be characterized as virtual and collaborative, graphic displays (HMD, Seethrough, CAVE), control devices (how the user enters information in the system, user interface), system 10 (hardware for processing information) and communication network (medium where information travels, transmission protocols, etc.). All components form a set to provide the user with immersion in the simulated environment. Graphic displays are the medium in which visual information is passed to the system user. There are 15 different categories of graphic displays, ranging from conventional video monitors to head-mounted displays. It all depends on the objectives of the proposed system. If the system aims at greater user immersion, more complex and less common devices are used. The perspective that the user has in a virtual environment (AV) is usually displayed through windows. The user of the system is always associated with some representation in the AV. The window is always the point of view of the user's representation in relation to the virtual world, which is modified as the displacement.

Communication and control devices comprise all system input and output, how the user enters actions in the system, and how the system responds to them. Common devices for data entry are the mouse and keyboard, but for greater immersion in the environment, conventional devices are the best option because they have their actions closer to those performed naturally by humans.

Visualization devices such as HMD, see-through, and CAVE, and interaction devices such as datagloves, biofeedback sensors, and more can be seen in greater detail in the Overview of augmented reality presentation and in the book Virtual Reality: Fundamentals and Applications.

The hardware used, usually a computer or a notebook, is the processing system. He is responsible for managing all components of the virtual environment (user input processing, virtual environment simulation, control of information received or sent over the network.)

The communication network is the medium where information travels, and comprises information management protocols.

Communication in Collaborative Virtual Environments Support for communication in distributed virtual environments is

based on computer networks. There are two main schemes for information exchange: Client / Server Model and Peer-to-Peer.

The client / server model implies direct communication between a client application to a server, it is not possible for a client to communicate with another client directly without server mediation. This is the simplest model for implementing network communication. One disadvantage is the possibility of server failure, which would render the entire system inoperable. To work around the problem there is the server replication technique. Another disadvantage is that the communication bottleneck can be on the server, since all packet management and relay to other clients are done there, and if there are many connections, communication latency can increase considerably.

A peer-to-peer network consists of direct communication between hosts (clients compared to the client / server model) without a dedicated server. This model is more complex because it addresses beyond local management all remote connections made on the host. All communication activity on the network is governed by protocols. Protocols define the formats and order of the information sent and received by the devices that make up the network and the actions to be taken in the transmission and reception of messages. The protocol model widely used in computer networks is TCP / IP [ROD 2001]. TCP / IP is a protocol stack (layer) model that is primarily based on a connection-oriented transport service provided by the Transmission Control Protocol (TCP) and a connection-oriented network service provided by Internet ProtocoI (IP).

Related Collaborative Projects

Several researches and works have been carried out using computer supported collaboration.

In the Distributed and Local Sensing Techniques for Face-to-Face Collaboration article, a technique was developed that allows 10 users to collaborate on Tablet PCs, computers have sensors to establish a connection between them. Each tablet is equipped with several sensors (accelerometer, touch and brightness). The system recognizes when two tablets touch, looking for data spikes in the accelerometer. Touching establishes a connection between devices and users 15 can then interact in a collaborative "face-to-face" environment. The application, which runs on both mobile computers, allows you to share information such as a sketch of a drawing or enlarge the viewing area by occupying the entire display of both devices.

TeIeInViVoTM: Towards Collaborative Volume Visualization Enviroments developed by Coleman and others, TeIeInViVoTM is an application that supports collaboration for visualization and exploration of volumetric data. This system is a redesigned extension of InViVoTM, a volume viewer that is largely focused on the medical community and has an emphasis on ultrasound diagnostics. The enhanced application allows volumes to be manipulated collaboratively. You can load a volume from a certain part of the body, share it with other connected users allowing you to interfere, select and show a specific part of the volume, change the cutting plane, or measure structures in the volume. The implementation is old but demonstrates an interest in sharing and collaborating in the analysis of volumetric data from medical images.

The Arthur project is a tool that allows architects to immediately modify architectural models. Instead of physical models, computer-generated virtual models visible through head-mounted displays are used. Users share the visualization of the three-dimensional model, a city, for example, and can see it according to its 5 position in space. The application allows to change properties of the displayed objects (scale, positioning, etc.), the modifications are immediately noticed by all who use the system.

In the patent area, some relevant documents were found, which will be described below.

US 7197170 discloses a method for viewing and

anatomical measurements. The present invention differs from this document in that it is intended for functional volumes comprising the use of a mobile display device.

US 6608628B1 discloses a method and apparatus for medical interactive virtual imaging 15 for remote multiple users. The system allows multiple users to have a collaborative view and manipulate images of a given structure. The present invention differs from this document in that it comprises collaborative space-conscious volume visualization which allows anatomical exploration to take place directly over the real body through a mobile display, and where each user decides his own perspective in real time visualization.

US 2003/0179308 discloses a system for generating an augmented reality image, comprising camcorder, sensor and augmented reality processor, for generating a virtual image 25 superimposed on the video image. The present invention differs from this document in that it can hold the camera and display device physically together and movable and can be carried into a patient's body. The present invention also does not require external sensors and can produce an augmented reality image on the spot where physical reality occurs. In addition, the present invention has network communication to enable multiple mobile display devices to share the same physical / viral environment, each with its own viewpoint and exchanging information about this environment.

From what is clear from the researched literature, no documents were found anticipating or suggesting the teachings of the present invention, so that the solution proposed here has novelty and inventive activity in the state of the art.

Summary of the Invention

In one aspect, the present invention describes a collaborative space-conscious volume visualization method and system. A mobile display 10, preferably a tablet PC, or smartphone, or some type of digital pad is used, through which to view the patient's internal organs that are rendered from previously stored tomography data. Three main modules are part of the method and system implementation: volume viewer, augmented reality, and network communication. The three 15 modules work together and allow anatomy exploration to take place directly over the real body.

An object of the present invention is a collaborative space-conscious volume visualization system comprising the modules of: a) visualization of internal organ volumes;

b) augmented reality; and

c) communication by network.

In a preferred embodiment the volume viewer utilizes a touch screen feature.

A further object of the present invention is a collaborative space-conscious volume visualization method comprising the steps of:

a) obtain a mobile display comprising internal organ volume viewer, augmented reality and network communication;

b) superimpose the moving display of a) on a patient's body.

These and other objects of the invention will be immediately appreciated.

those skilled in the art and companies with interests in the art, and will be described in sufficient detail for reproduction in the following description. Brief Description of the Figures

Figure 1 reveals the volume contour at different distances, 800x600 resolution. The pixels occupied by the volume are white. The numbers refer to table 5.

Figure 2 shows a graph that uses the data in table 5 to show the drop in the number of pixels as the volume shifts. Where P = Pixels, D = Distance, PxV = PxVoIume.

Figure 3 shows a graph comparing system performance at different screen resolutions and distance between volume and camera, where D = Distance. Core2Quad test with 256x256x128 volume resolution. It is noticed that as the distance increases the frames per second also increase, and that the difference of SPF in the two resolutions tends to increase in the same proportion.

Figure 4 shows a graph comparing system performance over

different volume resolutions. Core2Quad, where D = Distance. The distance is varied equally for the two volume resolutions. 1280x800 screen resolution.

Figure 5 shows a graph comparing system performance at different volume resolutions. TurionX2, where D = Distance. The distance is varied equally for the two volume resolutions. Fixed screen resolution at 1280x800.

Figure 6 shows a graph comparing window mode with fullscreen mode. Measurement for 1280x800 resolution performed in window mode, 1280x1024 and 1920x1080 resolutions measured in fullscreen mode, R = Pixels. Core2Quad machine used, 256x256x128 volume resolution and 45cm camera-to-camera distance.

Figure 7 reveals a performance comparison, measured in frames per second, between different computer configurations. Using the 1280x800 screen resolution, 256x256x128 volume resolution and the 45cm camera-to-camera distance. Figure 8 shows a comparison of latency, measured in milliseconds, L = Latency (ms), for loading a 256x256x128 resolution volume. 1280x800 screen resolution.

Figure 9 shows a performance comparison in a normal use mode with different volume resolutions and distances, where D = Distance. Active cutting plane. At 45cm distance the volume is cut. At 80cm distance the volume is fully rendered.

Detailed Description of the Invention

The examples shown here are for the sole purpose of exemplifying a

of the numerous ways of carrying out the invention, however, without limiting the scope thereof.

Collaborative Space-Conscious Volume Visualization System

An object of the present invention is a collaborative space-conscious volume visualization system comprising the modules of:

a) visualization of volumes of internal organs;

b) augmented reality; and

c) communication by network.

In a preferred embodiment the volume viewer utilizes a touch screen feature.

Volume Viewer

Volumes are generated from medical imaging, volume datasets used with the system come from MRI or CT, but other types of datasets available or to be developed in the future 25 could also be used. It can be developed based on the modified High-Quality Pre-Integrated Volume Rendering algorithm to make touch-screen interaction available on mobile display devices such as tablet PCs.

Augmented Reality In the augmented reality module the generated volume is associated with a

fiducial marker. The marker is tracked by a camcorder allowing the captured scene to be inserted into the render window, and the volume of medical images to be virtually inserted into the scene according to the position and angle of the viewer using the viewing display.

Network Communication Network communication allows the exchange of information between

observers in the virtual environment enabling collaboration between participants. The information exchanged can be about points of interest and density window (domain) configuration parameters for viewing. The first allows to insert markings directly on the volume, 10 for example to indicate tumors or anatomical landmarks, and the second allows to change the density of the volume materials that will receive color or be transparent. Both features allow you to highlight some specific structure in the rendered volume.

Collaborative Space-Conscious Volumes Visualization Method A further object of the present invention is a visualization method

of collaborative space-conscious volumes comprising the steps of:

a) obtain a mobile display comprising internal organ volume viewer, augmented reality and network communication;

b) superimpose the moving display of a) on a patient's body.

20

Example 1. Preferred Realization

System Evaluation

This section proposes a quantitative method for system evaluation. The test aims to gauge system performance on 25 different devices, screen resolutions and volume resolutions (scalability). Application performance is measured in frames per second (FPS), the standard for benchmarking graphics applications, which means the number of frames that the device records, processes, or displays every second. A refresh rate that generates smooth, continuous transactions for 30 interactive computer graphics applications is 30 fps. At or above this refresh rate, you have the perceived continuity of the information being displayed in the render window. Below that, the continuity of the scene is lost causing frustration to the system user. The frame rate per second is directly proportional to the processing capacity of the hardware used. A machine with high processing power (cpu and gpu) 5 can usually achieve a good average frame rate per second. Hardware Experiment Setup

For the tests, three distinct common hardware profiles were selected. For ease of machine naming we will use the 10 nomenclature available in tables 1 and 2. Core2Quad and PentiumD machines are desktops, while TurionX2 and Core2Duo are notebooks. TurionX2 works in tablet PC mode. They have a performance scale ranging from the most powerful to the least powerful, in the order Core2Quad, Core2Duo, TurionX2 and PentiumD. One of the limitations of the system 15 developed is the inability to function on very old graphics cards, or that do not support programmable shaders, so other available PC tablets could not be used.

Desktop Core2Quad Pentium D Configuration Intel Core2Quad Q9550 2.83GHz Processor Intel Pentium D 2.83GHz Memory 4GB ddr 3 1GB ddr Nvidia GTS250 Graphics Card 512MB ddr3 Nvidia 7100 512MB ddr2 Notebook Core2Duo TurionX2 Configuration Intel Core2Duo T7250 Tur 2X Mobile Processor 2, 2GHz Mobile 2 , 20 GHz Memory 2GB ddr 2 4GB ddr2 Nvidia 8400gs Graphics Card 128 ddr3 ATI Radeon 3200 (Mem. Comp) Table 1 and 2. Configuration of the machines used in the experiments. The TurionX2 machine in the notebook category is a Tablet PC and has shared video memory. Volume

In the experiments, a conventional CT dataset was used, comprising the lower chest, considered from the base of the lungs to the kidneys. The volume is entitled "Volumel" and has its properties shown in table 3.

Nomenclature Configuration Volume 1 Resolution 128x128x68 256x256x128 Slices 44 Slice distances 0.36 cm Table 3: Specifications of the volume used in the hopes. The three values in the resolution column refer to the three-dimensional resolution of the volume.

í Screen Resolutions and Effective Volume Pixels over Certain Distances The screen resolutions used are shown in table 4.

The number of pixels painted in the viewport that are generated by volume voxels vary according to the distance between the actual patient and the mobile display. The shorter the distance, the more painted pixels 15. As this amount influences performance, some fixed distances were chosen for the tests. Table 5 illustrates what has been said.

Resolutions Number of Pixels 1 800x600 480000 2 1280x800 1024000 3 1280x1024 1310720 4 1920x1080 2073600 Table 4: System viewport resolutions used during the experiments.

Distances Total of Painted Pixels Pixels 1.1 23 480000 462600 1.2 1024000 999067 2.1 45 480000 135141 2.2 1024000 298520 3.1 80 480000 50702 3.2 1024000 96266 4.1 115 480000 24603 4.2 1024000 52360 5.1 150 480000 14874 5.2 1024000 31320 Table 5: Specification of distances (cm) used in the experiments. The distance is calculated between the volume marker and the video capture camera. This table also shows at each distance the number of pixels the volume occupies in the viewport for resolutions of 800x600 (in lines x.1) and 1280x800 (in lines x.2).

Figure 1 visually shows the decrease in voxel-generated pixels of volume as distance increases.

Video Capture Camera The camera used for testing was the Playstation Eye.

(PS3 Eye) developed by Sony. It has the ability to capture video at a higher resolution and refresh rate compared to standard webcams of similar price. Their technical specifications are described in table 6.

PS3 Eye Resolution 320x240 640x480 FPS 120 (max) 60 (max) USB connection Table 6. PS3 Eye camera specifications.

Test Protocol

We first set some distances between the volume and the camcorder, they allow us to find out if there is a performance drop as the number of pixels derived from the volume increases in the render window. It was also decided that the PS3 Eye camera would be employed in all tests performed and that its acquisition parameters would be: 640x480 resolution at 30 fps. In order to be able to analyze the data for the developed tests, it was thought of a structure for storing the necessary information (logs). The system created has been modified to support this functionality.

In a Yog fragment generated after applying one of the tests, the first line of the generated file serves to identify that this is a log file. After a default header is automatically written, the information entered refers to the machine used for the test, camera and system settings. At the end of the writing phase of the header, begins to capture the information that will be used for comparison. The information collected during this phase is:

· FPS: Marks frames per second during system execution.

• ACTIVE BRAND: Informs if the volume is being rendered (visible mark) during information capture.

• FULLSCREEN: Informs if the preview window is full screen.

• CUTTING PLANE: Informs if a cutting plane is active.

· CHANGING WINDOWS: Reports latency in milliseconds caused by the last change of viewports.

• CREATED: Informs if the viewport was changed at that moment. Results Obtained

Here are presented and discussed the data obtained in the realization of

experiments.

Performance over effective volume pixels

In this experiment the system performance is analyzed according to the number of pixels painted by the volume in the preview window. The purpose of the test is to find out if the amount of pixels generated by the volume interferes with system performance. For the test, the resolutions 1 (800x600) and 2 (1280x800) of table 4, the Core2Quad machine and the volume specified in table 3 with the highest resolution (256x256x128) were used. The test consists of measuring the application SPF at different distances. The 30 distance specifications and their relationship to the number of pixels painted in the viewport can be seen in table 5, Figure 1, and the graph.

Figure 2. The closer the volume is to the camcorder, the more pixels of the volume are painted. The graph in Figure 3 illustrates the results obtained after data tabulation.

The shader used in the system is implemented so that volume voxels 5 are not the impact factor on performance. The information contained in the graph in Figure 3 shows that the area of pixels belonging to the volume during rendering is the factor that effectively influences application performance. Note that at lower resolution the amount of painted pixels relative to the volume has a similar proportion to the higher resolution (similar amount of pixels that the volume occupies in relation to the viewport).

Performance at different volume resolutions

The experiment aims to find out if performance drops when using the same volume at different resolutions. The volume used was as specified in table 3. The screen resolution was kept fixed (1280x800) in all measurements made (resolution 2 of table 4) and the machines used were Core2Quad and TurionX2. Figures 4 and 5 show the graphs.

It is noticeable that system overhead by increasing the number of voxels for volume creation does not significantly influence system performance. Both using a volume with a larger number of voxels and using the one with less quantity. The values in the graphs at each distance are not inverted in any situation, this means that the higher resolution of the volume does not tend to outperform the lower resolution, but at most it is equalized. Percentage means the difference in performance between resolutions at each distance.

Performance measured in full screen

The full screen test consists of gauging the performance of the developed system with a fixed resolution for the render window (1280x800), and with a variation of screen resolution (using different monitors). When the system is set to full screen a larger amount of pixels are used to paint the same part of the image, thus À 29/30

fitting the rendered image to the full size of the screen. Both monitors used for full screen have the resolutions 3 (1280x1024) and 4 (1920x1080) of table 4. If we think that more pixels are used to represent the same part of an image to fit the total screen size 5, we can sooner conclude that there will be no performance difference as the viewport resolution will remain fixed at 1280x800. For the experiment was used, in addition to the parameters mentioned above, the Core2Quad machine and the volume in the highest resolution. Figure 6 shows the graph with the obtained results. It is noticed that, contrary to what was thought, according to the resolution

Fullscreen increases the application's FPS decreases. This is because even maintaining a fixed resolution in the application, when running at full screen a larger number of pixels need to be painted on the screen (83,585 more pixels at 1280x1024 and 305,983 more pixels at 1920x1080), which is why the loss is caused. of performance.

Performance and latency in different hardware configurations

Here the application performance was measured in different hardware configurations (tables 1 and 2). A fixed screen resolution (1280x800) was maintained, using the volume at the highest resolution with a distance of 45cm from the camera. The configuration of this test is considered an extreme situation (the one that stresses the most hardware), so worst-case performance will be evaluated.

The first graph (Figure 7) shows the performance of each hardware while running the tests. Note that the only machine that can achieve proper FPS is Core2Quad, which has the best hardware configuration.

The second graph (Figure 8) shows the speed for volume generation in milliseconds, the worst performing is the worst hardware (PentiumD). The other 3 have a 5ms difference from each other, from worst to best hardware. Which is considered good compared to the difference in SPF attained in the other graph. Latency is measured in milliseconds and means 30/30

i

time for volume generation when the volume preview window changes.

Performance in normal use on a tablet PC

The purpose of the test is to measure application performance under a normal use condition, medium case, maintaining an average resolution setting (800x600) with the lowest quality volume and active cutting plane. The parameters used maintain good rendering quality, the visual difference is barely noticeable compared to the higher parameters used in other tests. Figure 9 shows the results obtained.

The active clipping plane means that as you bring the marker closer to the camera, the associated volume is cut allowing its internal structure to be viewed. The voxels behind the section plane are ignored during the rendering process and therefore the FPS increases.

It is believed that with the advancement of mobile hardware technology, new

generations of tablet PCs, digital pads, and smartphones will enable even better performance, which will greatly enhance the interest and practical applicability of the invention presented herein.

Those skilled in the art will appreciate the knowledge presented herein and may reproduce the invention in the embodiments presented and in other embodiments within the scope of the appended claims.

Claims (10)

1. Collaborative Space-Conscious Volume Visualization Method and System Collaborative space-conscious volume visualization system characterized by comprising the modules of: a) visualization of internal organ volumes; b) augmented reality; and c) network communication. System according to Claim 1, characterized in that the volume display uses a touch screen feature. System according to claim 1 or 2, characterized in that the volumes are generated from medical images. System according to any one of claims 1 to 3, characterized in that the volume datasets used in the volume display module are from MRI or TC. System according to any one of claims 1 to 4, characterized in that the volume display module is developed based on the High-Quality Pre-Integrated Volume Rendering algorithm. System according to any one of claims 1 to 5, characterized in that the augmented reality module associates the generated volume with a fiducial marker tracked by a camera. System according to any one of claims 1 to 6, characterized in that the augmented reality module renders images obtained by the volume module superimposed on camera images according to the position and angle of the observer with respect to a marker or the characteristics of the tracked environment (unmarked). System according to any one of claims 1 to 7, characterized in that the network communication allows the exchange of information between observers in the virtual environment, allowing a collaboration between the participants. System according to any one of claims 1 to 8, characterized in that the network communication includes the ability to exchange information about points of interest and / or density window (domain) configuration parameters for viewing. 10. Collaborative space-conscious volume visualization method comprising the steps of: a) obtaining a mobile display comprising internal organ volume viewer, augmented reality and network communication; b) superimpose the movable display of a) on a patient's body.