Detailed Description
The technical scheme of the invention is further specifically described below through examples and with reference to the accompanying drawings. In the specification, the same or similar reference numerals denote the same or similar components. The following description of embodiments of the present invention with reference to the accompanying drawings is intended to illustrate the general inventive concept and should not be taken as limiting the invention.
In order to be able to precisely and quickly display the surface of the head Magnetic Resonance Imaging (MRI) image at the interactive interface to more precisely select key points for registration and/or to be able to intuitively display the positional relationship between the detection means (e.g. a treatment flap) and the patient's head, we therefore provide a method of generating a head surface rendering file.
The term "head" as used herein includes the scalp and face. Wherein the term "scalp" can be understood broadly as soft tissue overlaying the skull bone. The term "face" is to be understood broadly as all exposed skin in front of the head.
The term volume data as used herein is to be understood broadly as three-dimensional image data, which is used to describe a certain signal intensity distribution in a cuboid region of three-dimensional space, for example, an image obtained by magnetic resonance scanning is volume data.
The term "voxel" as used herein is similar to a pixel in a two-dimensional image and can be interpreted broadly as the smallest unit of digital data across a three-dimensional spatial partition.
The term "coronal plane" as used herein is to be understood broadly as a plane passing through the human body along the vertical and horizontal axes, dividing the human body into front and rear portions. The term "axial plane" is to be understood in a broad sense as a plane normal to the height of the body, which divides the body into an upper and a lower part. The term "sagittal plane" is to be understood broadly as a plane through the human body along the vertical axis and the longitudinal axis, which divides the human body into left and right parts.
Referring to fig. 1, a method of generating a head surface drawing file according to an embodiment of the present invention is shown. The generating method comprises the following steps:
step S1, a head magnetic resonance imaging image is preprocessed to obtain a head surface mask;
step S2, generating a head surface polygonal grid representation image by the head surface mask so as to obtain a vertex data file of a polygonal primitive in the head surface polygonal grid representation image;
and step S3, obtaining the vertex normal vector file of the polygon primitive according to the head polygon mesh representation image.
In one example, the head magnetic resonance imaging image includes a head T1 weighted image, a head T2 weighted image, a head diffusion weighted image, and the like. Preferably, the head magnetic resonance imaging image is a head T1 weighted image.
In one example, since high resolution T1 weighted images generated by magnetic resonance scanners are typically corrupted by susceptibility artifacts and radio frequency field inhomogeneities, it is desirable to normalize the head T1 weighted images prior to image preprocessing to reduce the effects of susceptibility artifacts and RF field inhomogeneities on the image. The normalized head T1 weighted image includes scale normalization and gray scale normalization. The scale normalization includes normalizing the three-dimensional size and spatial resolution of the weighted image of the head T1. The scale normalization can be performed first and then the gray scale normalization can be performed as needed by a person skilled in the art. Of course, the gradation normalization may be performed first, and then the scale normalization may be performed, so long as the three-dimensional size, the spatial resolution, and the gradation range of the image can be adjusted to a predetermined size.
In one example, the patient's head T1 weighted image is first loaded into memory, and the data format may be dicom, nifti, mgz or other format. The three-dimensional dimensions of the raw head MRI data are then normalized to, for example, 256 x 256, although those skilled in the art may also normalize the three-dimensional dimensions to other dimensions, provided that it is possible to achieve dimensional normalization of different scan data of the same patient or different patients to the same dimensions in each of the three dimensions.
In one example, after normalizing the three-dimensional size of the original head MRI image, the spatial resolution of the head MRI image may also be normalized to, for example, 1mm×1mm (i.e., the size of each voxel is 1mm×1 mm), it will be appreciated by those skilled in the art that the size of the voxel size in the image determines the magnitude of the spatial resolution, i.e., the smaller the size of the voxel, the higher the spatial resolution; and vice versa. The spatial resolution of the image can be adjusted by a person skilled in the art according to the actual needs.
In one example, after normalizing the spatial resolution of the head MRI image, gray scale normalization is also required, i.e., the gray scale value of each voxel is normalized to be in the range of 0 to 255. The head MRI image processed by scale normalization and gray scale normalization can adjust the three-dimensional size, the spatial resolution and the gray scale range of the original head T1 weighted image to a preset size so as to reduce the complexity of the processing of the subsequent steps, further improve the robustness of the whole processing flow, reduce and even eliminate the influence of susceptibility artifacts and RF field non-uniformity on the image, and further improve the accuracy of the processed image in the subsequent steps.
After the normalization of the image, the center position of the entire head in the weighted image of the head T1 may also be calculated, and the position of the head in the entire volume data may be adjusted according to the center position, so that the center position of the head is located at the center position of the volume data.
In one example, a method of obtaining and adjusting a head center position includes the steps of:
and counting according to the gray values of all voxels in the weighted image of the head T1 to obtain average coordinates of all voxels with gray values larger than a preset gray threshold (for example, 0), and taking the average coordinates as the center position of the head in the weighted image of the head T1. Then, an offset value between the coordinates (for example, (128,128,128)) of the head center position and the center position of the volume data is calculated, and then, all voxels in the weighted image of the head T1 are displaced according to the offset value, so that the head center position is located at the voxel center position. During displacement, a voxel is discarded when any one of its three-dimensional coordinates falls outside of the [0,256] coordinate range.
In one example, a head surface mask (mask) is obtained by image preprocessing. As shown in fig. 2a and 2b, the head T1 weighted image 10 is subjected to image segmentation to obtain a head mask 11. The image segmentation method may be a binary method, a region growing method, an image segmentation method based on template matching, an image segmentation method based on machine learning, or the like. In the research, it is found that when the binary method is adopted to segment the image, since some parts of the surface of the head are cavities (such as oral cavity, nasal cavity, external auditory canal, etc.) containing air, and some parts of the skull are easy to be mistakenly identified as the background due to low gray values, and thus an accurate head mask cannot be obtained. The image segmentation method based on template matching and machine learning has the defects of high development cost and long algorithm running time due to the fact that registration, data annotation and model training are required. After repeated research, it was found that segmenting the image using the region growing method can overcome the above problems and can obtain a more accurate head mask efficiently and quickly.
In one example, the region growing method is used to image segment the head T1 weighted image. Preferably, the image segmentation is performed using a region growing method. After the region growing method is executed on each coronal image of the same patient, the same region growing method can be executed on the whole head three-dimensional data, so that a mask of the whole head of the patient can be obtained without additional stitching. Of course, one skilled in the art may also use any one of the sagittal plane and axial plane of the weighted image of the head T1 for image segmentation.
In one example, in the region growing method, the upper left corner of each two-dimensional head T1 weighted image to be processed is set as a seed point, and region expansion is performed in 8 neighborhoods. In one example, when the head T1 weighted image is subjected to region growing, whether the pixel point is the same region as the seed point is determined by determining whether the gray level difference value between the seed point and the adjacent pixel point in the 8-neighbor region is within the gray level threshold range. For example, the gradation threshold range is set such that the gradation value is 10 or less. And when the gray difference between the seed point and the adjacent pixel point in the 8 neighborhood is less than or equal to 10, determining the pixel point as the pixel point in the same area with the seed point.
In one example, since the region growing method is relatively sensitive to noise and at the same time since the gray values of the auditory canal, nasal cavity and oral cavity are typically lower than the gray values of the head surface, resulting in some small holes being formed in the extracted coronal region of the head, i.e. in the head mask, we need to perform a second morphological treatment ("first morphological treatment" to be described in detail below) on the extracted head mask to fill the holes in the head mask.
In one example, the second morphological treatment is a closed operation. The closing operation is performed by expanding and then etching each extracted head coronal plane with a structural element (e.g. diamond-type structural element) to obtain a complete and refined head mask 11 (as shown in fig. 2 b). Not only is the hole in the head mask 11 eliminated by the closing operation, but the contour of the head mask 11 is smoothed.
In one example, the closed-cell processed head mask is passed through a first morphological process to obtain the head surface mask 12. The first morphological treatment is an etching operation and the head surface mask 12 is a mask having a predetermined thickness delta (as shown in fig. 3).
In one example, the structural element used in the etching operation is a spherical structural element, the radius of which is set to a preset thickness δ. In the research, when the thickness is only 1mm, the step effect of the polygonal grid representation image extracted in the subsequent step is obvious, and the reconstruction effect is poor. When the thickness exceeds 10mm, a remarkable double-layer surface is generated, which causes visual confusion and greatly affects the visual effect. Therefore we set the range of the preset thickness delta to be 1mm < delta < 10mm.
In order to improve the drawing efficiency, the data quality and generate better visual effect, we need to generate the head surface polygonal mesh representation image from the obtained head surface mask with preset thickness to extract the vertex data file of the polygonal primitive. In one example, the polygonal mesh representation image may be a triangular mesh representation image, a quadrilateral mesh representation image, or the like. Since bow tie problems occur when the number of sides of the pattern is greater than 3, such as a quadrilateral mesh image, while using an image represented by a triangular mesh enables the performance of the computer to be optimized, it is preferable to mask the head surface directly to generate a triangular mesh representation image. Of course, the person skilled in the art may also generate other types of polygonal mesh representation images from the head surface mask, i.e. images in which the number of sides of the polygon of the generated polygonal mesh representation image is larger than 3. When the generated image is a polygon mesh representation image of another type, the polygon mesh representation image of another type needs to be thinned (e.g., triangulated) and converted into triangle data. Since other types of polygons need to be converted into triangles, the number of patches in the polygon mesh representation image is doubled, and thus the calculation amount of a computer is increased.
In one example, the obtained head surface mask is passed through a moving Cube (Marching Cube) algorithm to generate a head surface triangle mesh representation image. Since the generated head surface mask with the preset thickness has some noise, so that the head surface mask has local roughening effect, and the roughening effect directly influences the smoothness of the triangular mesh representation image generated in the subsequent step, the head surface mask needs to be subjected to smoothing treatment to obtain the head surface mask after noise reduction. The smoothing process may be gaussian smoothing process by a gaussian convolution kernel, or constraint smoothing (i.e., smoothing constraint) process by a smoothing operator. While gaussian smoothing processes images, while relatively fast, breaks down some details, constraint smoothing can preserve more details. Meanwhile, constraint smoothing is suitable for processing data with smaller volume, so that better smoothing effect can be obtained while more details are kept. Whereas the head surface mask provided by the present invention has been size normalized and normalized to a smaller volume of data (e.g., 256 x 256), it is preferable that constraint smoothing noise reduction be performed on a head surface mask of a preset thickness.
In one example, the noise-reduced head surface mask is input into a moving cube algorithm and the head surface triangle mesh is output as a vertex data file representing triangle primitives in the image. In step S2, the vertex data file is vertex data including all triangle primitives in the image.
In one example, during a subsequent rendering process, we need to transmit the obtained vertex data file to the WebGL server for rendering. The vertex data file may be any data that we need, that is, we need to define all the data that make up the vertex according to the actual needs. For example, the attributes of the vertex data are defined to include vertex position, vertex texture, vertex color, and the like. But may also include diffuse and specular colors, coloring effects, matrix rows, etc.
In one example, the vertex position is at least one of an ordered vertex list and an ordered position list, although an ordered vertex index list may also be included. By orderly setting the vertex information (such as the number of vertices, the positions, the ordering, the number of triangular patches and the like) in the list, the vertex information can be combined with the vertex order (such as the clockwise direction) set in the layout in the subsequent rendering process, and the orientation of the triangular patches in the drawn image is further determined.
In one example, to ensure that the rendered image has a curved surface effect, we should also generate a vertex normal vector file. The vertex normal vector file is an ordered list of vertex normal vectors including the vertices of all triangular patches. In one example, the method of obtaining the vertex normal vector includes the steps of:
step S31, obtaining a shared vertex and normal vectors of all triangle primitives connected with the shared vertex according to the vertex data file of the head surface triangle mesh representation image;
step S32, carrying out mean value and normalization on normal vectors of all triangle primitives connected with the shared vertex to obtain the normal vector of the shared vertex;
step S33 iterates steps S31-S32 to obtain vertex normal vectors of all triangle primitives in the image.
In one example, this size model is difficult to store and render even today's graphics workstations because the amount of data for triangle primitives extracted in the acquired head surface triangle representation image can be as high as 5000K-2000K. In order to reduce the number of triangle primitives in the triangle mesh to obtain lightweight vertex data files and vertex normal vector files, thereby saving storage space, we can also sample the head surface triangle mesh representation image. The step S31 further includes:
sampling the head surface polygon mesh representation image to remove a predetermined ratio of vertices to obtain a vertex data file retaining vertices (i.e., a lightweight vertex data file);
and obtaining normal vectors (namely lightweight vertex normal vector files) of all shared vertexes and all triangle primitives connected with the corresponding shared vertexes according to the vertex data file of the reserved vertexes.
The lightweight vertex data file and the lightweight vertex normal vector file are obtained through sampling, so that a user can view the head surface drawing file in real time and view the head surface image through WebGL rendering in real time without being limited by the available memory capacity of a browser and network bandwidth.
In one example, the sampling method is a grid reduction algorithm, and a static reduction algorithm, a dynamic reduction algorithm, or a viewpoint-based reduction algorithm may be employed. Preferably, the sampling method is a dynamic reduced algorithm, such as an edge collapse operation. In one example, the edge collapse operation may be implemented using, for example, an edge collapse reduction algorithm, a grid reduction algorithm based on a quadratic error metric.
In one example, the setting of the predetermined ratio of vertices to be removed is a condition for the termination of the iteration, i.e., when the ratio of the deleted vertices reaches the predetermined ratio, the iteration is stopped. In one example, the predetermined ratio r is set in a range of 0.5.ltoreq.r.ltoreq.0.9. For example, when the predetermined ratio r is set to 0.9, it means that 90% of vertices are removed and only 10% of vertices remain. By the design, more detail information in the image can be reserved.
In one example, step S32 further includes:
averaging the normal vector of all triangle primitives connected with the shared vertex to obtain the normal vector average value of the shared vertex;
normalizing the normal vector average to obtain the normal vector of the shared vertex.
In one example, the method of generating the head surface drawing file further includes the steps of:
saving the lightweight vertex data file and the lightweight vertex normal vector file in a file format (such as a. Obj,. Json and other formats) supported by a Web graphic library (WebGL); and/or
The lightweight vertex data file and the lightweight vertex normal vector file are compressed by the compression format (e.g. the. Obj. Gz file format) pair supported by the Web graphics library (WebGL) to reduce the time of network transmission, so that the user can view the head surface rendering file in real time and render through WebGL to view the rendered head surface image without being limited by the memory capacity and network bandwidth available to the browser.
Taking the obj format as an example, when performing a save operation, a mesh (mesh) type (such as P represents a polygon, L represents a line segment, etc.), a mesh material, a vertex coordinate, a vertex normal vector, an RGBD value, and a vertex combination index of a triangle patch (i.e., a triangle primitive) need to be written in sequence, and finally saved as a surface drawing file with ". Obj" as a suffix. To reduce network transmission time, the above-described surface drawing file may be compressed using script commands to obtain a compressed surface drawing file ending with ". Obj.gz", typically with a compression rate of around 50%.
In one example, a method of rendering a head surface image is provided according to another embodiment of the present invention. The rendering method of the head surface image comprises the following steps:
generating a head surface drawing file according to the generation method of the head surface drawing file;
rendering by a shader to obtain a rendered head surface image.
In one example, the head surface rendering file is sequentially input into a vertex shader and a fragment shader for rendering to obtain a rendered head surface image. The vertex shader performs a number of computations on the vertex data file and the vertex normal vector file to obtain the position of the vertex on the interactive interface, the vertex color, etc. The patch shader calculates the final color of the patch on the interactive interface through the vertex data file and the vertex normal vector file.
In one example, the rendering framework of vertex data files and vertex normal vector files needs to be able to support WebGL. Taking the VUE frame as an example, the saved or compressed surface drawing file is imported into the VUE frame, so that the surface drawing effect of the head surface image can be checked in the visual window.
In one example, a readable storage medium is provided according to yet another embodiment of the present invention. A "readable storage medium" of embodiments of the present invention refers to any medium that participates in providing programs or instructions to a processor for execution. Such a medium may take many forms, including but not limited to, non-volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as a storage device. Volatile media includes dynamic memory, such as main memory. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise a bus. Transmission media can also take the form of acoustic or light waves, such as those generated during Radio Frequency (RF) and Infrared (IR) data communications. Common forms of readable storage media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a computer can read.
The readable storage medium stores thereon a program or instructions that, when executed by a processor, perform the above-described method of generating a head surface drawing file and method of rendering a head surface image.
The method of generating a head surface drawing file, the method of rendering a head surface image, and the readable storage medium according to the present invention have at least one of the following advantages:
(1) The generation method of the head surface drawing file, the rendering method of the head surface image and the readable storage medium can enable a user to obtain the head surface drawing file suitable for Web interactive interface rendering only by inputting the head magnetic resonance imaging image;
(2) The generation method of the head surface drawing file, the rendering method of the head surface image and the readable storage medium can better support the use of a neurosurgery imaging system based on cloud service, and fill the blank that the display of the outer surface of the head is lacking in the traditional brain analysis software;
(3) According to the method for generating the head surface drawing file, the method for rendering the head surface image and the readable storage medium, provided by the invention, a ratio of vertexes are removed by sampling the head surface polygonal grid image so as to lighten the vertex data file and the vertex normal vector file, so that the storage space is saved, the transmission efficiency between networks is improved, and meanwhile, a user can check the scalp surface drawing file in real time;
(4) According to the method for generating the head surface drawing file, the method for rendering the head surface image and the readable storage medium, normalization, segmentation, smoothing, sampling and other methods are comprehensively considered in the process of generating the head surface drawing file, so that the whole processing process is more robust, the head surface drawing file is more accurate and smooth, and the visual effect is improved.
Although a few embodiments of the present general inventive concept have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles and spirit of the general inventive concept, the scope of which is defined in the claims and their equivalents.