CN117252937A

CN117252937A - Image generation method, device, apparatus, medium, and program product

Info

Publication number: CN117252937A
Application number: CN202211328364.3A
Authority: CN
Inventors: 贺继帅; 廖俊; 黄凯; 姚建华; 陈翔; 赵爽
Original assignee: Tencent Technology Shenzhen Co Ltd; Xiangya Hospital of Central South University
Current assignee: Tencent Technology Shenzhen Co Ltd; Xiangya Hospital of Central South University
Priority date: 2022-10-27
Filing date: 2022-10-27
Publication date: 2023-12-19

Abstract

The application discloses an image generation method, an image generation device, an image generation medium and a program product, and belongs to the field of image processing. The method comprises the following steps: acquiring a medical image of focal tissue, the medical image including a focal contour of the focal tissue; identifying pathological sampling areas corresponding to the focal outlines; generating a sampling grid for meshing a pathological sampling area, wherein the sampling grid comprises at least two sub-grids which cover the pathological sampling area and are arranged in an array; each sub-grid covered pathologic sampling area comprises focus tissues; a medical image covered with a grid of material is displayed. By adopting the scheme, the drawing grids can be automatically generated without manual sketching of doctors, errors and randomness of manual planning of the drawing grids can be reduced to the greatest extent, a uniform and homogeneous division scheme of the drawing grids is provided, doctors can be assisted in pathological drawing, errors are avoided during drawing, and the doctors can be assisted in making subsequent pathological analysis and treatment plans.

Description

Image generation method, device, apparatus, medium, and program product

Technical Field

The present invention relates to the field of image processing, and in particular, to an image generating method, apparatus, device, medium, and program product.

Background

Mohs microscopy (Mohs Micrographic Surgery, NMS) is a surgical excision procedure that is common at this stage. In the Mohs operation, a macroscopic tumor is excised first, then a Mohs grid is divided into a basal part and a boundary part after the tumor excision, materials are obtained, and whether the obtained materials have tumor residues or not is checked under a microscope. If tumor residues exist, the corresponding sampling position is found, secondary excision is carried out, and the process is repeated until no tumor residues exist.

In the related art, mohs grids for Mohs surgery are manually sketched on medical images based on doctors, and then guide the doctors to carry out subsequent material drawing work. However, the morse mesh divided by the related art method has large differences in shape, size, etc., and is prone to errors in drawing materials.

Disclosure of Invention

The application provides an image generation method, an image generation device, a medium and a program product. The technical scheme is as follows:

according to an aspect of the present application, there is provided an image generation method, the method including:

acquiring a medical image of a focal tissue, the medical image comprising a focal contour of the focal tissue;

Identifying a pathological sampling area corresponding to the focus outline;

generating a sampling grid for meshing the pathological sampling area, wherein the sampling grid comprises at least two sub-grids which cover the pathological sampling area and are arranged in an array; each sub-grid covers the pathological material-drawing area and comprises focus tissues;

displaying the medical image covered with the material drawing grid.

According to another aspect of the present application, there is provided an image generation method, the method including:

displaying the medical image covered with a sampling grid, wherein the sampling grid comprises at least two sub-grids which cover pathological sampling areas corresponding to the focus outline and are arranged in an array; the lesion tissue is contained in the pathological sampling area covered by each sub-grid.

According to another aspect of the present application, there is provided an image generating apparatus including:

an acquisition module for acquiring a medical image of a focal tissue, the medical image comprising a focal contour of the focal tissue;

The identification module is used for identifying a pathological sampling area corresponding to the focus outline;

the generation module is used for generating a sampling grid for meshing the pathological sampling area, and the sampling grid comprises at least two sub-grids which cover the pathological sampling area and are arranged in an array; each sub-grid covers the pathological material-drawing area and comprises focus tissues;

and the display module is used for displaying the medical image covered with the material drawing grid.

the display module is used for displaying the medical image covered with the material drawing grid, and the material drawing grid comprises at least two sub-grids which cover pathological material drawing areas corresponding to the focus outline and are arranged in an array; the lesion tissue is contained in the pathological sampling area covered by each sub-grid.

According to another aspect of the present application, there is provided a computer device comprising: a processor and a memory storing a computer program that is loaded and executed by the processor to implement the image generation method as described above.

According to another aspect of the present application, there is provided a computer-readable storage medium storing a computer program loaded and executed by a processor to implement the image generation method as described above.

According to another aspect of the present application, there is provided a computer program product comprising computer instructions stored in a computer readable storage medium, from which a processor retrieves the computer instructions, causing the processor to load and execute the image generation method provided in the above aspect.

The beneficial effects that technical scheme that this application embodiment provided include at least:

by acquiring the medical image of the focus tissue, the medical image comprises the focus outline of the focus tissue, and then, the pathological material taking area corresponding to the focus outline is identified, so that the automatic identification of the pathological material taking area is realized, and the processing efficiency of the medical image is improved. Then, by generating a sampling grid for meshing a pathological sampling area, the sampling grid comprises at least two sub-grids which cover the pathological sampling area and are arranged in an array, and the pathological sampling area covered by each sub-grid comprises focus tissues; accordingly, the drawing grids can be automatically generated without manual sketching by doctors, errors and randomness of manual planning of the drawing grids can be reduced to the greatest extent, a uniform and homogeneous drawing grid dividing scheme is provided, and materials can be drawn into focus tissues in each sub-grid, so that the pathological drawing efficiency is improved in an auxiliary manner; finally, medical images covered with sampling grids are displayed, so that doctors can be assisted in pathological sampling, errors generated during sampling are avoided, and the doctors can be assisted in subsequent pathological analysis and treatment planning.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 illustrates a block diagram of a computer system provided by an exemplary embodiment;

FIG. 2 illustrates an application scenario diagram of an image generation method provided by an exemplary embodiment;

FIG. 3 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

FIG. 4 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

FIG. 5 illustrates a schematic diagram of an image generation method provided by an exemplary embodiment;

FIG. 6 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

FIG. 7 illustrates a schematic diagram of an image generation method provided by an exemplary embodiment;

FIG. 8 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

FIG. 9 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

FIG. 10 illustrates a schematic diagram of an image generation method provided by an exemplary embodiment;

FIG. 11 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

FIG. 12 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

FIG. 13 illustrates a schematic diagram of an image generation method provided by an exemplary embodiment;

FIG. 14 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

FIG. 15 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

FIG. 16 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

FIG. 17 is a schematic diagram of an image generation method provided by an exemplary embodiment;

FIG. 18 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

FIG. 19 illustrates a schematic diagram of an image generation method provided by an exemplary embodiment;

FIG. 20 illustrates a flowchart of an image generation method provided by an exemplary embodiment;

fig. 21 is a block diagram showing the structure of an image generating apparatus provided by an exemplary embodiment;

fig. 22 is a block diagram showing the structure of an image generating apparatus provided by an exemplary embodiment;

Fig. 23 shows a block diagram of a computer device provided by an exemplary embodiment.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.

The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in this application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.

It should be understood that, although the terms first, second, etc. may be used herein to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.

FIG. 1 illustrates a block diagram of a computer system provided in an exemplary embodiment of the present application. The computer system may be implemented as a system architecture for an image generation method. The computer system includes: a terminal 120 and a server 140.

The terminal 120 may be an electronic device such as a mobile phone, a tablet computer, a vehicle-mounted terminal (car), a wearable device, a PC (Personal Computer ), an unmanned reservation terminal, or the like. The terminal 120 may be provided with a client for running a target application, which may be a specific application for image processing of medical images, or may be another application provided with an image processing function of medical images, which is not limited in this application. In addition, the Application is not limited to the form of the target Application, and may be a web page, including but not limited to an App (Application), an applet, etc. installed in the terminal 120.

The server 140 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server providing cloud computing services, a cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, a content delivery network (Content Delivery Network, CDN), and cloud servers of basic cloud computing services such as big data and an artificial intelligent platform. The server 140 may be a background server of the target application program, and is configured to provide a background service for a client of the target application program.

Cloud Technology (Cloud Technology) refers to a hosting Technology that unifies serial resources such as hardware, software, network and the like in a wide area network or a local area network to realize calculation, storage, processing and sharing of data. The cloud technology is based on the general names of network technology, information technology, integration technology, management platform technology, application technology and the like applied by the cloud computing business mode, can form a resource pool, and is flexible and convenient as required. Cloud computing technology will become an important support. Background services of technical networking systems require a large amount of computing, storage resources, such as video websites, picture-like websites, and more portals. Along with the high development and application of the internet industry, each article possibly has an own identification mark in the future, the identification mark needs to be transmitted to a background system for logic processing, data with different levels can be processed separately, and various industry data needs strong system rear shield support and can be realized only through cloud computing.

In some embodiments, the server 140 described above may also be implemented as a node in a blockchain system. Blockchain (Blockchain) is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanisms, encryption algorithms, and the like. The blockchain is essentially a decentralised database, and is a series of data blocks which are generated by association by using a cryptography method, and each data block contains information of a batch of network transactions and is used for verifying the validity (anti-counterfeiting) of the information and generating a next block. The blockchain may include a blockchain underlying platform, a platform product services layer, and an application services layer.

Communication between the terminal 120 and the server 140 may be through a network, such as a wired or wireless network.

In the image generating method provided by the embodiment of the present application, the execution subject of each step may be a computer device, where the computer device refers to an electronic device having data computing, processing and storage capabilities. Taking the implementation environment of the solution shown in fig. 1 as an example, the image generating method may be executed by the terminal 120, for example, the image generating method may be executed by a client terminal that installs the running target application program in the terminal 120, or the image generating method may be executed by the server 140, or the image generating method may be executed by the terminal 120 and the server 140 in an interactive and coordinated manner, which is not limited in this application.

Those skilled in the art will recognize that the number of terminals may be greater or lesser. Such as the above-mentioned terminals may be only one, or the above-mentioned terminals may be several tens or hundreds, or more. The number of terminals and the device type are not limited in the embodiment of the present application.

Embodiments of the present application relate to artificial intelligence and computer vision techniques.

Artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use the knowledge to obtain optimal results. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision.

The artificial intelligence technology is a comprehensive subject, and relates to the technology with wide fields, namely the technology with a hardware level and the technology with a software level. Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.

With research and advancement of artificial intelligence technology, research and application of artificial intelligence technology is being developed in various fields, such as common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned, automatic driving, unmanned aerial vehicles, robots, smart medical treatment, smart customer service, etc., and it is believed that with the development of technology, artificial intelligence technology will be applied in more fields and with increasing importance value.

Computer Vision (CV) is a science of how to "look" a machine, and more specifically, a camera and a Computer are used to replace human eyes to perform machine Vision such as recognition and measurement on a target, and further perform graphic processing, so that the Computer is processed into an image more suitable for human eyes to observe or transmit to an instrument to detect. As a scientific discipline, computer vision research-related theory and technology has attempted to build artificial intelligence systems that can acquire information from images or multidimensional data. Computer vision techniques typically include image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D techniques, virtual reality, augmented reality, synchronous positioning, map construction, and other techniques, as well as common biometric recognition techniques such as face recognition, fingerprint recognition, and the like.

The image generating method of the embodiment of the application can be used as a medical auxiliary method, can assist doctors to carry out pathological sampling on focal tissues, and is applied to various medical and medical related technical fields of Mohs (Mohs) microscopic operation (Mohs Micrographic Surgery, NMS) (Mohs operation for short), biopsy, aspiration biopsy and puncture biopsy pathological detection, and can be used for generating sampling grids corresponding to medical images of focal tissues and displaying the medical images covered with the sampling grids so as to assist doctors to carry out next operation.

Fig. 2 is a schematic view of an application scenario of an image generating method according to an exemplary embodiment of the present application. The application scenario mainly involves a doctor 200, a patient 201, a shooting component 202, a projection component 203, a terminal 206 and a (background) server 205.

Specifically, during a Morse operation performed on a patient 201 by a physician 200, the imaging assembly 202 may image focal tissue of the patient 201 to obtain a medical image 204, the medical image 204 including a focal contour of the focal tissue. The server 205 acquires a medical image of a lesion tissue, identifies a pathological material-drawing region corresponding to a lesion outline, and generates a material-drawing grid for meshing the pathological material-drawing region, wherein the material-drawing grid comprises at least two sub-grids which cover the pathological material-drawing region and are arranged in an array, and the pathological material-drawing region covered by each sub-grid comprises the lesion tissue. Then, the server 205 generates a medical image 207 covered with a material drawing mesh and transmits it to the terminal 206, and displays the medical image 207 covered with the material drawing mesh on the terminal 206, so that the doctor 200 can perform subsequent pathological material drawing on the pathological material drawing region with reference to the medical image 207 covered with the material drawing mesh.

Optionally, the projection component 203 may also project the sampling grid onto a pathological sampling region of the patient 201, on which the sampling grid is displayed. Thus, the doctor 200 can perform subsequent pathological sampling of the pathological sampling region with reference to the sampling grid.

Optionally, the shooting component 202 is a device with an image capturing function, and may mainly include a lens and an auto-focusing component, with an auto-focusing function, including at least one of the following: the shooting component 202 is provided with an automatic focusing function (for example, a mechanical component in the camera moves a photosensitive chip up and down to focus) and is provided with a common lens without the automatic focusing function; the photographing assembly 202 drives an auto-focus lens with an auto-focus part using an adapter; the shooting component 202 is provided with a focusing lens; the photographing component 202 is a normal camera and is provided with a normal lens, and focuses through an additional liquid lens.

Optionally, the projection component 203 is a device with a screen projection function for projecting the material-drawing grid onto the pathological material-drawing region of the patient 201, which may be at least one of the following: digital projector, liquid crystal projector, laser projector.

Common surgical excision methods include: local dilation ablation (Wide Local Excision, WLE) and Mohs microscopy surgery (Mohs Micrographic Surgery, NMS). WLE is a procedure in which a portion of surrounding tissue is resected more than at the time of tumor resection to prevent tumor remnants. However, NMS is of great advantage over having a significantly infiltrated and spread skin tumor, since the tumor is difficult to accurately determine the boundary. The NMS method process can be summarized as: after primary tumor resection, sampling is carried out on the basal part and the boundary part after tumor resection, the sampled tissue is rapidly paraffin-sliced, pathological sections are manufactured, and whether tumor residues exist is judged through microscopic pathological detection. If the residual exists, the corresponding sampling position is found, secondary excision is carried out, sampling and pathological detection are carried out again until no tumor residual exists. Compared with WLE, the NMS method can furthest reduce the damage to healthy tissues and the recurrence rate of tumors. In the operation process of the NMS method, one of the more difficult-to-process steps is the material drawing process, in particular how to divide the material drawing area. Due to the limitation of the size of the microscope slide, the size of the tumor material is not suitable to be too large or too small, and the length is generally about 20 mm. For some larger tumors, the number of the obtained regions and the obtained parts is often large, and can reach tens or hundreds. If different sampling areas are manually sketched by doctors, errors are easy to generate, the subsequent pathological detection is possibly difficult to carry out, the sampling areas are easy to overlap or the sampling sizes are seriously inconsistent, and unified and homogenous manual planning is difficult to realize.

Based on this, taking morse operation as an example, fig. 3 shows a flowchart of an image generation method provided in an exemplary embodiment of the present application, and the method is applied to the terminal 120 shown in fig. 1 or a client or server 140 installed on the terminal 120 to support image processing, and the method includes:

at step 320, a medical image of the focal tissue is acquired, the medical image including a focal contour of the focal tissue.

Focal tissue refers to tissue with pathogenic microorganisms, any tissue organ may become a focus, and the focal tissue may be tumor tissue. Illustratively, a lesion contour refers to a continuous contour curve of lesion tissue. The lesion contour may be regular or irregular in shape. The medical image is an image obtained by taking a picture of a lesion tissue after the lesion tissue is resected by the Morse operation.

Illustratively, the focal tissue is imaged by an imaging assembly to obtain a medical image of the focal tissue.

Alternatively, the photographing component may be a first photographing component including at least one of an industrial camera, a color (RGB) camera, a two-dimensional (2D) camera. The focal tissue is photographed by the photographing assembly and the resulting medical image is a two-dimensional planar image containing two-dimensional information of the focal tissue.

Optionally, the photographing component may be a second photographing component including at least one of a depth camera, a stereoscopic color (RGBD) camera, a point cloud camera, a three-dimensional (3D) camera. The focal tissue is photographed by the photographing component, and the obtained medical image is a two-dimensional planar image containing two-dimensional information of the focal tissue and/or a three-dimensional point cloud image containing three-dimensional information of the focal tissue.

Optionally, the type of shooting component is determined according to the location of the focal tissue.

For example, when the focal tissue is located at a relatively flat or less curved position, such as in a limb, torso, etc., a first imaging assembly may be selected that is biased toward acquiring a two-dimensional planar image. When the focal tissue is in the uneven position or has larger curvature, such as in the nose, the head, the joints and other parts, the second shooting component which is biased to collect the three-dimensional point cloud image can be selected, the three-dimensional information of the focal tissue is further introduced, and the accuracy of the subsequent medical image processing is improved.

Step 340, identifying the pathological sampling area corresponding to the focus outline.

The pathological sampling area is an area for sampling after tumor excision and preparing pathological sections by rapidly paraffin-producing the sampling tissues.

Optionally, the pathologic sampling area may be an area corresponding to a focal contour of the focal tissue, and the pathologic sampling area is an area where the focal tissue is located, and only the focal tissue is included in the pathologic sampling area. The pathologically-obtained region may be a region in which the outline of the lesion tissue is determined by performing the expansion, and the pathologically-obtained region may include the lesion tissue and at least a part of healthy tissue.

In some embodiments, the focal profile of the focal tissue is identified first, and then the pathologically-derived region corresponding to the focal profile is identified.

Alternatively, the lesion contour of the lesion tissue may be automatically identified. And identifying the focus outline of focus tissues in the medical image through an artificial intelligence AI model, and then identifying a pathological sampling area corresponding to the focus outline. The artificial intelligence AI model is a pre-trained neural network model, which may be at least one of the following types: image segmentation Unet model and semantic segmentation model deep V < 3+ > model.

Alternatively, the focal profile of the focal tissue may also be manually delineated by a physician. In response to the contouring operation, a lesion contour of the lesion tissue in the medical image is identified. And then identifying the pathological sampling area corresponding to the focus outline.

Alternatively, the focal profile of the focal tissue may also be determined in combination with automatic identification and manual delineation by a physician. An initial lesion contour of a lesion tissue in the medical image is identified by the artificial intelligence AI model, and then the initial lesion contour is adjusted in response to a contour delineating operation to determine a lesion contour of the lesion tissue. Accordingly, errors generated by automatically identifying the focus outline can be prevented, and doctors can manually modify or redraw the automatically identified focus outline to obtain a more accurate focus outline.

Step 360, generating a sampling grid for meshing a pathological sampling area, wherein the sampling grid comprises at least two sub-grids which cover the pathological sampling area and are arranged in an array; the pathological sampling area covered by each sub-grid contains focus tissues.

The sampling grid is obtained by dividing the pathological sampling area into grids. The sampling grid can divide the pathological sampling area into a plurality of small sampling areas, and at least two sub-grids in the sampling grid can be used for assisting in positioning specific sampling positions of the pathological sampling areas and assisting a doctor in carrying out subsequent pathological sampling.

Illustratively, in applying the mesh to Morse, the mesh may also be referred to as a Morse (Mohs) mesh.

Optionally, the sampling grid comprises at least two sub-grids covering the pathological sampling area and arranged in an array. The array arrangement refers to that at least two sub-grids in the material-drawing grid are regularly arranged according to rows and columns.

Illustratively, each sub-grid covers a lesion tissue contained within a pathologically-derived region. That is, in the case where the lesion tissue is irregular, the material drawing grid is also irregular. Accordingly, when the pathological sampling is carried out based on the sampling grids in the follow-up process, the focus tissue can be sampled based on each sub-grid, and the efficiency of the pathological sampling can be improved in an auxiliary mode.

Alternatively, in combination with actual clinical experience, at least two sub-grids in the grid are referred to as at least four sub-grids.

Optionally, the sub-grid parameters of at least two sub-grids of the material grid are the same, and the sub-grid parameters include at least one of the size, shape, area, and interior angle size of the sub-grid.

For example, at least two sub-grids of the material selection grid may be triangles, squares, matrices, parallelograms, diamonds, sectors, etc., and may be automatically determined according to the lesion type of the lesion tissue, or may be designated by a doctor based on clinical experience. For example, when the sub-grid is square, the side length of the square may be set to 20mm-30mm.

In some embodiments, the subgrid parameters of at least two subgrids in the material-drawing grid may be unchanged by default, and the same subgrid parameters are used for grid division of the pathological material-drawing region corresponding to the lesion tissue; or, the sub-grid parameters are preset and can be set based on the specific type of the focus tissue; alternatively, the sub-grid parameters are dynamically changed, possibly in accordance with parameters related to the capturing component of the medical image.

In an alternative embodiment, after generating the mesh for meshing the pathologically-derived region, the method further comprises: numbering at least two sub-grids in the material drawing grid to identify at least two sub-grids in the material drawing grid.

Optionally, at least two sub-grids in the material drawing grid are numbered with one or more combinations of special characters, letters, numbers to identify at least two sub-grids in the material drawing grid.

Alternatively, the numbering mode may be that the numbering is sequentially performed from the outermost ring subgrid to the innermost ring subgrid of the material drawing grid or from the innermost ring subgrid to the outermost ring subgrid of the material drawing grid according to the clockwise or counterclockwise direction. The method can also be that the grids are sequentially numbered from the uppermost layer of the grids to the lowermost layer of the grids according to the S-shaped direction, or the grids are sequentially numbered from the lowermost layer of the grids to the uppermost layer of the grids, or the grids are sequentially numbered from the leftmost layer of the grids to the rightmost layer of the grids, or the grids are sequentially numbered from the rightmost layer of the grids to the leftmost layer of the grids. In this embodiment, the numbering mode is not limited, and at least two sub-grids in the sampling grid can be uniquely identified.

Step 380, a medical image overlaid with a mesh of material is displayed.

Illustratively, after generating a sampling grid for meshing a pathological sampling region, a medical image covered with the sampling grid is displayed to assist a doctor in subsequent pathological sampling.

In some embodiments, the method further comprises: and displaying the numbers of at least two sub-grids in the material selection grid. Is used for assisting doctors to know the specific material drawing position of pathological material drawing areas, and is convenient for secondary operation and secondary material drawing when focus tissues have residues.

In summary, according to the method provided by the embodiment of the application, by acquiring the medical image of the focal tissue, the medical image includes the focal contour of the focal tissue, and then, identifying the pathological material sampling area corresponding to the focal contour, thereby realizing automatic identification of the pathological material sampling area and improving the processing efficiency of the medical image. Then, by generating a sampling grid for meshing a pathological sampling area, the sampling grid comprises at least two sub-grids which cover the pathological sampling area and are arranged in an array, and the pathological sampling area covered by each sub-grid comprises focus tissues; accordingly, the drawing grids can be automatically generated without manual sketching by doctors, errors and randomness of manual planning of the drawing grids can be reduced to the greatest extent, a uniform and homogeneous drawing grid dividing scheme is provided, and materials can be drawn into focus tissues in each sub-grid, so that the pathological drawing efficiency is improved in an auxiliary manner; finally, medical images covered with sampling grids are displayed, so that doctors can be assisted in pathological sampling, errors generated during sampling are avoided, and the doctors can be assisted in subsequent pathological analysis and treatment planning.

In an alternative embodiment of the present application, step 360 generates a mesh of material for meshing the pathologically-derived region, comprising: a material-drawing grid for meshing a pathologically-drawn region is generated based on at least one reference point in the medical image.

Illustratively, the reference points are pixel points that serve as references in the process of generating the grid of material, the reference points including at least one. Alternatively, the at least one reference point may be predetermined prior to generating the mesh of interest, or may be determined iteratively in real time during the generation of the mesh of interest.

Alternatively, in the case where the reference point is one, the reference point may be a center of the entire sampling grid, and the sampling grid for meshing the pathological sampling region centering on the reference point is expanded.

For example, in the case where the reference point is one, based on the reference point in the medical image, the first circle, the second circle, and the nth circle sub-grid surrounding the reference point are sequentially expanded and generated, n is greater than 0, and n is an integer. The ith circle of sub-grids are not overlapped with the (i+1) th circle of sub-grids, i is greater than 0 and less than or equal to n, and i is an integer.

Optionally, in the case that the reference points are multiple, the reference points may be used as vertices of the subgrid, or as midpoints of edges of the subgrid, or as trisection points or quarterion points of edges of the subgrid, so that at least two subgrids arranged in an array are generated based on the reference points of each iteration by continuous iterative expansion, and finally a material drawing grid for grid division of the pathological material drawing area is generated. The expansion generates at least two sub-grids arranged in an array, namely, one circle of sub-grids can be generated, and the sub-grids which surround the reference point and are in a shape of a Chinese character 'tian'.

For example, in the case that the number of reference points is plural, the first reference point is taken as the vertex of the sub-grid of the first iteration, the plurality of sub-grids arranged in the shape of a Chinese character 'tian' are generated by expansion, then, based on the sub-grids generated by the first iteration, the second reference point of the next iteration is determined, and expansion is continued to generate the plurality of sub-grids arranged in the shape of a Chinese character 'tian' around each second reference point.

Alternatively, in the case where there are a plurality of reference points, all the reference points in the medical image may be predetermined, and the reference points may be sequentially connected in a certain order, thereby generating a sampling grid for meshing the pathological sampling region.

Illustratively, fig. 4 shows a flowchart of an image generation method provided in an exemplary embodiment of the present application, generating a material drawing grid for meshing a pathologically-drawn region based on at least one reference point in a medical image, which may be implemented as the following steps:

at step 362, a center point of the lesion contour is determined.

Alternatively, the center point of the lesion contour may be automatically identified using an image detection algorithm, or may be manually delineated by a physician. The image detection algorithm is implemented by using an Open source computer vision library Open CV.

Step 363, expanding to generate at least two sub-grids arranged in an array by taking the central point as a reference point.

Illustratively, at least two sub-grids arranged in an array are generated by expanding by taking the center point of the focus outline as a reference point. Alternatively, the reference point may be as the center of the entire grid of interest, or as a sub-grid vertex, or as the midpoint of a sub-grid edge, or as a tri-or quarter-point of a sub-grid edge.

For example, in the case where the reference point is the center of the entire drawing grid and the subgrid is square, with the center point as the reference point, a circle of subgrids arranged in an array around the reference point may be expanded, and a circle of subgrids includes at least two subgrids.

For example, in the case that the reference point is used as a sub-grid vertex and the sub-grid is square, the four sub-grids arranged in an array can be generated by expanding with the center point as the reference point, and the common sub-grid vertex of the four sub-grids is the center point.

For example, in the case that the reference point is taken as the midpoint of the side of the sub-grid and the sub-grid is square, the two sub-grids arranged in an array can be expanded and generated by taking the center point as the reference point, and the midpoint of the common side of the two sub-grids is the center point.

Optionally, the method further comprises: and taking the central point as a reference point, and generating a coordinate system of the medical image based on the reference point. Accordingly, the coordinate positions of the reference points and the coordinate positions of at least two sub-grids of the array arrangement generated by expansion can be determined in the coordinate system of the medical image, and the accuracy of the expansion process is improved.

And step 365, ending the expansion to generate at least two sub-grids under the condition that the intersection exists between the at least two sub-grids and the whole outline of the focus outline, so as to obtain a sampling grid for meshing the pathological sampling area.

In an exemplary case, in which the intersection exists between at least two sub-grids and the whole outline of the lesion outline, the obtained grid is completely beyond the whole lesion outline, and at this time, the expansion is finished to generate at least two sub-grids, so as to obtain the obtained grid for grid division of the pathological obtained region.

Step 367, under the condition that at least two sub-grids and at least a part of the outlines of the focus do not have intersection, using the newly added sub-grid vertices of at least two sub-grids generated by the expansion of the current round as the reference points of the next round, and continuing to use the reference points of the next round as expansion points to expand to generate at least two sub-grids arranged in an array.

Illustratively, the newly added sub-grid vertices refer to the sub-grid vertices of at least two sub-grids generated by this round of expansion. In the process of generating the material-drawing grid, if the process is carried out by taking the reference point as the sub-grid vertex, the newly added sub-grid vertex is the other sub-grid vertex except the reference point used in the round in each sub-grid vertex of at least two sub-grids generated by expansion.

In an exemplary case, when at least two sub-grids and at least a part of the outline of the focus do not have intersection, the obtained grid is not completely beyond the whole focus outline, at least a part of the focus outline is outside the obtained grid, at this time, under the condition that the reference points are multiple, the vertex of the new sub-grid of at least two sub-grids generated by the expansion of the present round is used as the reference point of the next round, the reference point of the next round is used as the expansion point, and at least two sub-grids arranged in an array are generated by expansion. In the case that the reference point is one, the step 363 is continued with the center point as the reference point, and at least two sub-grids of the array arrangement are generated by expansion.

Optionally, the next round of reference points are used as expansion points, and at least two sub-grids arranged in an array are generated by expansion, which can be a circle of sub-grids or a sub-grid which surrounds the expansion points and takes a shape of a Chinese character 'tian'.

In some embodiments, step 367 above may be implemented as: and under the condition that at least two sub-grids and at least a part of the outlines of the focus do not have intersection, taking newly added sub-grid vertexes of at least two sub-grids generated by the expansion of the round as reference points of the next round, continuing to take the reference points of the next round as expansion points, and expanding to generate at least two sub-grids which are distributed in a shape of a Chinese character 'tian' and surround the expansion points.

In this embodiment, at least two sub-grids arranged in an array are generated by iterative continuous expansion, so that at least two sub-grids in the generated sampling grids can be ensured to completely cover a pathological sampling area, and missing sampling is avoided. Under the condition that one circle of sub-grids is generated by each iteration expansion, the pathological sampling area can be fully covered to the greatest extent, and the accuracy of the generated sampling grids is improved. When the sub-grid which is generated by one expansion point in the current iteration and is in the shape of a Chinese character 'tian' is generated at each iteration and exceeds the edge area of the pathologic sampling area, the expansion point does not need to carry out the next iteration, the next iteration of other expansion points can not be influenced, and the processing efficiency of the medical image can be improved on the basis of ensuring the complete coverage of the pathologic sampling area to the greatest extent.

As an example, fig. 5 shows a schematic diagram of an image generating method provided by an exemplary embodiment of the present application. In this embodiment, taking each sub-grid of the material-drawing grid as a square, taking a reference point as a sub-grid vertex, expanding to generate at least two sub-grids which are arranged in a shape of a Chinese character 'tian' around an expansion point as an example.

As shown in fig. 5-1, a medical image of a focal tissue is shown, the medical image includes a focal contour of the focal tissue, and a center point of the focal contour is a point O. As shown in fig. 5-2, the coordinate system of the medical image is determined with the point O as the origin of coordinates, the first quadrant of the coordinate system being denoted as Q1, the second quadrant being denoted as Q2, the third quadrant being denoted as Q3, and the fourth quadrant being denoted as Q4. With the center point O as a reference point, four sub-grids arranged in an array are generated by expansion and are respectively expressed as square sub-grids a, b, c, d. Other sub-grid vertices except for the sub-grid vertex O in each sub-grid vertex of the four square sub-grids are called as new sub-grid vertices, and the total number of the new sub-grid vertices is 8, namely sub-grid vertices A, B, C, D, E, F, G, H. As is apparent from fig. 5-2, there is no intersection between the four square sub-grids and the lesion outline, so that the newly added sub-grid vertices of at least two square sub-grids generated by the expansion of the present round are used as reference points of the next round, the reference points of the next round are used as expansion points, and at least two square sub-grids distributed in a shape of a Chinese character 'tian' around the expansion points are generated by expansion. For example, as shown in fig. 5-3, with sub-grid vertex a as an expansion point, expanding to generate square sub-grid a, f, e, h arranged in a shape of a Chinese character 'tian' around sub-grid vertex a; expanding to generate square sub-grids a, B, f, g which are distributed in a shape of a Chinese character 'tian' around the sub-grid vertexes B by taking the sub-grid vertexes B as expansion points; the sub-grid vertices D are used as expansion points, and square sub-grids a, c, h, i which are distributed in a shape of a Chinese character 'tian' and surround the sub-grid vertices D are generated by expansion. 5-4, until expansion is finished to generate at least two square sub-grids under the condition that intersection exists between at least two square sub-grids and the whole outline of the lesion outline, and obtaining a sampling grid for meshing a pathological sampling area.

In some embodiments, referring to fig. 4, generating a sampling grid for meshing a pathologically sampled region based on at least one reference point in a medical image may be implemented as:

at step 362, a center point of the lesion contour is determined.

Alternatively, the center point of the lesion contour may be automatically identified using an image detection algorithm, or may be manually delineated by a physician. The image detection algorithm may be any image detection algorithm in the Open source computer vision library Open CV.

Step 364, determining the center point as a reference point, and determining a coordinate system based on the reference point.

For example, the center point is determined as a reference point, and the coordinate system is determined based on the reference point. Optionally, the coordinate system is a planar coordinate system, the lateral axis of the coordinate system being parallel to the length of the medical image and the longitudinal axis of the coordinate system being parallel to the width of the medical image.

In step 366, in the coordinate system, sub-grid vertex coordinates of at least two sub-grids of the array arrangement are calculated.

Illustratively, according to the sub-grid parameters of at least two sub-grids, sub-grid vertex coordinates of at least two sub-grids arranged in an array are calculated in a coordinate system, so that the sub-grid vertex coordinates can be connected subsequently to generate a material-drawing grid.

In some embodiments, the sub-grid parameters of the at least two sub-grids include at least one of a size, shape, area, interior angle size of the sub-grid; wherein the sub-grid parameters are fixed by default; or, the sub-grid parameters are preset; alternatively, the sub-grid parameters are dynamically changed.

Alternatively, in case the shape of the sub-grid is not rectangular or square, the sub-grid parameters comprise the internal angle size of the sub-grid. For example, the subgrid is a triangle, and the interior angle size refers to the size of each interior angle of the triangle. The subgrid is a diamond, and the interior angle size refers to the size of each interior angle of the diamond.

In step 368, a sampling mesh for meshing the pathologically sampling region is generated based on the vertex coordinates of each sub-mesh.

Optionally, based on the vertex coordinates of each sub-grid, sequentially connecting the vertex coordinates of each sub-grid in a certain order, and generating a sampling grid for performing grid division on the pathologically sampling area.

By way of example, fig. 6 shows a flowchart of an image generation method provided by an exemplary embodiment of the present application, step 368 may be further implemented as the following steps:

step 410, correspondingly generating a sub-grid based on the vertex coordinates of at least three sub-grids; at least one edge of the subgrid is parallel to a target axis of the coordinate axes.

Illustratively, at least two sub-grids of the material selection grid may be one of triangles, rectangles, squares, diamonds, and parallelograms. Based on the at least three sub-grid vertex coordinates, a sub-grid is correspondingly generated, and at least one edge of the sub-grid is parallel to a target axis in the coordinate axes, wherein the target axis can be a transverse axis or a longitudinal axis.

As an example, fig. 7 shows a schematic diagram of an image generating method provided by an exemplary embodiment of the present application. The calculated sub-grid vertex coordinates of at least two sub-grids of the array arrangement are shown in fig. 7-1. 7-2, when the subgrid is a triangle, a triangle subgrid is correspondingly generated based on the vertex coordinates of the three subgrids, and at least one side of the triangle subgrid is parallel to the transverse axis or parallel to the longitudinal axis. 7-3, when the subgrid is a parallelogram, a parallelogram subgrid is correspondingly generated based on the vertex coordinates of the four subgrids, and at least one side of the parallelogram subgrid is parallel to the transverse axis. 7-4, when the subgrid is square, a square subgrid is correspondingly generated based on the vertex coordinates of the four subgrids, and at least one side of the square subgrid is parallel to the transverse axis or parallel to the longitudinal axis.

It should be noted that fig. 7 illustrates a manner of generating a sub-grid based on at least three sub-grid vertex coordinates according to an exemplary embodiment. According to the actual technical needs, the direction of the subgrid may also be changed, for example, the parallelogram subgrid divided in fig. 7-3 may be set to uniformly tilt to the right, or other subgrid generating modes may also be set, for example, a triangle subgrid is correspondingly generated based on the vertex coordinates of four subgrids, which is not limited in this embodiment.

In step 411, a target sub-grid of the at least two sub-grids is determined as a sub-grid belonging to the material-drawing grid, and at least one sub-grid vertex of the target sub-grid is located in the lesion outline.

Illustratively, a target subgrid refers to a subgrid belonging to the grid of interest among at least two subgrids. Optionally, determining a target subgrid of the at least two subgrids as a subgrid belonging to the material-drawing grid, wherein at least one subgrid vertex of the target subgrid is positioned in the focus contour. Thus, it is ensured that the mesh of material can completely exceed the entire lesion contour.

Taking fig. 7-4 as an example, the subgrid at the top right corner of the first quadrant Q1, the subgrid at the top left corner of the second quadrant Q2, and the subgrid at the bottom left corner of the third quadrant Q3 are subgrids not belonging to the material drawing grids, and then need to be removed.

In this embodiment, the sub-grid vertex coordinates of at least two sub-grids are directly calculated, and are connected to generate the material-drawing grid, so that iteration is not required, the influence of errors generated by the previous iteration on the next iteration or the influence on the whole material-drawing grid can be effectively avoided, and the accuracy of the generated material-drawing grid can be improved.

In an alternative embodiment of the present application, the method further comprises: removing repeated sub-grids in the at least two sub-grids; or, removing the sub-grids which have no intersection with the focus outline and do not contain focus tissues in the at least two sub-grids.

Optionally, when the material-drawing grids are generated based on the steps, the sub-grids are repeatedly divided for at least a part of the pathological material-drawing area, or the sub-grids are also divided for the healthy tissue outside the lesion outline, so that the repeated sub-grids in at least two sub-grids need to be removed, or the sub-grids which do not intersect with the lesion outline and do not contain the lesion tissue in at least two sub-grids are removed, that is, the sub-grids which only cover the healthy tissue are removed, so that the pathological material-drawing area covered by each sub-grid is ensured to contain the lesion tissue.

In this embodiment, by removing the repeated subgrid or the subgrid which does not have intersection with the outline of the lesion and does not include the lesion tissue, the accuracy of the generated sampling grid can be improved, which is beneficial to subsequent pathological sampling.

In an alternative embodiment of the present application, fig. 8 shows a flowchart of an image generation method provided in an exemplary embodiment of the present application, the method further comprising the steps of:

step 402, a binary mask image of a medical image is acquired.

Illustratively, the binary mask (mask) image refers to an image obtained by subjecting a medical image to graying and binarization, and the binary mask image of the medical image may assist in performing lesion contour analysis.

Alternatively, in a binary mask image of a medical image, the pixel value of the lesion tissue may be set to 255 and the pixel value of healthy tissue located outside the lesion contour of the lesion tissue may be set to 0.

Optionally, a binary mask image of the medical image is acquired.

At step 404, at least two contour points on the lesion contour in the binary mask image are determined.

Illustratively, at least two contour points on the lesion contour in the binary mask image are determined. Wherein, 8 adjacent pixel points of the bright point in the binary mask image are all bright points, the bright point is the internal point of the focus contour, otherwise the contour point of the focus contour point.

Step 406, calculating a distance between each two contour points of the at least two contour points.

Illustratively, after determining at least two contour points, a distance between each two contour points of the at least two contour points is calculated.

Step 408, determining a long axis of the lesion contour based on the distance, the long axis being used to assist in determining clinical medical parameters of the lesion tissue.

Illustratively, the line segment corresponding to the longest distance is determined as the long axis of the lesion contour, which is used to assist in determining clinical medical parameters of the lesion tissue.

Alternatively, in the case where the lesion contour is in a relatively uniform circular or elliptical shape, the center point of the major axis is determined as the center point of the lesion contour.

In this embodiment, the long axis of the lesion outline is determined based on the binary mask image of the medical image, which can assist in determining clinical medical parameters of the lesion tissue, is favorable for generating a sampling grid, can assist a doctor in performing pathological analysis on a pathological sampling area, and provides effective guidance for the subsequent treatment plan formulation.

In an alternative embodiment of the present application, fig. 9 shows a flowchart of an image generating method provided in an exemplary embodiment of the present application, the method further comprising:

step 502, a perpendicular bisector of a long axis of a lesion contour is determined.

Illustratively, the perpendicular bisector of the long axis is a line perpendicular to the long axis and bisecting the long axis. Optionally, a perpendicular bisector of the long axis of the lesion contour is determined based on a center point of the long axis of the lesion contour.

Step 504, determining a short axis of the lesion contour based on the intersection of the perpendicular bisector and the lesion contour.

Illustratively, a line segment corresponding to the two intersection points is determined as a short axis of the lesion contour based on the intersection point of the perpendicular bisector and the lesion contour. The short axis is used to assist in determining clinical medical parameters of the focal tissue.

Step 506, determining a long axis length and a short axis length of the focus outline according to the pixel lengths of the long axis and the short axis on the medical image and the space distance between each pixel of the medical image, wherein the long axis length and the short axis length are used for assisting in pathological analysis of the pathological sampling area.

The long axis length and the short axis length refer to the long axis actual length and the short axis actual length of the focus outline, belong to clinical medical parameters, are mainly used for representing the size of focus tissues and assisting doctors in carrying out pathological analysis on pathological sampling areas. Among these, pathological analyses include, but are not limited to, whether the lesion tissue is analyzed for pathological nature, inflammation or tumor, benign or malignant, and the like.

Optionally, the long axis length of the lesion contour is determined by calculating the product of the pixel length of the long axis and the spatial distance between each pixel of the medical image based on the pixel length of the long axis on the medical image and the spatial distance between each pixel of the medical image.

Optionally, a short axial length of the lesion contour is determined by calculating a product of the pixel length of the short axis and the spatial distance between each pixel of the medical image based on the pixel length of the short axis on the medical image and the spatial distance between each pixel of the medical image.

In some embodiments, the spatial distance between each pixel of the medical image may be determined from a camera parameter of the medical image, which may be an internal reference.

Optionally, the spatial resolution of the medical image under internal reference of the different cameras, i.e. the actual physical spatial distance represented between each pixel of the medical image, is measured in advance. After the medical image is acquired, an internal reference in camera parameters of the medical image is acquired, and a spatial resolution corresponding to the internal reference is determined, so that a spatial distance between each pixel of the medical image can be determined.

In some embodiments, the method further comprises: and determining the actual area of the focus outline according to the pixel area of the focus outline in the medical image and the actual area of the pixel corresponding to each pixel of the medical image, wherein the actual area of the focus outline is used for assisting in pathological analysis of a pathological sampling area.

Optionally, the product of the pixel area and the pixel actual area is calculated according to the pixel area of the focus contour in the medical image and the pixel actual area corresponding to each pixel of the medical image, so as to determine the actual area of the focus contour.

In the embodiment, the size of the focus tissue can be accurately represented by determining clinical medical parameters such as long axial length, short axial length and area, thereby assisting doctors in carrying out pathological analysis on pathological material taking areas and providing effective guidance for subsequent treatment planning.

As an example, fig. 10 shows a schematic diagram of an image generation method provided by an exemplary embodiment of the present application. Fig. 10 shows a binary mask image of a medical image, with white portions of the binary mask image representing focal tissue and black portions representing healthy tissue. Contour points on the lesion contour in the binary mask image were determined to be A, B, C, D, E, F, G, H, I, J. And calculating the distance between every two contour points, and determining a line segment AE corresponding to the maximum distance AE as the long axis of the focus contour. Next, a perpendicular bisector of the long axis AE of the lesion contour is determined, and a line segment corresponding to an intersection of the perpendicular bisector and the lesion contour is used to determine the short axis of the lesion contour. Finally, the long axis length and the short axis length of the focus outline can be respectively determined according to the pixel lengths of the long axis and the short axis on the medical image and the space distance between each pixel of the medical image, and the long axis length and the short axis length are used for assisting a doctor in carrying out subsequent pathological analysis on a pathological sampling area corresponding to the focus outline.

In an alternative embodiment of the present application, the sub-grid parameters of the at least two sub-grids comprise at least one of a size, a shape, an area, an interior angle size of the sub-grid; wherein the sub-grid parameters are fixed by default; or, the sub-grid parameters are preset; alternatively, the sub-grid parameters are dynamically changed. The selection can be specifically performed according to the actual technical requirements so as to improve the accuracy of the generated material-drawing grid.

Illustratively, in the case where the sub-grid parameters are dynamically changed, fig. 11 shows a flowchart of an image generating method provided in an exemplary embodiment of the present application, the method further includes:

in step 602, camera parameters of the medical image are acquired, the camera parameters including at least one of internal parameters, external parameters, and shooting distance.

Illustratively, a medical image is obtained by photographing focal tissue with a photographing assembly. The parameters of the shooting component are called as camera parameters, and the camera parameters comprise at least one of internal parameters, external parameters and shooting distances.

Optionally, the internal parameters include at least one of a focal length, a pixel size, and an internal parameter matrix, the external parameters include at least one of a camera position, a rotation matrix, and a translation matrix, and the photographing distance refers to a distance between a lens of the photographing assembly and the focal tissue.

Step 604 adjusts sub-grid parameters based on camera parameters of the medical image.

For example, based on camera parameters of a medical image, sub-grid parameters may be adjusted in real time to make the partitioned material grid more suitable for the medical image.

In this embodiment, the parameters of the subgrid can be adjusted in real time, so that the partitioned material-drawing grid is more suitable for medical images, and the accuracy of the generated material-drawing grid is improved.

In some embodiments, after the generation of the sampling grid for meshing the pathological sampling region, there may be a case where a part of the sub-grids contain very little lesion tissue and contain very much healthy tissue, and if the part of the sub-grids is subjected to pathological sampling, medical resources may be wasted. In this case, the method further comprises: and carrying out grid combination on at least two sub-grids which are in the material-drawing grids and meet combination conditions. After merging, the sub-grid parameters of at least a part of the sub-grids in the material-drawing grids are different from those of other sub-grids, namely, the sub-grid parameters of all the sub-grids in the material-drawing grids are not completely the same.

Exemplary, fig. 12 shows a flowchart of an image generating method according to an exemplary embodiment of the present application, where mesh merging is performed on at least two sub-meshes in a material mesh, where the sub-meshes meet a merging condition, including:

A first sub-grid is determined from the at least two sub-grids based on the size of the lesion tissue comprised by the at least two sub-grids, step 701.

Illustratively, the first sub-grid is a sub-grid that requires grid merging. Typically, the first sub-grid is a sub-grid located in the edge region of the lesion outline, the first sub-grid containing both lesion tissue and healthy tissue.

Optionally, the first submesh is automatically identified. At this time, a first sub-grid is determined from the at least two sub-grids based on the size of the lesion tissue contained in the at least two sub-grids. For example, a sub-grid containing lesion tissue of a size less than one-fourth of the entire sub-grid size is determined as the first sub-grid. Alternatively, a sub-grid of lesion tissue of a size smaller than that of healthy tissue is determined as the first sub-grid.

Optionally, the first submesh is manually delineated by a physician. At this time, in response to the sub-grid selection operation, a first sub-grid is determined from at least two sub-grids. In this case, the first sub-grid may be determined by the physician based on the actual technical needs.

Step 702, determining a second sub-grid closest to the first sub-grid from other sub-grids in the at least two sub-grids except the first sub-grid.

Illustratively, the second sub-grid is a sub-grid that may be merged with the first sub-grid. The second sub-grid has a common edge with the first sub-grid. The second sub-grid comprises one.

Optionally, a second sub-grid closest to the first sub-grid is determined from other sub-grids of the at least two sub-grids than the first sub-grid.

Step 705, mesh merging is performed on the first sub-mesh and the second sub-mesh.

Optionally, the first sub-grid and the second sub-grid are combined to realize optimization of the material-drawing grid.

Step 703, determining a third sub-grid which is closest to the first sub-grid and contains the largest focus organization from other sub-grids except the first sub-grid in the at least two sub-grids.

Illustratively, the third sub-grid is a sub-grid that may be merged with the first sub-grid. The third sub-grid has a common edge with the first sub-grid. The third sub-grid comprises one. In the case where the second sub-grid is one or more, that is, the second sub-grid closest to the first sub-grid is one or more, the third sub-grid may be the sub-grid having the largest lesion tissue among the second sub-grids.

Optionally, a third sub-grid closest to the first sub-grid and containing the largest focal tissue is determined from other sub-grids of the at least two sub-grids than the first sub-grid.

Step 706, mesh merging is performed on the first sub-mesh and the third sub-mesh.

Optionally, the first sub-grid and the third sub-grid are combined to realize optimization of the material-drawing grid.

Step 704, determining a fourth sub-grid closest to the first sub-grid and closest to the target direction of the center point of the focus contour from other sub-grids except the first sub-grid in the at least two sub-grids.

The fourth sub-grid is, for example, a sub-grid that may be grid-consolidated with the first sub-grid. The fourth sub-grid has a common edge with the first sub-grid. The fourth sub-grid comprises one. If the third subgrid is one or more, that is, if the third subgrid closest to the first subgrid and including the lesion tissue having the largest structure is one or more, the fourth subgrid may be one closest to the target direction of the center point of the lesion contour, which is determined from the third subgrid.

Optionally, the target directional distance refers to a lateral or longitudinal axis component of the distance, and the target directional distance from the center point of the lesion contour refers to a lateral or longitudinal axis component of the distance between the center point of the sub-grid and the center point of the lesion contour.

Optionally, a fourth sub-grid closest to the first sub-grid and closest to the target direction of the center point of the lesion contour is determined from other sub-grids of the at least two sub-grids than the first sub-grid.

Step 707, mesh merging is performed on the first sub-mesh and the fourth sub-mesh.

Optionally, the first sub-grid and the fourth sub-grid are combined to realize optimization of the material-drawing grid.

In some embodiments, the above three merging modes may be selected according to practical technical needs, for example, when the first sub-grid only has one nearest sub-grid, grid merging may be performed by determining the second sub-grid. When more than one nearest sub-grid exists in the first sub-grid and the sizes of focus tissues contained in each nearest sub-grid are different, grid combination can be performed by determining the third sub-grid. When more than one nearest sub-grid exists in the first sub-grid and the sizes of focus tissues contained in each nearest sub-grid are the same, grid combination can be performed by determining a fourth sub-grid.

In this embodiment, optimization of the material-drawing grid can be achieved by performing grid combination on at least two sub-grids conforming to the combination condition in the material-drawing grid. By adopting different merging modes to merge grids, the accuracy of grid merging can be improved, and the optimization effect of the obtained grids can be improved, so that the method is beneficial to obtaining the subsequent pathological materials.

In some embodiments, at least two sub-grids of the material-drawing grids, which meet the merging condition, are first grid-merged, and then at least two sub-grids of the material-drawing grids are numbered to identify at least two sub-grids of the material-drawing grids.

As an example, fig. 13 shows a schematic diagram of an image generating method provided by an exemplary embodiment of the present application. A schematic diagram of a material drawing grid for meshing a pathological material drawing region is shown in fig. 13-1. Based on the size of the lesion tissue contained in the at least two sub-grids, determining a first sub-grid needing to be grid-combined from the at least two sub-grids, wherein the first sub-grid in the embodiment is sub-grid y.

Merging mode one: and determining a second sub-grid closest to the first sub-grid from other sub-grids except the first sub-grid in the at least two sub-grids, wherein when the second sub-grid is only sub-grid x closest to the first sub-grid, the second sub-grid is sub-grid x, and combining the first sub-grid y with the second sub-grid x.

Combining mode II: determining a third sub-grid which is closest to the first sub-grid and contains the focus organization which is the largest from other sub-grids except the first sub-grid in at least two sub-grids, wherein the third sub-grid is the sub-grid x when the focus organization contained in the sub-grid x is the largest in the sub-grid x in the fig. 13-1, and combining the first sub-grid y with the third sub-grid x.

And combining mode III: a fourth sub-grid closest to the first sub-grid and closest to the target direction of the center point of the lesion contour is determined from the other sub-grids other than the first sub-grid among the at least two sub-grids. In this embodiment, the target direction distance is set as the longitudinal axis component of the distance, the distance between the sub-grid x in fig. 13-1 and the center point O of the lesion contour is the line segment OM, and the target direction distance is the longitudinal axis component of the line segment OM. The distance between the sub-grid z and the center point O of the focus contour is a line segment ON, the target direction distance is the longitudinal axis component of the line segment ON, wherein the longitudinal axis component of the distance from the center point of the focus contour is the longitudinal axis component of the line segment OM, the fourth sub-grid is a sub-grid x, and the first sub-grid y and the fourth sub-grid x are combined in a grid mode.

The grid-merged and removed grid-drawn material is shown in fig. 13-2, and the grid-drawn material is shown in fig. 13-3, wherein at least two sub-grids in the grid-drawn material are numbered, and the grid-drawn material can be finally used for display.

In an alternative embodiment of the present application, fig. 14 is a flowchart of an image generating method according to an exemplary embodiment of the present application, after obtaining a medical image of a focal tissue in step 320, before identifying a pathological sampling area corresponding to a focal contour in step 340, further including:

Step 321, identifying a focus contour of focus tissue in the medical image through the AI model.

Optionally, the lesion contour of the lesion tissue in the medical image is automatically identified. Specifically, a lesion contour of a lesion tissue in a medical image is identified by an AI model. The artificial intelligence AI model is a pre-trained neural network model, which may be at least one of the following types: image segmentation Unet model and semantic segmentation model deep V < 3+ > model.

In some embodiments, the method further comprises: a lesion contour of a lesion tissue in the medical image is displayed.

Illustratively, after automatically identifying the lesion contour, the lesion contour of the lesion tissue in the medical image is displayed to assist the doctor in knowing whether the automatically identified lesion contour is accurate, facilitating manual modification or re-delineation.

In the embodiment, through the pre-trained AI model, the automatic identification of the focus outline of focus tissues in the medical image can be realized, and the processing efficiency of the medical image is improved.

In an alternative embodiment of the present application, although most of the focal tissue requiring morse surgery is flattened after resection, the base of some cases may have a large curvature, which may result in some error in the grid of material. Based on this, please continue to refer to fig. 14, after obtaining the medical image of the focal tissue in step 320, before identifying the pathological material region corresponding to the focal contour in step 340, the method further includes:

Step 322, a three-dimensional point cloud image of the focal tissue is acquired.

For example, when the focal tissue is in a rugged or highly curved position, such as in a nose, a head, a joint, etc., 3D information of the focal tissue may be further introduced for improving accuracy of medical image processing.

Optionally, a three-dimensional point cloud image of the focal tissue is acquired.

Step 323, generating a three-dimensional model of the focus tissue based on the three-dimensional point cloud image, and flattening the three-dimensional model to obtain a two-dimensional flattened image of the focus tissue.

Illustratively, a three-dimensional model (3D mesh) of the focal tissue is generated based on the three-dimensional point cloud image, and the three-dimensional model is flattened to obtain a two-dimensional flattened image (UV Field) of the focal tissue.

Optionally, a rolling sphere method is used to generate the three-dimensional model, and a flattening algorithm (Boundary First Flatten, BFF) is used to flatten the three-dimensional model. Wherein, the point cloud of the three-dimensional model has a one-to-one correspondence with the pixel points in the two-dimensional flattened image, the pixels in the medical image have a one-to-one correspondence with the pixels in the two-dimensional flattened image.

At step 324, a lesion contour of the lesion tissue is determined on the two-dimensional flattened image.

Optionally, a lesion contour of the lesion tissue is determined on the two-dimensional flattened image based on a one-to-one correspondence of pixels in the medical image and pixels in the two-dimensional flattened image. Therefore, the subsequent identification of the pathological material-drawing area corresponding to the lesion outline and the generation of the material-drawing grid for grid division of the pathological material-drawing area are performed on the basis of the lesion outline on the two-dimensional flattened image.

In some embodiments, after the three-dimensional model of the focal tissue is generated based on the three-dimensional point cloud image, a sampling grid for meshing the pathological sampling area may be directly generated on the three-dimensional model, and the specific generation mode in this case is not limited in the embodiments of the present application.

In this embodiment, by further introducing three-dimensional information of the focal tissue, the algorithm accuracy can be further improved, and the accuracy of the generated material-drawing grid can be improved.

In an alternative embodiment of the present application, please continue to refer to fig. 14, the method further includes:

step 325, equidistantly expanding the outline of the focus to determine the outline of the material, wherein the outline of the material is used for determining the pathological material taking area.

Illustratively, the profile of a lesion refers to the profile of a lesion. Optionally, the focal contour is equidistantly expanded to determine a sampling contour, and the sampling contour is used for determining a pathological sampling area, so that a sampling grid generated subsequently can be fully contained in the edge area of focal tissue, and the leakage of the edge area is prevented.

Optionally, the equidistant expansion distance is set according to the actual technical requirement. The method can be set to be expanded by about 2mm in combination with clinical practice.

Optionally, coordinates of at least two contour points on the lesion contour are determined, and based on the coordinates of the at least two contour points on the lesion contour, the lesion contour is equidistantly expanded by adopting a polygon scaling algorithm (Vatti's Clipping Algorithm), and the material drawing contour is determined.

In this example, the profile of the lesion is determined by equidistant flaring of the profile, which is used to determine the pathological region of interest. Can ensure that the material-drawing grids generated in the follow-up process can be fully contained in the edge area of the focus tissue, and prevent the focus tissue in the edge area from being missed.

In some embodiments, it may also be possible to extrapolate the lesion contours non-equidistantly, to determine the material profile. For example, a relatively smooth portion of the lesion contour is equally-flared, and a relatively non-smooth portion of the lesion contour is non-equally-flared, such that the lesion contour is relatively smooth overall, to facilitate the partitioning of the mesh of material.

In some embodiments, fig. 15 shows a flowchart of an image generation method provided by an exemplary embodiment of the present application, after generating a material grid for meshing a pathological material region, before displaying a medical image covered with the material grid, further comprising:

And 371, drawing the material drawing grids on the three-dimensional model to obtain the three-dimensional model drawn with the material drawing grids.

Optionally, in the case that the generation of the sampling grid for meshing the pathological sampling region is performed on the basis of the lesion outline on the two-dimensional flattened image, the sampling grid is drawn onto the three-dimensional model, and the three-dimensional model on which the sampling grid is drawn is obtained.

Optionally, drawing the material drawing grid on the three-dimensional model based on the one-to-one correspondence relation between the point cloud of the three-dimensional model and the pixel points in the two-dimensional flattened image, so as to obtain the three-dimensional model drawn with the material drawing grid.

Step 372, rendering the three-dimensional model drawn with the material drawing grids to obtain a medical image covered with the material drawing grids.

Optionally, a 3D rendering mode in computer vision technology is adopted to render (render) the three-dimensional model drawn with the material drawing grid, and a medical image covered with the material drawing grid is obtained.

Optionally, the 3D rendering mode includes at least one of scan line rendering and optical energy transmission.

In this embodiment, the three-dimensional model drawn with the material drawing grids is rendered to obtain the medical image covered with the material drawing grids, so that the material drawing grids can be more attached to the region of the focal tissue in the medical image, and meanwhile, the problem of inaccuracy in material drawing grids at the region of the uneven focal tissue can be solved.

In some embodiments, after pathological sampling is performed, the sampled tissue may be rapidly paraffin-tableted, fabricated into pathological sections, and examined under a microscope for residual lesions, a process also known as rapid pathology. And under the condition that focus residues exist, performing secondary excision and secondary pathological sampling on the corresponding positions. For rapid pathology detection, the detection and treatment time still takes 4 to 6 hours, during which a certain change of focal tissue is unavoidable. Therefore, it is necessary to accurately locate the position of a pathologically-drawn region where a lesion residue exists on a lesion tissue where a certain movement change occurs.

Optionally, the method further comprises: acquiring a second medical image of the focal tissue, the second medical image being different from at least a portion of the focal contours in the medical image; and displaying a second medical image with a second material drawing grid, wherein the second material drawing grid is provided with at least two sub-grids corresponding to the material drawing grid, and pathological material drawing areas covered by the same sub-grid in the two medical images are the same.

The second medical image is an image obtained by taking a second time after the lesion tissue is resected by the Morse operation. Illustratively, after rapid pathology, in the case where it is judged that there is still a lesion residual by microscopic pathology detection, secondary excision and secondary pathology sampling are required. The second time refers to any point in time after the rapid pathology.

Illustratively, the second medical image is different from at least a portion of the lesion contour in the medical image. Reasons for the difference in at least a portion of the lesion contours include at least one of: the change in position of the camera assembly, the change in camera parameters of the camera assembly, the change in focal tissue itself, the change in position of the patient.

In some embodiments, the focal tissue is imaged at a second time by the same imaging assembly as the medical image, resulting in a second medical image of the focal tissue at the second time.

The second material-drawing grid is obtained by transforming the material-drawing grid based on the image registration parameter, and is not regenerated based on the second medical image. The image registration parameters are configured based on the medical image and the second medical image, the image registration parameters are used for transforming the material drawing grids, the types of the image registration parameters are also used for indicating the transformation modes of the material drawing grids, the types of the image registration parameters are different, and the transformation modes of the material drawing grids are different.

Illustratively, the second mesh has at least two sub-meshes corresponding to the meshes, and the same sub-mesh covers the same pathological area of material in both medical images. The sub-grid parameters of at least two sub-grids in the second material-drawing grid are the same as or different from those of at least two sub-grids in the material-drawing grid, and the sub-grid parameters of at least two sub-grids in the second material-drawing grid are the same as or different from those of at least two sub-grids in the material-drawing grid. The sub-grid parameters include at least one of a size, shape, area, and interior angle size of the sub-grid.

Optionally, a second medical image with a second mesh of material is displayed. The second material-drawing grid is provided with at least two sub-grids corresponding to the material-drawing grid, and the pathological material-drawing areas covered by the same sub-grid in the two medical images are the same.

For example, the sub-grids in the sampling grids are square, a first sub-grid in the sampling grids covers a central position area of focus tissue in the medical image, then the sampling grids are transformed to obtain second sampling grids, the sub-grids in the second sampling grids can be diamond-shaped, the second sub-grids corresponding to the first sub-grids cover a central position area of focus tissue in the second medical image.

In some embodiments, the second mesh has a number of at least two sub-meshes corresponding to the mesh, and the same sub-mesh is numbered the same in both medical images. Therefore, doctors can refer to the numbers of the sampling grids to determine the same pathological sampling area characterized by the same numbers in two medical images, and secondary excision and secondary pathological sampling are conveniently carried out on the pathological sampling area when focus residues exist in the pathological sampling area.

In some embodiments, the method further comprises: and displaying the numbers of at least two sub-grids in the second material drawing grid to assist doctors to accurately reposition specific material drawing positions.

In some embodiments, the method further comprises: determining a first sub-grid corresponding to the pathological sampling area with focus residues in the sampling grids, displaying a second sub-grid corresponding to the first sub-grid in the second sampling grid, and covering the pathological sampling area with focus residues by the second sub-grid.

Optionally, after the rapid pathology is performed, the pathological sampling area with focus residue can be determined, in this case, all sampling grids may not be transformed and repositioned, and only the corresponding position of the pathological sampling area with focus residue in the focus tissue after the rapid pathology is transformed and repositioned, so as to assist the doctor to perform secondary excision and secondary pathological sampling on the position.

Alternatively, the second-derived mesh may be applied during a Morse procedure, and may also be referred to as a second Morse (Mohs) mesh.

Fig. 16 shows a flowchart of an image generation method according to an exemplary embodiment of the present application, which is applied to the terminal 120 shown in fig. 1 or a client supporting image processing installed on the terminal 120 for illustration, and includes:

In step 820, a medical image of the focal tissue is acquired, the medical image including a focal contour of the focal tissue.

By way of example, focal tissue refers to tissue having pathogenic microorganisms, any tissue organ may become focal, for example, focal tissue may be tumor tissue. Lesion contour refers to a continuous contour of the lesion tissue. The lesion contour may be regular or irregular in shape. The medical image is an image obtained by taking a picture of tumor tissue after the tumor tissue is resected by the Morse operation.

Optionally, the medical image of the focal tissue is a two-dimensional image, and the terminal acquires the medical image of the focal tissue, the medical image including a focal contour of the focal tissue.

Step 840, displaying a medical image covered with a material drawing grid comprising at least two sub-grids covering a pathological material drawing area corresponding to the outline of the focus and arranged in an array; the pathological sampling area covered by each sub-grid contains focus tissues.

Illustratively, the server identifies a pathologically-drawn region corresponding to the lesion outline and generates a draw grid for meshing the pathologically-drawn region. Thus, the terminal displays the medical image covered with the sampling grid, and the sampling grid comprises at least two sub-grids which cover the pathological sampling area corresponding to the focus outline and are arranged in an array; the pathological sampling area covered by each sub-grid contains focus tissues.

In some embodiments, the method further comprises: a medical image of the focal tissue is displayed.

The terminal also displays the medical image of the focus tissue after acquiring the medical image of the focus tissue, so as to assist doctors in knowing whether the medical image is clear, the shooting position is accurate, whether the medical image contains all focus tissues, and the like, thereby improving the accuracy of medical image processing.

The computer device, after automatically identifying the lesion outline in the background, displays the lesion outline of the lesion tissue in the medical image by the terminal, so as to assist doctors in knowing whether the automatically identified lesion outline is accurate, and facilitate manual modification or re-sketching.

In some embodiments, the sampling grid may also be projected onto the pathological sampling region by a projection component, the sampling grid being displayed on the pathological sampling region. In this case, the terminal may not need to display the medical image covered with the material drawing grid.

In summary, according to the method provided by the embodiment of the present application, by acquiring a medical image of a focal tissue, the medical image including a focal contour of the focal tissue, and then displaying the medical image covered with a sampling grid, where the sampling grid includes at least two sub-grids covering a pathological sampling area corresponding to the focal contour and arranged in an array; the pathological sampling area covered by each sub-grid contains focus tissues. Accordingly, the drawing grids can be automatically generated without manual sketching of doctors, errors and randomness of manual planning of the drawing grids can be reduced to the greatest extent, a uniform and homogeneous division scheme of the drawing grids is provided, doctors can be assisted in pathological drawing, errors are avoided in drawing, focus tissues can be obtained in each sub-grid, efficiency of pathological drawing is improved in an auxiliary mode, and doctors can be assisted in making subsequent pathological analysis and treatment plans.

As an example, next, an overall flow of the image generation method of the embodiment of the present application will be described.

Referring to fig. 17 and 18, the present embodiment provides an image generating method based on two-dimensional information, and referring to fig. 19 and 20, the present embodiment also provides an image generating method based on two-dimensional information and three-dimensional information.

Alternatively, when the focal tissue is located at a relatively flat or less curved position, such as in the four limbs, trunk, etc., an image generating method based on two-dimensional information may be used to process the medical image. When the focal tissue is in uneven position or has larger curvature, such as in the nose, head, joints and other parts, three-dimensional information can be introduced, and an image generation method based on the two-dimensional information and the three-dimensional information is adopted to process medical images.

1. Image generation method based on two-dimensional information

Step 10, obtaining a medical image of focal tissue, the medical image including a focal contour of the focal tissue.

Alternatively, the focal tissue is photographed by an industrial camera or an RGB camera, and a medical image of the focal tissue is obtained, the medical image including a focal contour of the focal tissue, as shown in fig. 17-1. Wherein the medical image shown in fig. 17-1 may be displayed on a terminal.

And 11, identifying the focus outline of focus tissues in the medical image through an AI model.

Optionally, the lesion contours of the lesion tissue in the medical image are identified by pre-trained image segmentation Unet model and/or semantic segmentation model Deeplab V3+ model, as shown in FIG. 17-2. Wherein the lesion outline shown in fig. 17-2 may be displayed on the terminal.

And 12, equidistantly expanding the outline of the focus, determining the outline of the material, wherein the outline of the material is used for determining the pathological material taking area, and identifying the pathological material taking area.

Alternatively, a polygonal scaling algorithm (Vatti's Clipping Algorithm) is used to scale the lesion contour equally, set the scale-out by 2mm, determine the sampling contour, and the sampling contour is used to determine the pathological sampling region, and identify the pathological sampling region, as shown in fig. 17-3. Wherein the profile of the material shown in fig. 17-3 may be displayed on the terminal.

Step 13, generating a sampling grid for meshing a pathological sampling area, wherein the sampling grid comprises at least two sub-grids which cover the pathological sampling area and are arranged in an array; the pathological sampling area covered by each sub-grid contains focus tissues.

Optionally, after the generation of the material-drawing grid, at least two sub-grids in the material-drawing grid are numbered with numbers, and the final material-drawing grid is shown in fig. 17-4. The drawing grids, four quadrants, coordinate axes and numbers shown in fig. 17-4 can be displayed on the terminal.

In some embodiments, in actual clinical applications, the material grid comprises at least four sub-grids, i.e. at least one sub-grid in each of the four quadrants. After the sub-grids in the grid are combined, the number of sub-grids may be less than four, in which case the user is generally required to reset the sub-grid parameters.

Step 14, displaying the medical image covered with the material drawing grid.

Optionally, the medical image covered with the sampling grid is displayed through a display screen of the terminal, virtual reality VR glasses, or the like, or the sampling grid is projected onto the pathological sampling area through the projection component.

2. Image generation method based on two-dimensional information and three-dimensional information

In step 20, a medical image of the focal tissue is acquired, the medical image comprising a focal contour of the focal tissue.

Alternatively, the focal tissue is photographed by an RGBD camera, and a medical image of the focal tissue is obtained, where the medical image includes a focal contour of the focal tissue, as shown in FIG. 19-1. Wherein the medical image shown in fig. 19-1 may be displayed on a terminal.

In step 22, a lesion contour of a lesion tissue in the medical image is identified by the AI model.

Optionally, the lesion contours of the lesion tissue in the medical image are identified by pre-trained image segmentation Unet model and/or semantic segmentation model Deeplab V3+ model, as shown in FIG. 19-2. Wherein the lesion outline shown in fig. 19-2 may be displayed on the terminal.

And step 21, acquiring a three-dimensional point cloud image of focus tissues.

Optionally, the focal tissue is photographed by an RGBD camera, and a three-dimensional point cloud image of the focal tissue is obtained, as shown in fig. 19-4.

And step 23, generating a three-dimensional model of the focus tissue based on the three-dimensional point cloud image, and flattening the three-dimensional model to obtain a two-dimensional flattened image of the focus tissue.

Alternatively, a three-dimensional model (3D mesh) of the lesion tissue is generated using a rolling sphere method based on the three-dimensional point cloud image, as shown in fig. 19 to 5. The three-dimensional model is flattened using a flattening algorithm (Boundary First Flatten, BFF) to obtain a two-dimensional flattened image (UV Field) of the focal tissue, as shown in fig. 19-6. Wherein, the point cloud of the three-dimensional model has a one-to-one correspondence with the pixel points in the two-dimensional flattened image, the pixels in the medical image have a one-to-one correspondence with the pixels in the two-dimensional flattened image. Wherein the two-dimensional flattened image shown in fig. 19-6 may be displayed on a terminal.

Alternatively, the above steps 20 and 22 may be performed in parallel with the steps 21 and 23.

And step 24, determining the focus outline of the focus tissue on the two-dimensional flattened image.

Alternatively, a lesion contour of the lesion tissue is determined on the two-dimensional flattened image based on a one-to-one correspondence of pixels in the medical image and pixels in the two-dimensional flattened image, as shown in fig. 19-7. Wherein the lesion contours on the two-dimensional flattened image shown in fig. 19-7 may be displayed on a terminal end.

And 25, equidistantly expanding the focus outline, determining a sampling outline, wherein the sampling outline is used for determining a pathological sampling area, and identifying the pathological sampling area.

Optionally, a polygonal scaling algorithm (Vatti's Clipping Algorithm) is adopted to perform equidistant external expansion on the lesion contour, set the equidistant external expansion for 2mm, determine a sampling contour, and the sampling contour is used for determining a pathological sampling area to identify the pathological sampling area. Wherein the profile of the material may be displayed on the terminal.

Alternatively, the above steps may also be to identify the focal contour of the focal tissue in the medical image, as shown in fig. 19-2, then to expand the focal contour equidistantly, to determine the material drawing contour, as shown in fig. 19-3, and then to determine the material drawing contour on the two-dimensional flattened image, as shown in fig. 19-7.

Step 26, generating a sampling grid for meshing a pathological sampling area, wherein the sampling grid comprises at least two sub-grids which cover the pathological sampling area and are arranged in an array; the pathological sampling area covered by each sub-grid contains focus tissues.

Alternatively, a material-drawing grid for meshing the pathological material-drawing region is generated, as shown in fig. 19 to 8. After the material-drawing grids are generated, at least two sub-grids in the material-drawing grids are numbered by numbers.

And step 27, drawing the material drawing grids on the three-dimensional model to obtain the three-dimensional model drawn with the material drawing grids.

And 28, rendering the three-dimensional model drawn with the material drawing grids to obtain a medical image covered with the material drawing grids.

Optionally, a 3D rendering mode in computer vision technology is adopted to render the three-dimensional model drawn with the material drawing grid, and a medical image covered with the material drawing grid is obtained, as shown in fig. 19-9. The 3D rendering mode comprises at least one of scanning line rendering and light energy transmission. Wherein two medical images covered with a mesh of material shown in fig. 19-9 may be displayed on the terminal.

Step 29, displaying the medical image covered with the material drawing grid.

Fig. 21 is a block diagram showing a configuration of an image generating apparatus according to an exemplary embodiment of the present application, the apparatus including:

an acquisition module 910 is configured to acquire a medical image of a focal tissue, where the medical image includes a focal contour of the focal tissue.

The identifying module 920 is configured to identify a pathological sampling area corresponding to the lesion contour.

A generating module 930, configured to generate a sampling grid for meshing the pathological sampling region, where the sampling grid includes at least two sub-grids covering the pathological sampling region and arranged in an array; the lesion tissue is contained in the pathological sampling area covered by each sub-grid.

A display module 940 for displaying the medical image covered with the material drawing grid.

In one example, the generating module 930 is configured to:

and generating a sampling grid for meshing the pathological sampling area based on at least one reference point in the medical image.

In one example, the generating module 930 is configured to:

determining a center point of the lesion contour;

expanding to generate at least two sub-grids arranged in an array by taking the central point as the reference point;

under the condition that intersection exists between the at least two sub-grids and the whole outline of the focus outline, ending expansion to generate the at least two sub-grids, and obtaining a sampling grid for carrying out grid division on the pathological sampling area;

and under the condition that intersection does not exist between the at least two sub-grids and at least part of the focus outline, using the newly added sub-grid vertexes of the at least two sub-grids generated by the expansion of the current round as a next round of reference points, and continuously using the next round of reference points as expansion points to expand and generate the at least two sub-grids arranged in the array.

In one example, the generating module 930 is configured to:

and under the condition that intersection does not exist between the at least two sub-grids and at least part of the focus contour, using newly added sub-grid vertexes of the at least two sub-grids generated by the expansion of the current round as a next round of reference points, and continuously using the next round of reference points as expansion points to expand and generate the at least two sub-grids which are distributed in a shape of a Chinese character 'tian' around the expansion points.

In one example, the generating module 930 is configured to:

determining a center point of the lesion contour;

determining the center point as the reference point, and determining a coordinate system based on the reference point;

calculating sub-grid vertex coordinates of at least two sub-grids arranged in an array in the coordinate system;

and generating a sampling grid for meshing the pathological sampling area based on the vertex coordinates of each sub-grid.

In one example, the generating module 930 is configured to:

correspondingly generating a sub-grid based on at least three sub-grid vertex coordinates; at least one edge of the subgrid is parallel to a target axis of the coordinate axes;

and determining a target sub-grid in at least two sub-grids as a sub-grid belonging to the material-drawing grid, wherein at least one sub-grid vertex of the target sub-grid is positioned in the focus outline.

In one example, the apparatus further comprises: a removal module;

in one example, the removal module is configured to:

removing repeated sub-grids in the at least two sub-grids;

or alternatively, the first and second heat exchangers may be,

removing a sub-grid of the at least two sub-grids that does not have an intersection with the lesion outline and does not contain the lesion tissue.

In one example, the apparatus further comprises: a parameter determination module;

in one example, the parameter determination module is configured to:

acquiring a binary mask image of the medical image;

determining at least two contour points on the lesion contour in the binary mask image;

calculating the distance between every two contour points of the at least two contour points;

a long axis of the lesion contour is determined based on the distance, the long axis being used to assist in determining clinical medical parameters of the lesion tissue.

In one example, the parameter determination module is configured to:

determining a perpendicular bisector of the long axis of the lesion contour;

determining a minor axis of the lesion contour based on an intersection of the perpendicular bisector and the lesion contour;

and respectively determining the long axis length and the short axis length of the focus outline according to the pixel lengths of the long axis and the short axis on the medical image and the space distance between each pixel of the medical image, wherein the long axis length and the short axis length are used for assisting in pathological analysis of the pathological sampling area.

In one example, the sub-grid parameters of the at least two sub-grids include at least one of a size, a shape, an area, an interior angle size of the sub-grid;

Wherein the sub-grid parameters are constant by default; or, the sub-grid parameters are preset; alternatively, the sub-grid parameters are dynamically changed.

In one example, the apparatus further comprises: an adjustment module;

in one example, the adjustment module is configured to:

acquiring camera parameters of the medical image, wherein the camera parameters comprise at least one of internal parameters, external parameters and shooting distances;

the sub-grid parameters are adjusted based on the camera parameters of the medical image.

In one example, the apparatus further comprises: a merging module;

in one example, the merging module is configured to:

and carrying out grid combination on at least two sub-grids which meet combination conditions in the material-drawing grids.

In one example, the merging module is configured to:

determining a first sub-grid from the at least two sub-grids based on the size of the lesion tissue contained by the at least two sub-grids;

determining a second sub-grid closest to the first sub-grid from other sub-grids except the first sub-grid in the at least two sub-grids;

and merging the first sub-grid and the second sub-grid.

In one example, the merging module is configured to:

determining a third sub-grid which is closest to the first sub-grid and contains the largest focus organization from other sub-grids except the first sub-grid in the at least two sub-grids;

and merging the first sub-grid and the third sub-grid.

In one example, the merging module is configured to:

determining a fourth sub-grid which is closest to the first sub-grid and closest to the target direction of the center point of the focus outline from other sub-grids except the first sub-grid in the at least two sub-grids;

and merging the first sub-grid and the fourth sub-grid.

In one example, the identification module 920 is configured to:

the lesion contour of the lesion tissue in the medical image is identified by an AI model.

In one example, the identification module 920 is configured to:

identifying, by an AI model, the lesion contour of the lesion tissue in the medical image;

acquiring a three-dimensional point cloud image of the focus tissue;

generating a three-dimensional model of the focus tissue based on the three-dimensional point cloud image, flattening the three-dimensional model to obtain a two-dimensional flattened image of the focus tissue;

the lesion contour of the lesion tissue is determined on the two-dimensional flattened image.

In one example, the identification module 920 is configured to:

and carrying out equidistant expansion on the focus outline to determine the material drawing outline, wherein the material drawing outline is used for determining the pathological material drawing area.

In one example, the generating module 930 is configured to:

drawing the material drawing grid onto the three-dimensional model to obtain the three-dimensional model drawn with the material drawing grid;

rendering the three-dimensional model drawn with the material drawing grid to obtain the medical image covered with the material drawing grid.

Fig. 22 shows a block diagram of an image generating apparatus according to an exemplary embodiment of the present application, the apparatus including:

An acquisition module 950 for acquiring a medical image of a focal tissue, the medical image comprising a focal contour of the focal tissue.

The display module 960 is configured to display the medical image covered with a material-drawing grid, where the material-drawing grid includes at least two sub-grids covering a pathological material-drawing area corresponding to the lesion outline and arranged in an array; the lesion tissue is contained in the pathological sampling area covered by each sub-grid.

The embodiment of the application also provides a computer device, which comprises: a processor and a memory, the memory storing a computer program; and a processor for executing the computer program in the memory to implement the image generation method provided by each method embodiment.

Optionally, the computer device is a server. Illustratively, fig. 23 is a block diagram of a server provided in an exemplary embodiment of the present application.

In general, the server 1000 includes: a processor 1001 and a memory 1002.

The processor 1001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor 1001 may be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 1001 may also include a main processor, which is a processor for processing data in an awake state, also referred to as a central processor (Central Processing Unit, CPU), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 may be integrated with an image processor (Graphics Processing Unit, GPU) for use in the rendering and drawing of content required to be displayed by the display screen. In some embodiments, the processor 1001 may also include an artificial intelligence (Artificial Intelligence, AI) processor for processing computing operations related to machine learning.

Memory 1002 may include one or more computer-readable storage media, which may be non-transitory. Memory 1002 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 1002 is used to store at least one instruction for execution by processor 1001 to implement the image generation methods provided by the method embodiments of the present application.

In some embodiments, the server 1000 may further optionally include: an input interface 1003 and an output interface 1004. The processor 1001, the memory 1002, the input interface 1003, and the output interface 1004 may be connected by a bus or signal lines. The respective peripheral devices may be connected to the input interface 1003 and the output interface 1004 through buses, signal lines, or circuit boards. Input interface 1003, output interface 1004 may be used to connect at least one Input/Output (I/O) related peripheral device to processor 1001 and memory 1002. In some embodiments, the processor 1001, the memory 1002, and the input interface 1003, the output interface 1004 are integrated on the same chip or circuit board; in some other embodiments, any one or both of the processor 1001, the memory 1002, and the input interface 1003, the output interface 1004 may be implemented on a separate chip or circuit board, which is not limited by the embodiments of the present application.

Those skilled in the art will appreciate that the architecture shown in fig. 23 is not limiting of the computer device and may include more or fewer components than shown, or may combine certain components, or employ a different arrangement of components.

In an exemplary embodiment, the present application also provides a chip comprising programmable logic circuits and/or program instructions for implementing the image generation method of the above-described aspects when the chip is run on a computer device.

The present application provides a computer-readable storage medium storing a computer program loaded and executed by a processor to implement the image generation method provided by the above-described method embodiments.

The present application provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device obtains the computer instructions from a computer-readable storage medium, causing the processor to load and execute the image generation method provided by the above-described method embodiment.

The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.

Those of ordinary skill in the art will appreciate that all or a portion of the steps implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, where the program may be stored in a computer readable storage medium, and the above mentioned computer readable storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

Those skilled in the art will appreciate that in one or more of the examples described above, the functions described in the embodiments of the present application may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, these functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer-readable storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

The foregoing description of the preferred embodiments is merely exemplary in nature and is in no way intended to limit the invention, since it is intended that all modifications, equivalents, improvements, etc. that fall within the spirit and scope of the invention.

Claims

1. An image generation method, the method comprising:

identifying a pathological sampling area corresponding to the focus outline;

displaying the medical image covered with the material drawing grid.

2. The method of claim 1, wherein the generating a mesh of material for meshing the pathologically-drawn region comprises:

3. The method of claim 2, wherein the generating a mesh of material for meshing the pathologically-drawn region based on at least one reference point in the medical image comprises:

determining a center point of the lesion contour;

4. A method according to claim 3, wherein, in the case that there is no intersection between the at least two sub-grids and at least a portion of the outline of the lesion, expanding the at least two sub-grids of the array arrangement using the newly added sub-grid vertices of the at least two sub-grids generated by the expansion of the present round as a next round of reference points, and continuing to use the next round of reference points as expansion points, includes:

5. The method of claim 2, wherein the generating a mesh of material for meshing the pathologically-drawn region based on at least one reference point in the medical image comprises:

determining a center point of the lesion contour;

6. The method of claim 5, wherein generating a mesh for meshing the pathologically-derived region based on the respective sub-mesh vertex coordinates comprises:

7. The method according to any one of claims 3 to 6, further comprising:

removing repeated sub-grids in the at least two sub-grids;

or alternatively, the first and second heat exchangers may be,

8. The method according to any one of claims 1 to 7, further comprising:

acquiring a binary mask image of the medical image;

9. The method of claim 8, wherein the method further comprises:

determining a perpendicular bisector of the long axis of the lesion contour;

10. The method of any one of claims 1 to 7, wherein the sub-grid parameters of the at least two sub-grids comprise at least one of a size, a shape, an area, an interior angle size of the sub-grid;

11. The method according to claim 10, wherein the method further comprises:

12. The method according to any one of claims 1 to 11, further comprising:

13. The method of claim 12, wherein mesh merging at least two sub-meshes of the mesh that meet a merging condition comprises:

and merging the first sub-grid and the second sub-grid.

14. The method of claim 12, wherein mesh merging at least two sub-meshes of the mesh that meet a merging condition comprises:

and merging the first sub-grid and the third sub-grid.

15. The method of claim 12, wherein mesh merging at least two sub-meshes of the mesh that meet a merging condition comprises:

and merging the first sub-grid and the fourth sub-grid.

16. The method according to any one of claims 1 to 15, further comprising, after said obtaining a medical image of the focal tissue, before said identifying the region of pathological material to which the focal profile corresponds:

17. The method according to any one of claims 1 to 15, further comprising, after said obtaining a medical image of the focal tissue, before said identifying the region of pathological material to which the focal profile corresponds:

acquiring a three-dimensional point cloud image of the focus tissue;

18. The method according to claim 16 or 17, characterized in that the method further comprises:

19. The method of claim 17, wherein after the generating a material grid for meshing the pathological material region, before the displaying the medical image overlaid with the material grid, further comprises:

20. An image generation method, the method comprising:

21. An image generation apparatus, the apparatus comprising:

22. An image generation apparatus, the apparatus comprising:

23. A computer device, the computer device comprising: a processor and a memory storing a computer program that is loaded and executed by the processor to implement the image generation method of any one of claims 1 to 20.

24. A computer readable storage medium storing a computer program loaded and executed by a processor to implement the image generation method of any one of claims 1 to 20.

25. A computer program product, characterized in that it comprises computer instructions stored in a computer-readable storage medium, from which a processor obtains the computer instructions, such that the processor loads and executes to implement the image generation method according to any of claims 1 to 20.