WO2023280197A1

WO2023280197A1 - Image processing method and related apparatus, electronic device, storage medium and program

Info

Publication number: WO2023280197A1
Application number: PCT/CN2022/104069
Authority: WO
Inventors: 宋佳; 王一博; 姜超
Original assignee: 上海商汤智能科技有限公司
Priority date: 2021-07-07
Filing date: 2022-07-06
Publication date: 2023-01-12
Also published as: CN113506313A

Abstract

Embodiments of the present application disclose an image processing method and a related apparatus, an electronic device, a computer storage medium and a computer program. The image processing method comprises: acquiring within an image to be measured an original region where a target object is located, wherein several first pixel points are comprised in the original region, and there are several second pixel points outside of the original region; performing expansion detection in parallel on the second pixel points to obtain a detection result, wherein the detection result comprises whether the second pixel points are used as new first pixel points; and on the basis of the original region and the detection result, obtaining an expanded region of the target object.

Description

Image processing method and related device, electronic equipment, storage medium and program

Cross References to Related Applications

This application is based on the Chinese patent application with the application number 202110767334.1, the filing date is July 7, 2021, and the title is "image processing method and related devices, electronic equipment, and storage medium", and claims the priority of the Chinese patent application. The entire content of this Chinese patent application is hereby incorporated by reference into this application.

technical field

This application relates to the technical field of image processing, involving but not limited to an image processing method and related devices, electronic equipment, storage media and programs.

Background technique

With the development of information technology, image detection has been widely used in many industries such as medical treatment and transportation. Among many image detection methods, target detection can detect the image area of the target object in the image for subsequent image analysis.

In real-world scenarios, there may be technical requirements for area expansion in image areas, and area expansion often faces dual challenges of accuracy and speed. For example, in order to prevent recurrence after lesion resection or inactivation, it is usually necessary to expand the lesion on the basis of the actual original area, and resect or inactivate the tissue in the expanded area, in order to achieve the effect of completely killing the lesion.

Contents of the invention

Embodiments of the present application provide an image processing method and related devices, electronic equipment, computer storage media, and computer programs.

An embodiment of the present application provides an image processing method, which is applied to electronic devices. The method includes: obtaining the original area where the target object is located in the image to be tested, wherein the original area contains several first pixel points, and there are A number of second pixels; perform outreach detection on each second pixel in parallel to obtain a detection result, wherein the detection result includes whether to use the second pixel as a new first pixel; based on the original area and the detection result, obtain the target The bounding area of the object.

Therefore, the original area where the target object is located in the image to be tested is obtained, and the original area contains several first pixels, and there are several second pixels outside the original area. On this basis, the external expansion is performed on each second pixel in parallel Detection, to obtain the detection result, and the detection result includes whether to use the second pixel as the new first pixel, so that based on the original area and the detection result, the expanded area of the target object is obtained. On the one hand, the expanded detection is based on each first pixel. It is performed by two pixels, so it can realize pixel-level area expansion, which is beneficial to improve the accuracy of area expansion. On the other hand, because the expansion detection is executed in parallel, it is beneficial to improve the speed of area expansion. Therefore, the accuracy and speed of area expansion can be improved.

In some embodiments, the performing the extension detection on each of the second pixel points in parallel includes: performing the extension detection on each of the second pixel points in parallel in a manner that each calculation kernel runs in parallel. Therefore, each computing core runs in parallel the extension detection performed on each second pixel point, which can further improve the calculation efficiency of region extension.

In some embodiments, the first reference areas where the second pixel points are respectively located overlap with the original area. Therefore, for each second pixel point, the first reference area where it is located intersects with the original one, so before the extension detection, the pixels located outside the original area and farther away from the original area can be excluded, which is beneficial to Increase the speed of area expansion.

In some embodiments, performing the outreach detection on each of the second pixel points in parallel to obtain the detection result includes: obtaining a second reference area surrounding the second pixel point, and the second reference area contains at least one first pixel points; for the second pixel point and each first pixel point in the second reference area, respectively acquire the physical distance from each first pixel point to the second pixel point; and obtain the detection result based on the physical distance. Therefore, a second reference area surrounding the second pixel is obtained, and the second reference area contains at least one first pixel, and based on the physical distance from the first pixel to the second pixel in the second reference area, the detected As a result, it is possible to reduce the possibility of the area expanding beyond the safe distance, which is beneficial to improve the safety of the area expanding.

In some embodiments, obtaining the detection result based on the physical distance includes: when the minimum physical distance is lower than a preset threshold, determining the detection result includes using the second pixel as a new first pixel; at the minimum physical distance If it is not lower than the preset threshold, determining the detection result includes not using the second pixel as a new first pixel. Therefore, when the minimum physical distance is lower than the preset threshold, determining the detection result includes taking the second pixel as the new first pixel, and when the minimum physical distance is not lower than the preset threshold, determining the detection result Including not using the second pixel as the new first pixel, it can further ensure that the area expansion does not exceed a safe distance by constraining the minimum physical distance, and improve the safety of the area expansion.

In some embodiments, there are several third pixel points outside the original area, and there is no intersection between the first reference area where each third pixel point is located and the original area; based on the original area and the detection result, the outer area of the target object Before expanding the area, the method further includes: when the detection result includes the second pixel as the new first pixel, using the third pixel that meets the preset condition as the new second pixel, and re-executing the The step of performing out-expansion detection on each second pixel in parallel to obtain a detection result; wherein, the preset condition includes: the first reference area where the third pixel is located contains the first pixel updated by the second pixel.

Therefore, there are several third pixels outside the original area, and there is no intersection between the first reference area where each third pixel is located and the original area, when the detection result includes the second pixel as the new first pixel Next, use the third pixel that satisfies the preset condition as a new second pixel, and re-execute the step of performing the expansion detection on each second pixel in parallel to obtain the detection result, and the preset condition includes: the third pixel The first reference area where the point is located contains the first pixel point updated by the second pixel point, that is, in the process of area expansion, the second pixel point can be updated to the first pixel point, and the third pixel point will also be updated according to Satisfy the preset condition and use it as a new second pixel point, and re-perform the expansion detection on each second pixel point, so that the area expansion can be carried out in a reciprocating manner, which is beneficial to further improve the accuracy of the area expansion.

In some embodiments, based on the original area and the detection result, obtaining the extended area of the target object includes: in response to detecting that each second pixel point has performed an extended detection, obtaining the first pixel point in the original area and the first pixel point in the original area by The second pixel is updated to obtain the connected domain formed by the first pixel, and the connected domain is used as an extended area. Therefore, when it is detected that each second pixel point has performed the extension detection, the connected domain formed by the first pixel point in the original area and the first pixel point updated by the second pixel point is obtained, and The connected domain is used as the expansion area, that is, the expansion of the area ends with the expansion detection of each second pixel, and the connected domain formed by all the first pixels is used as the expansion area, which is conducive to improving the expansion. area accuracy.

In some embodiments, the image to be tested is a medical image, the target object is a lesion, and the medical image also includes several target tissues; the method further includes: detecting the infringement data of each first pixel point on several target tissues; wherein, the infringement The data includes: the target organization violated by the first pixel. Therefore, the image to be tested is a medical image, and the target object is a lesion. The medical image also includes several target tissues. Including the target tissue violated by the first pixel, that is, the infringement data of each first pixel against the target tissue can be recorded, so that reference information can be provided during the application process, which is conducive to improving user experience.

In some embodiments, the image to be tested includes a plurality of target objects; the method further includes: displaying an object list; wherein the object list includes identifiers of a plurality of target objects; Instructions, for the target object corresponding to the identifier, perform the step of performing the expansion detection on each of the second pixel points in parallel to obtain the detection result and the subsequent steps to obtain the expansion area.

Therefore, a plurality of target objects are identified in the image to be tested. On this basis, an object list is displayed, and the object list includes indicators of a plurality of target objects. , for the target object corresponding to the identifier, execute the expansion detection for each second pixel point in parallel, obtain the detection result and the subsequent steps, and obtain the expansion area, so it is possible to have multiple target objects in the image to be tested In this case, regional expansion is performed according to user instructions, which is conducive to improving user experience.

In some embodiments, after obtaining the expanded area of the target object based on the original area and the detection result, the method further includes: obtaining the surface mesh of the target object by triangulation based on the expanded area; The surface grid of the object is rendered to obtain the image model of the target object; the image model of the target object is displayed in the image display interface.

Therefore, after obtaining the extended area, the surface mesh of the target object is obtained based on the triangulation of the extended area, and the rendering parameters of the target object are used to render the surface mesh of the target object to obtain the image model of the target object. The image model of the target object is displayed on the display interface, so it can be beneficial to intuitively display the three-dimensional information of the target object and improve user experience.

The embodiment of the present application also provides an image processing device, including: an original area acquisition module, a pixel expansion detection module, and an expansion area acquisition module, and the original area acquisition module is configured to acquire the original area where the target object in the image to be tested is located , wherein, the original area contains a number of first pixels, and there are a number of second pixels outside the original area; the pixel expansion detection module is configured to perform an expansion detection on each second pixel in parallel to obtain a detection result, wherein, the detection The result includes whether to use the second pixel point as the new first pixel point; the extended area obtaining module is configured to obtain the extended area of the target object based on the original area and the detection result.

In some embodiments, the pixel extension detection module is configured to perform the extension detection on each of the second pixel points in parallel in a manner that each calculation kernel runs in parallel. Therefore, each computing core runs in parallel the extension detection performed on each second pixel point, which can further improve the calculation efficiency of region extension.

In some embodiments, the pixel extension detection module includes a reference area acquisition submodule configured to acquire a second reference area surrounding the second pixel point; the pixel extension detection module includes a physical distance calculation submodule configured for the second reference The second pixel point and each first pixel point in the area respectively obtain the physical distance from each first pixel point to the second pixel point; the pixel expansion detection module includes a detection result acquisition sub-module configured to obtain a detection result based on the physical distance result. Therefore, obtaining the second reference area surrounding the second pixel, and obtaining the detection result based on the physical distance from the first pixel to the second pixel in the second reference area, can reduce the possibility of the area expanding beyond the safety distance, It is conducive to improving the security of regional expansion.

In some embodiments, the detection result acquisition submodule includes a first determination unit configured to determine the detection result when the minimum physical distance is lower than a preset threshold, including taking the second pixel as a new first pixel; detecting The result acquisition sub-module includes a second determination unit configured to determine that the detection result includes not using the second pixel as a new first pixel when the minimum physical distance is not lower than a preset threshold. Therefore, when the minimum physical distance is lower than the preset threshold, determining the detection result includes taking the second pixel as the new first pixel, and when the minimum physical distance is not lower than the preset threshold, determining the detection result Including not using the second pixel as the new first pixel, it can further ensure that the area expansion does not exceed a safe distance by constraining the minimum physical distance, and improve the safety of the area expansion.

In some embodiments, there are several third pixel points outside the original area, and there is no intersection between the first reference area where each third pixel point is located and the original area; the image processing device also includes a duplication detection module configured to Including the case where the second pixel is used as the new first pixel, the third pixel that meets the preset condition is used as the new second pixel, and re-performs the extension detection for each second pixel in parallel, The step of obtaining the detection result; wherein, the preset condition includes: the first reference area where the third pixel is located contains the first pixel updated by the second pixel.

In some embodiments, the extended area acquisition module is specifically configured to acquire the first pixel points updated by the first pixel points in the original area and the second pixel points in the original area in response to detecting that each second pixel point has performed the extended detection. The connected domain formed by the pixels, and the connected domain is used as the external expansion area. Therefore, when it is detected that each second pixel point has performed the extension detection, the connected domain formed by the first pixel point in the original area and the first pixel point updated by the second pixel point is obtained, and The connected domain is used as the expansion area, that is, the expansion of the area ends with the expansion detection of each second pixel, and the connected domain formed by all the first pixels is used as the expansion area, which is conducive to improving the expansion. area accuracy.

In some embodiments, the image to be tested is a medical image, the target object is a lesion, and the medical image also includes several target tissues; the image processing device further includes an infringement detection module configured to detect the impact of each first pixel on several target tissues. The violation data; wherein, the violation data includes: the target organization violated by the first pixel. Therefore, the image to be tested is a medical image, and the target object is a lesion. The medical image also includes several target tissues. Including the target tissue violated by the first pixel, that is, the infringement data of each first pixel on the target tissue can be recorded, so that reference information can be provided during the application process, which is conducive to improving user experience.

In some embodiments, multiple target objects are identified in the image to be tested; the image processing device also includes a list display module configured to display a list of objects; wherein, the object list includes identifiers of multiple target objects; the image processing device also includes The external expansion interaction module is configured to respond to the identifier being in the selected state and the external expansion instruction input by the user, in combination with the pixel external expansion detection module and the external expansion area acquisition module, for the target object corresponding to the identifier, to perform an operation on each second pixel point Executing the expansion detection in parallel, the step of obtaining the detection result and the subsequent steps to obtain the expansion area. Therefore, a plurality of target objects are identified in the image to be tested. On this basis, an object list is displayed, and the object list includes identifiers of a plurality of target objects. , for the target object corresponding to the identifier, execute the expansion detection for each second pixel point in parallel, obtain the detection result and the subsequent steps, and obtain the expansion area, so it is possible to have multiple target objects in the image to be tested In this case, regional expansion is performed according to user instructions, which is conducive to improving user experience.

In some embodiments, the image processing device includes a triangulation module configured to triangulate the surface mesh of the target object based on the extended area; the image processing device includes a model rendering module configured to render the target object using the rendering parameters of the target object The surface grid is rendered to obtain the image model of the target object; the image processing device includes a model display module configured to display the image model of the target object on the image display interface.

An embodiment of the present application also provides an electronic device, including a memory and a processor coupled to each other, and the processor is configured to execute program instructions stored in the memory to implement any one of the above image processing methods.

In some embodiments, the aforementioned electronic device is a terminal device or a server.

The embodiment of the present application also provides a computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, any one of the above-mentioned image processing methods is realized.

The embodiment of the present application also provides a computer program, including computer readable code, when the computer readable code is run in the electronic device, the processor in the electronic device executes any one of the above image processing method.

The above scheme obtains the original area where the target object is located in the image to be tested, and the original area contains a number of first pixels, and there are a number of second pixels outside the original area. On this basis, each second pixel is executed in parallel. Expand the detection to obtain the detection result, and the detection result includes whether to use the second pixel as the new first pixel, so that based on the original area and the detection result, the extended area of the target object is obtained. On the one hand, because the external expansion detection is based on each It is executed on the second pixel, so it can realize pixel-level area expansion, which is beneficial to improve the accuracy of area expansion. On the other hand, because the expansion detection is executed in parallel, it is beneficial to improve the speed of area expansion. Therefore, the accuracy and speed of area expansion can be improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Description of drawings

The accompanying drawings here are incorporated into the description and constitute a part of the present description. These drawings show embodiments consistent with the present disclosure, and are used together with the description to explain the technical solution of the present disclosure.

FIG. 1 is a schematic flow diagram of an image processing method provided in an embodiment of the present application;

Fig. 2 is a schematic diagram of the area expansion provided by the embodiment of the present application;

Fig. 3 is a schematic diagram of the image display interface provided by the embodiment of the present application;

Fig. 4 is another schematic diagram of the image display interface provided by the embodiment of the present application;

Fig. 5 is another schematic diagram of the image display interface provided by the embodiment of the present application;

Fig. 6 is a schematic frame diagram of an image processing device provided by an embodiment of the present application;

Fig. 7 is a schematic frame diagram of an electronic device provided by an embodiment of the present application;

Fig. 8 is a schematic diagram of a computer-readable storage medium provided by an embodiment of the present application.

detailed description

The solutions of the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

In the following description, for purposes of illustration rather than limitation, specific details, such as specific system architectures, interfaces, and techniques, are set forth in order to provide a thorough understanding of the present application.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations. In addition, the term "at least one" herein means any one of a variety or any combination of at least two of the more, for example, including at least one of A, B, and C, which may mean including from A, Any one or more elements selected from the set formed by B and C. The character "/" in this article generally indicates that the contextual objects are an "or" relationship. In addition, "many" herein means two or more than two.

In the embodiment of the present application, the execution subject of the image processing method may be an electronic device, for example, the electronic device may be a terminal device, a server or other processing device, wherein the terminal device may be a user equipment (User Equipment, UE), a mobile device, User terminals, terminals, cellular phones, cordless phones, personal digital assistants (Personal Digital Assistant, PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc. In some possible implementation manners, the image processing method may be implemented by a processor invoking computer-readable instructions stored in a memory.

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of an image processing method provided by an embodiment of the present application. The process flow of the image processing method may include the following steps:

Step S11: Obtain the original area where the target object is located in the image to be tested.

In an implementation scenario, the image to be tested and the target object can be set according to actual applications. For example, in a medical scene, the image to be tested may include medical images such as computed tomography (CT), magnetic resonance (Magnetic Resonance, MR), etc., and the target object may include lesions such as tumors, cysts, and abscesses; or, In a traffic scene, the images to be tested may include surveillance images, and target objects may include pedestrians, vehicles, and the like. Other situations can be deduced by analogy, and no more examples will be given here.

In an implementation scenario, in order to improve the recognition efficiency, an object recognition network can be pre-trained, so that the object recognition network can be used to recognize the image to be tested and obtain the original area where the target object is located. For example, the object recognition network may include but not limited to: Region-Convolutional Neural Networks (R-CNN), Fully Convolutional Network (Fully Convolutional Network, FCN), etc., which are not limited here. Before training the object recognition network, sample images containing target objects can be collected in advance, and the sample images are marked with the sample category to which each sample pixel belongs (for example, a certain pixel is marked as belonging to the target object), and then the object recognition network can be used to Recognize the sample image to obtain the predicted category to which each pixel in the sample image belongs. Finally, the difference between the sample category and the predicted category can be used to adjust the network parameters of the object recognition network so that the object recognition network can learn during the training process. to the image features of the target object. Based on this, the image to be tested can be identified by using the trained and converged object recognition network to obtain the pixel category to which each pixel in the image to be tested belongs, and the connected domain formed by the pixels belonging to the target object can be used as the original area where the target object is located . In the case that the target objects are set as lesions, pedestrians, vehicles, etc. according to the actual application, the object recognition network for identifying lesions and the object recognition network for identifying pedestrians and vehicles can be obtained by analogy training, which will not be repeated here. example.

It should be noted that, in the embodiment of the present application, the image to be tested may not be limited to a two-dimensional image or a three-dimensional image. For example, in a traffic scene, the image to be tested may include but not limited to two-dimensional images such as surveillance images; while in a medical scene, the image to be tested may include but not limited to three-dimensional images such as CT, MR, etc., that is, in this case , the image to be tested is substantially volume data, and the "first pixel", "second pixel", and "third pixel" described in the embodiments of the present application may represent voxels of the volume data.

In the embodiment of the present application, the original area contains several first pixels, that is, the pixels contained in the original area can be directly used as the first pixels. In order to facilitate the distinction of each first pixel point, each first pixel point may be assigned a first identifier respectively, and the first identifiers of each first pixel point in the original area are different. For example, natural numbers such as 0, 1, 2, and 3 may be respectively assigned to each first pixel point as a first identifier of each first pixel point.

In the embodiment of the present application, there are several second pixel points outside the original area. For example, the pixels outside the original area can be directly used as the second pixels; or, in order to increase the speed of area expansion, the first reference area where each second pixel is located has an intersection with the original area, that is, for each As far as a second pixel point is concerned, the first reference area where it is located has an intersection with the original area.

In an implementation scenario, the center of the first reference area where the second pixel is located may be the second pixel, that is, for each second pixel, a neighborhood (such as a rectangle) may be determined centered on itself. area, circular area, etc.), as the first reference area where it is located. That is to say, if the neighborhood of a certain pixel outside the original area intersects with the original area, it means that there is a first pixel in the neighborhood, and this pixel can be used as the second pixel.

In an implementation scenario, the size of the first reference area may be set according to actual conditions. For example, in the case of high precision requirements for area expansion, the size of the first reference area can be set to be slightly smaller, for example, in the case of a two-dimensional image to be tested, it can be set to 3*3, and In the case that the image to be tested is a 3D image, it can be set to 3*3*3; or, in the case of relatively loose requirements on the accuracy of area expansion, the size of the first reference area can be set slightly larger, such as When the image to be tested is a two-dimensional image, it can be set to 5*5, and when the image to be tested is a three-dimensional image, it can be set to 5*5*5. Other situations can be deduced by analogy, and no more examples will be given here.

In an implementation scenario, in order to distinguish from the first pixel, each second pixel may be assigned a second identifier, and the second identifier is different from the first identifier. For example, -1 (or -2, -3, etc.) may be assigned to each second pixel as the second identifier of the second pixel.

Step S12: Execute out-expansion detection on each second pixel point in parallel to obtain a detection result.

In the embodiment of the present application, the detection result includes whether the second pixel is used as the new first pixel, for example, the detection result includes the second pixel as the new first pixel, or the detection result includes not using the second pixel as the new first pixel. It should be noted that if the detection result includes the second pixel as the new first pixel, it means that the second pixel can be used as the new first pixel. On the contrary, if the detection result includes not using the second pixel as the new first pixel, it means that the second pixel is maintained as the second pixel.

In an implementation scenario, the extension detection performed on each second pixel point may be executed by each computing core in parallel. It should be noted that, in order to further improve the effect of parallel execution, the computing core may be each core of a graphics processing unit (Graphics Processing Unit, GPU). For example, parallel acceleration can be performed based on Compute Unified Device Architecture (CUDA), and you can refer to CUDA related technical details. The above method can further improve the calculation efficiency of the region expansion.

In an implementation scenario, in the process of performing extension detection on each second pixel point in parallel, a second reference area surrounding the second pixel point may be acquired, and for the second pixel point in the second reference area and each second pixel point For one pixel, the physical distance from each first pixel to the second pixel can be obtained respectively, and a detection result can be obtained based on the physical distance. That is to say, when performing extension detection on each second pixel point, it is necessary to obtain a second reference area surrounding the second pixel point, and the second pixel point in the second reference area and each first pixel point For the pixels, the physical distances from each first pixel to the second pixel are obtained respectively. The above method can reduce the possibility of area expansion beyond the safe distance, and is conducive to improving the safety of area expansion.

In an implementation scenario, the center of the second reference area surrounding the second pixel point may be the second pixel point, that is, the second reference area may be set as a neighborhood of the second pixel point.

In an implementation scenario, in order to improve the consistency in the area expansion process, the size of the second reference area and the first reference area may be set to be the same. For example, when the image to be tested is a two-dimensional image, the sizes of the first reference area and the second reference area can both be set to 3*3; or, when the image to be tested is a three-dimensional image, the first reference area and the size of the second reference area can both be set to 3*3*3. In the above manner, it is possible to screen whether there is the first pixel point and the first pixel point with the closest physical distance in the same neighborhood, so as to improve the consistency in the process of area expansion.

In an implementation scenario, the pixel distance from the first pixel point to the second pixel point in the second reference area can be obtained first, and then based on the conversion unit between the image pixel distance and the actual pixel distance (that is, how much physical distance is equal to a pixel distance ), convert the calculated pixel distance into physical distance. For example, the physical distance where one pixel distance is equal to 5 mm, or the physical distance where one pixel distance is equal to 1 mm, is not limited here. In addition, in order to facilitate the calculation of the pixel distance, a location identifier can be assigned to each pixel in the image to be tested in advance, and the location identifiers of each pixel are different. On this basis, according to the difference between the location identifiers, Calculate the pixel distance. Taking the image to be tested as a two-dimensional image as an example, the pixel at position (0,0) can be assigned to position identifier 0, the pixel at position (0,1) can be assigned to position identifier 1, and the pixel at position (0, The pixel at the j) position is assigned to the position identifier j, and the pixel at the (i, j) position is assigned to the position identifier i+j, and so on, and examples are not given here. On this basis, for the first pixel point whose position identifier is P1, it can be determined that the first pixel point is located in the i1th row of the image to be tested, and i1=P1/N+1, where / represents the quotient operation, and N represents The total number of columns of pixels in the image to be tested; at the same time, it can be determined that the second pixel is located in the j1th column of the image to be tested, and j1=P1%N+1, where % represents a remainder operation. And for the second pixel point whose position identifier is P2, it can be determined that the second pixel point is located in the i2th row of the image to be tested, and i2=P2/N+1; at the same time, it can be determined that the second pixel point is located in the image to be tested Column j2, and j2=P2%N+1. On this basis, the pixel distance between the two is calculated based on the pixel position (i1, j1) of the first pixel point and the pixel position (i2, j2) of the second pixel point. Other situations can be deduced by analogy, and no more examples will be given here.

In an implementation scenario, when the minimum physical distance is lower than the preset threshold, it may be determined that the detection result includes the second pixel as the new first pixel, and when the minimum physical distance is not lower than the preset threshold In this case, it may be determined that the detection result includes not using the second pixel as the new first pixel. It should be noted that the preset threshold can be set according to the safety distance of the area expansion, for example, the safety distance can be directly set as the preset threshold, or, if a certain degree of error range is acceptable, the safety distance can also be set to The sum (or difference) of the distance and the preset value is set as the preset threshold, which is not limited here. The above method can further ensure that the expansion of the area does not exceed a safe distance by constraining the minimum physical distance, and improve the security of the expansion of the area.

In an implementation scenario, FIG. 2 is a schematic diagram of the area expansion provided by the embodiment of the present application. As shown in FIG. 2 , the squares represent each pixel in the image to be tested 201, wherein the hatched squares with diagonal lines represent the original area The first pixel in 202. On this basis, for the pixel points outside the original area (as shown in the dotted shadow filled square), if there is an intersection between the first reference area where the pixel point is located (the bold dotted line box in the figure) and the original area 202 , then this pixel can be used as the second pixel. In order to facilitate the display of the drawings, Fig. 2 only shows one of the second pixels by way of example. In the actual application process, each second pixel outside the original area can be determined according to the above-mentioned related descriptions. Two pixels. Based on this, for each second pixel point, the second reference area surrounding the second pixel point can be obtained. As mentioned above, in order to ensure the consistency of the area expansion process, the second reference area can have the same The size, that is, the second reference area may be the bold dashed box shown in the figure. In addition, the second reference area contains at least one first pixel point, on this basis, for the second pixel point in the second reference area (that is, the dotted shaded grid in the figure) and each first pixel point (that is, In the figure, the slanted line shades fill the squares), the physical distance from each first pixel point to the second pixel point can be calculated (the calculation process can refer to the above-mentioned related description), and when the minimum physical distance is less than the preset threshold value, the The second pixel is used as the new first pixel, and in the case that the minimum physical distance is not less than the preset threshold, the second pixel is not used as the new first pixel. The other second pixels can be deduced by analogy, and finally the connected domain formed by the first pixel in the original area and the first pixel updated by the second pixel can be used as the extended area 203 . It should be noted that, for the convenience of description, Figure 2 and the above text part illustrate the specific process of area expansion from a two-dimensional perspective. When the image to be tested is volume data, it can be deduced by analogy, and no more examples are given here. .

In an implementation scenario, there are several third pixels outside the original area, and there is no intersection between the first reference area where each third pixel is located and the original area, in order to further improve the accuracy of area expansion, you can also In the case that the detection result includes the second pixel as the new first pixel, the third pixel that meets the preset condition is used as the new second pixel, and the above-mentioned parallel execution for each second pixel is re-executed Outward expansion detection is a step of obtaining a detection result, and the preset condition includes: the first reference area where the third pixel point is located includes the first pixel point updated by the second pixel point. In the above method, in the process of area expansion, as the second pixel point is updated to the first pixel point, the third pixel point will also be used as the new second pixel point according to the preset condition, and each second pixel point will be reset The pixel points perform the expansion detection, so that the area expansion can be carried out in such a reciprocating manner, which is conducive to further improving the accuracy of the area expansion.

In an implementation scenario, in order to facilitate the distinction from the aforementioned first pixel and second pixel, a third identifier can also be assigned to each third pixel, and the third identifier is the same as the first identifier and the second identifier different. For example, when a natural number is assigned to each first pixel as the first identifier of the first pixel, and -1 is assigned to each second pixel as the second identifier of the second pixel, -2 can be assigned (or -3, -4, etc.) Assign each third pixel point as a third identifier of the third pixel point.

In an implementation scenario, as mentioned above, the first reference area where the second pixel is located may be centered on the second pixel. In this case, the first reference area where the third pixel is located may also be centered on the third The pixel is the center, which can improve the consistency of the region expansion process.

In an implementation scenario, as mentioned above, the image to be tested and the target object can be set according to the actual application. The object to be tested can be a medical image, and the target object can be a lesion. In this case, in the process of area expansion, it is also possible to detect the infringement data of each first pixel point on several target tissues, and the infringement data can include the first The target organization violated by the pixel. The above method can record the violation data of each first pixel point against the target tissue, so that reference information can be provided during the application process, which is beneficial to improve user experience.

In an implementation scenario, the first pixel includes the first pixel in the original area and the first pixel updated by the second pixel. For example, when the first pixel is located within the tissue area of the target tissue, it can be considered that the first pixel violates the target tissue; otherwise, if the first pixel is located outside the tissue area of the target tissue, it can be considered that the first pixel does not The subject organization is not infringed.

In an implementation scenario, the target tissue may be set according to the medical tissue where the lesion is located. For example, when the lesion is located in the liver, the target tissue may include, but not limited to, gallbladder, spleen, pancreas, etc. Other situations can be deduced by analogy, and no more examples will be given here.

Step S13: Obtain the extended area of the target object based on the original area and the detection result.

For example, in the case where it is detected that each second pixel point has been subjected to extension detection, the connected domain formed by the first pixel point in the original area and the first pixel point updated by the second pixel point may be obtained, and The connected domain is used as the extension area. As mentioned above, the first pixel is assigned a first identifier whose value is a natural number, the second pixel is assigned a second identifier whose value is a negative number, and the third pixel is assigned a third identifier whose value is a negative number. On this basis Above, the connected domain formed by all the pixels corresponding to the identifier whose value is a natural number can be used as the extended area. In the above method, the area expansion ends when all the second pixels have performed the expansion detection, and finally the connected domain formed by all the first pixels is used as the expanded area, which is beneficial to improve the accuracy of the expanded area.

In an implementation scenario, after obtaining the extended area, the surface mesh of the target object can be obtained by triangulation based on the extended area, and the surface mesh (mesh) can be rendered using the rendering parameters of the target object to obtain the target object's surface mesh. an image model, and display the image model of the target object in the image display interface. The above-mentioned method can help to intuitively display the three-dimensional information of the target object, and is beneficial to improve user experience.

In an implementation scenario, during the triangulation process, the scatter points on the surface of the organization area can be used to form the vertices of the triangles, the line segments connecting the vertices form the sides of the triangles, and each triangle corresponds to a face, and complex objects can be simulated through triangulation For the specific process of triangulation, such as human body, vehicle, building, etc., please refer to the technical details of triangulation.

In an implementation scenario, the dyeing parameters may include but not limited to: color, transparency, material, etc., which are not limited here. For example, in the case that the target object is a lesion, the rendering parameters of the lesion may be set as: yellow, 10% transparency, and rough surface material. Other situations can be deduced by analogy, and no more examples will be given here.

In an implementation scenario, please refer to FIG. 3 and FIG. 4 in conjunction. FIG. 3 is a schematic diagram of the image display interface provided by the embodiment of the present application. In FIG. 3, the image of the liver region includes the hepatic vein 301, the hepatic artery 302, and the hepatic portal vein 303 and the first lesion area 304; the lesion area shown in Figure 3 is an area that has not been processed by area expansion; after performing area expansion processing on the lesion area shown in Figure 3 according to the image processing method of the embodiment of the present application, it can be The second lesion area 401 shown in FIG. 4 is obtained. It should be noted that what is shown in FIG. 3 and FIG. 4 is only a situation that may exist in the application process, and does not limit the actual effect of area expansion in other scenarios.

In an implementation scenario, when multiple target objects are identified in the image to be tested, an object list may be displayed, and the object list includes identifiers of multiple target objects, so that the identifier may be selected in response to the user's The input expansion command executes the expansion detection on each second pixel in parallel for the target object corresponding to the identifier, and obtains the detection result and subsequent steps to obtain the expansion area. For the specific process, please refer to the above-mentioned relevant description. In the above manner, when there are multiple target objects in the image to be tested, the area can be expanded according to the user's instruction, which is beneficial to improve user experience.

In an implementation scenario, the identifier may include, but is not limited to: the name, code, serial number, etc. of the target object, which is not limited here. For example, in the case where the target object is a lesion, if three liver cysts are identified in the image to be tested, the identifiers of these three liver cysts can be "liver cyst 1", "liver cyst 2", " Liver cyst 3". Other situations can be deduced by analogy, and no more examples will be given here.

In an implementation scenario, in order to improve user interaction comfort, in response to the identifier being in a selected state, the image model of the target object corresponding to the identifier can be highlighted in a preset manner on the image display interface. The preset methods may include but are not limited to: highlighting, thickening edges, etc., which are not limited here. In addition, regarding the manner of obtaining the image model, reference may be made to the foregoing related descriptions.

In the related technology, since the area expansion is related to the postoperative effect of the patient, it has high requirements in terms of accuracy and speed. In view of this, how to improve the accuracy and speed of region expansion has become an urgent problem to be solved. However, in the embodiment of the present application, the original area where the target object in the image to be tested is located can be obtained, and the original area contains several first pixel points, and there are several second pixel points outside the original area. Execute the expansion detection of two pixels in parallel to obtain the detection result, and the detection result includes whether to use the second pixel as the new first pixel, so as to obtain the expansion area of the target object based on the original area and the detection result. On the one hand, due to The expansion detection is performed based on each second pixel, so it can realize pixel-level area expansion, which is beneficial to improve the accuracy of area expansion. On the other hand, because the expansion detection is performed in parallel, it is beneficial to improve the area. The speed of expansion. Therefore, the accuracy and speed of area expansion can be improved.

In some embodiments, as described in the foregoing embodiments, the image to be tested and the target object can be set according to actual applications. For example, the image to be tested may be a medical image, and the target object may be a lesion. In this case, the image to be tested may include several medical tissues, and the several medical tissues include the lesion. Based on this, the image display interface may also display an organization list, and the organization list displays identifiers of corresponding medical organizations in the image space. Based on the selection state of the identifier, the image model of the medical tissue corresponding to the identifier can be displayed on the image display interface according to the display strategy matching the selection state, and the selection state includes the first state representing being selected and the state representing not being selected. In the second state, the display strategy matching the first state is different from the display strategy matching the second state. The above solution can display medical tissues in different display strategies on the image display interface according to whether the medical tissues are in the selected state or not selected state, that is, it can support the user to independently select the display strategy of the medical tissues, so that the images can be The display interface displays and distinguishes various medical tissues, which is conducive to intuitively and accurately reflecting the relative positional relationship between the medical tissues.

In an implementation scenario, the identifier may include but not limited to: the name, code, serial number, etc. of the medical organization, which are not limited here. Please refer to FIG. 5 in conjunction with FIG. 5. FIG. 5 is another schematic diagram of the image display interface provided by the embodiment of the present application. As shown in FIG. Taking a scanned image as an example, the tissue list 501 may display identifiers of various medical tissues in the abdomen: "hepatic vein", "hepatic portal vein", "inferior vena cava", "abdominal artery", "biliary duct", "left liver", "Right liver" and the watershed segmentation of the liver, etc. When the medical image is a scanned image of other parts, it can be deduced by analogy, and no more examples will be given here.

In an implementation scenario, please continue to refer to Figure 5. When the check box in front of the identifier is checked, it can indicate that the medical organization corresponding to the identifier is in the first state of being selected, and check in front of the identifier If the box is not checked, it may indicate that the medical organization corresponding to the identifier is in the second state of being unselected. In addition, the display strategy matching the first state may include but not limited to: display, etc., and the display strategy matching the second state may include but not limited to: hiding, etc.; or, the display strategy matching the first state may include but It is not limited to: displaying in a highlighted manner, etc., and the display strategy matching the second state may include but not limited to: displaying in a normal display manner, for details, please refer to the following related descriptions.

In one implementation scenario, the image display interface includes at least one of a first display area 502 and a second display area 503, the first display area 502 is used to display an image model of medical tissue, and the second display area 503 is used to display several 2D image at preset orientation. It should be noted that several preset orientations may specifically include but not limited to: transverse, coronal, sagittal, etc., which are not limited here; in addition, the two-dimensional image may specifically be a multi-planar reconstruction map (Multi-Planner Reformation, MPR ). The multi-planar reconstruction image is to obtain two-dimensional images of human tissues and organs in any orientation (such as the aforementioned transverse, coronal, sagittal, and oblique) from the original transverse axial image after post-processing. For the specific process of post-processing, please refer to MPR-related technical details. Taking several preset orientations including transverse, coronal and sagittal as examples, as shown in Figure 5, the first display area 502 can display the image model of the medical tissue, and the second display area 503 can respectively display the transverse multi-planar reconstruction 504. The coronal multi-plane reconstruction diagram 505 and the sagittal multi-plane reconstruction diagram 506, that is, the first display area 502 can display the medical tissue in a three-dimensional angle, and the second display area 503 can display the medical tissue in a two-dimensional angle, so it can be displayed in the image The display interface displays medical tissues in different dimensions at the same time, which is conducive to improving the richness of image information displayed on the image display interface.

In one implementation scenario, the image display interface includes a first display area 502, and when the selected state of the identifier is the first state, the image model of the medical tissue corresponding to the identifier can be displayed, and in the selected state of the identifier In the case of the second state, the image model of the medical tissue corresponding to the identifier may be hidden. In addition, in the case of displaying the image model of the medical tissue corresponding to the identifier, the image model of the medical tissue corresponding to the identifier may be further displayed in a preset manner. The preset methods may include, but are not limited to: thickening of edges, highlighting, and the like. The above method can support user-defined selection of the medical tissue that needs to be highlighted in the first display area 502, which is beneficial to support the user to focus on observing the highlighted medical tissue, and by hiding the medical tissue in the second state, it is possible for the user to focus on observing the highlighted medical tissue. The medical organization excludes the interference of other medical organizations, which is conducive to improving the user experience.

In one implementation scenario, the image display interface includes a second display area 503, and when the selection state of the identifier is the first state, the medical tissue corresponding to the identifier in the two-dimensional image can be displayed in a highlighted manner, and in the When the selection state of the identifier is the second state, the medical tissue corresponding to the identifier in the two-dimensional image may be displayed in a conventional display manner. For example, the highlighting method may include but not limited to: thicken the edge, highlight, etc., and the normal display method may include the original display method of the multi-plane reconstruction map, such as: the default grayscale image of the multi-plane reconstruction map, which is not mentioned here. Do limited. In the above manner, the medical tissue can be displayed in a highlighted display mode or a normal display mode in the second display area 503 according to whether the medical tissue is selected by the user, that is, it can support user-defined selection of the medical tissue that needs to be highlighted in the second display area 503, It is beneficial to support the user to focus on observing the prominent medical tissue, and by displaying the medical tissue in the second state in a conventional display manner, the interference of other medical tissues can be excluded when the user focuses on the prominent medical tissue, which is beneficial to improving user experience.

In one implementation scenario, the image display interface includes a first display area 502 and a second display area 503, and when the selection state of the identifier is the first state, the first display area 502 may display the medical information corresponding to the identifier. tissue image model, and display the medical tissue corresponding to the identifier in the two-dimensional image in a highlighted manner in the second display area 503; The image model of the medical tissue corresponding to the identifier is hidden, and the medical tissue corresponding to the identifier in the two-dimensional image is displayed in a conventional display manner in the second display area 503 . For details, refer to the related description above. The above method can not only display medical tissues in different dimensions on the image display interface at the same time, which is conducive to improving the richness of image information displayed on the image display interface, but also can display user It is expected that the prominent medical tissue will allow users to intuitively connect the corresponding medical tissue in two different angles, three-dimensional and two-dimensional, to improve user experience.

In an implementation scenario, please continue to refer to FIG. 5 , in application scenarios such as operation planning, the organization list may include identifiers of several medical organizations, and the several medical organizations may include lesions. On this basis, the user can check the identifier of the medical tissue "focus" and the identifiers of other medical tissues of interest, so that the medical tissue "focus" and other medical tissues can be displayed in the first display area 502 of the image display interface , and hide the image models of medical tissues corresponding to the unchecked identifiers in the first display area 502. Organize "focuses" and other medical tissues, and display medical tissues corresponding to unchecked identifiers in a conventional display manner on the multi-planar reconstruction map in the second display area 503 . On this basis, the user can conveniently and intuitively understand the relative positional relationship between the medical tissue "lesion" and other medical tissues of interest. Other situations can be deduced by analogy, and no more examples will be given here.

In some embodiments, as described in the foregoing embodiments, the image to be tested and the target object can be set according to actual applications. For example, the image to be tested can be a medical image, and the target object can be a lesion, but in a real scene, multiple medical images are often scanned. For example, during the enhanced scanning of the liver, multiple medical images including but not limited to portal venous images, arterial images, etc. can be obtained, and other cases can be deduced by analogy, which will not be exemplified here. For example, the plurality of medical images may include a first image and at least one second image. It should be noted that "first" and "second" can be used to distinguish medical images in terms of nomenclature, and do not mean the order of scanning or the degree of importance. For example, different medical images may have different emphases on whether the display of each medical tissue is clear or not. Still taking multiple medical images as an example of multi-phase images obtained by scanning the "liver". Generally speaking, images in the portal venous phase can clearly display the lesion, hepatic portal vein and hepatic vein, but the hepatic artery is not obvious, while the image in the arterial phase The image can clearly show the hepatic artery, but the lesion, hepatic portal vein and hepatic vein are not obvious. Other situations can be deduced by analogy, and no more examples will be given here. Based on this, it is possible to identify a first tissue region of a first medical tissue in a first image, respectively identify a second tissue region of a second medical tissue in at least one second image, and project the second tissue region onto the first The image space of the image, and the first tissue regions of the first medical tissue are located in the image space, on this basis, three-dimensional modeling can be performed based on the tissue regions of each medical tissue in the image space to obtain the image model of the medical tissue, and Medical organizations include a first medical organization and a second medical organization. It should be noted that the image space of the first image can be regarded as the coordinate space where the first image is located, and the dimension of the image space can be specifically determined according to the dimension of the medical image. For example, when the medical image is a three-dimensional volume data, the first image can be regarded as a volume data shaped as a cuboid. On this basis, one of the vertices of the cuboid can be used as the origin of the coordinate space, and based on the edge where the vertex is located , to establish the coordinate axes of the coordinate space, thereby establishing the image space for obtaining the first image. Other situations can be deduced by analogy, and no more examples will be given here; in addition, for the specific process of three-dimensional modeling, please refer to the relevant descriptions in the foregoing application embodiments. In the above method, since the medical tissues in multiple medical images can be fused into the same image space through projection, and the corresponding medical tissues in the image space are displayed on the image display interface, the relationship between medical tissues can be directly displayed directly on the image display interface. The relative positional relationship is conducive to improving the efficiency of image reading.

In an implementation scenario, in order to improve the recognition efficiency, a first region recognition network and a second region recognition network can be pre-trained, on this basis, the first region recognition network can be used to recognize the first image, and the first The first tissue area of the medical tissue is identified, and the second image is identified by using the second area identification network to obtain the second tissue area of the second medical tissue. For example, the first area recognition network may include but not limited to: R-CNN, FCN, etc., which is not limited here. Similarly, the second area recognition network may include but not limited to: R-CNN, FCN, etc., which is not limited here.

In an implementation scenario, taking the first image as the aforementioned portal phase image as an example, before training the first region recognition network, sample images of portal phase images can be collected in advance, and the sample images are marked with the samples to which each pixel belongs category (for example, a certain pixel is marked as belonging to the hepatic portal vein, another pixel is marked as belonging to the hepatic vein, and another pixel is marked as belonging to the lesion), and then the sample image is identified by using the first region recognition network, The predicted category of each pixel in the sample image is obtained, and finally the difference between the sample category and the predicted category can be used to adjust the network parameters of the first region recognition network, so that the first region recognition network learns the liver respectively during the training process. Image features of portal vein, hepatic vein, and lesion. Based on this, the first image can be identified by using the first region recognition network that has been trained and converged to obtain the pixel category to which each pixel in the first image belongs, and the connected domain formed by the pixels belonging to the hepatic portal vein can be used as the hepatic portal vein. For the first tissue area, the connected domain formed by the pixel points belonging to the hepatic vein is used as the first tissue area of the hepatic vein, and the connected domain formed by the pixel points belonging to the lesion is used as the first tissue area of the lesion. Other situations can be deduced by analogy, and no more examples will be given here.

In one implementation scenario, taking the second image as the aforementioned arterial phase image as an example, before training the second region recognition network, a sample image of the arterial phase image can be collected in advance, and the sample image is marked with the sample category to which each pixel belongs ( For example, a certain pixel is marked as belonging to the hepatic artery), and then the second area recognition network is used to identify the sample image to obtain the predicted category to which each pixel in the sample image belongs, and finally the relationship between the sample category and the predicted category can be used The difference is to adjust the network parameters of the second region recognition network, so that the second region recognition network can learn the image features of the hepatic artery during the training process. Based on this, the second image can be identified by using the second region recognition network that has converged in training to obtain the pixel category to which each pixel in the second image belongs, and the connected domain formed by the pixels belonging to the hepatic artery can be used as the hepatic artery. Second organizational area. Other situations can be deduced by analogy, and no more examples will be given here.

In an implementation scenario, in order to improve the accuracy of projection, the second tissue region may be projected into the image space based on the registration parameters between the first image and the second image. The above method can improve the accuracy of the projection, thereby being beneficial to improve the accuracy of the relative positional relationship between the medical tissues visually displayed on the image display interface.

In an implementation scenario, the first target area of the target object in the first image may be identified, and the second target area of the target object in at least one second image may be identified respectively, and for each second image, the second target area and the The first target regions are aligned to obtain registration parameters between the second image and the first image. Taking the example where the target object includes the liver and the first image is the portal phase image, and the second image is the arterial phase image, the first target area of the liver in the portal phase image can be identified, and the second target area of the liver in the arterial phase image can be identified region, by aligning the first target region with the second target region, registration parameters between the portal venous phase image and the arterial phase image can be obtained. Other situations can be deduced by analogy, and no more examples will be given here. In the manner described above, the registration parameters can align the second target area of the target object in the second image with the first target area in the first image, which is beneficial to improve the accuracy of the registration parameters.

In an implementation scenario, in the above alignment process, the second target area needs to undergo rigid body transformations such as rotation and offset and non-rigid body transformations such as deformation to be aligned with the first target area, and the above registration parameters may specifically include a rigid body registration A quasi-matrix and an offset field, on this basis, the rigid body registration matrix can be used to project the second tissue region into the image space, and then the offset field can be used to deform the second tissue region in the image space; or , in the above alignment process, the second target area can be aligned with the second target area only through rigid body transformations such as rotation and offset, and the above registration parameters can specifically include a rigid body registration matrix. On this basis, you can The second tissue region is projected into image space directly using the rigid body registration matrix.

In an implementation scenario, please refer to Fig. 3 or Fig. 4. In the case of multiple medical images including portal venous images and arterial images, after the above identification, projection, and modeling, the liver can be displayed on the image display interface. The portal vein, hepatic vein, hepatic artery, and lesion are simultaneously displayed, so that doctors can intuitively understand the relative positional relationship between the lesion and the hepatic portal vein, hepatic vein, and hepatic artery.

In some embodiments, as described in the foregoing embodiments, the image to be tested and the target object may be set according to actual applications. For example, the image to be tested may be a medical image, the target object may be a lesion on a target organ such as liver or lung, and the medical tissue in the image to be tested may include a lesion. In order to give sufficient reference in the application process such as surgical planning, the target blood vessels of the target organ can be identified, and based on the target blood vessels of the target organ, the target organ can be divided into several watershed segments by using the watershed algorithm. Therefore, the medical tissues may further include the above-mentioned watershed segments, that is, some medical tissues in the medical image are lesions, and some medical tissues are watershed segments of target organs. It should be noted that, in the embodiment of the present application, in addition to the two medical tissues of the lesion and the watershed segment, it does not exclude that the medical image also includes other types of medical tissues, such as other organs, which are not limited here. On this basis, the three-dimensional model of the medical tissue can be displayed based on the rendering parameters of the medical tissue, and the rendering parameters of different medical tissues are not completely the same. For the acquisition process of the 3D model, please refer to the relevant description in the foregoing application embodiments. In the above method, based on the target blood vessels of the target organ, the target organ is divided into several watershed segments by using the watershed algorithm, and based on the rendering parameters of the medical tissue, the three-dimensional model of the medical tissue is displayed, and the medical tissue includes the watershed segment and the lesion. The rendering of the tissue is not exactly the same, so on the one hand, it can visually display and distinguish different medical tissues, and on the other hand, it can also visually show how the lesion invades each watershed segment, so as to provide sufficient reference for doctors in the application process such as surgical planning. Help to improve user experience.

In an implementation scenario, the watershed algorithm can include two types, one is the watershed algorithm based on the overflow process, and its intuitive idea comes from topography, and the other is to correlate the pixels with the catchment area and calculate them to the minimum value The shortest topological distance of . For the specific process of the watershed algorithm, please refer to the specific technical details of the watershed algorithm.

In an implementation scenario, taking the target organ including the liver as an example, and the target blood vessel including the hepatic portal vein, on this basis, the following watershed segmentation can be obtained based on the hepatic portal vein: caudate lobe, upper part of the left outer lobe, left outer lobe Lower segment, left inner lobe, lower right anterior lobe, upper right anterior lobe, lower right posterior lobe, upper right posterior lobe. Other situations can be deduced by analogy, and no more examples will be given here.

In an implementation scenario, data of invasion of the target tissue by the lesion may be detected, and the target tissue may include at least one of target blood vessels and watershed segments, and an early warning prompt may be output based on the invasion data. For example, the invasion data may include at least one of the volume of the lesion, the surface area of the lesion, the long diameter of the lesion, and the short diameter of the lesion. It should be noted that the long diameter of the lesion represents the diameter of the longest point of the lesion, and the short diameter of the lesion represents the The diameter at the shortest point. In addition, in some embodiments, the volume may represent the volume of the intersecting portion of the lesion and the target tissue, the surface area may represent the surface area of the intersecting portion of the lesion and the target tissue, and the long diameter may represent the longest diameter of the intersecting portion of the lesion and the target tissue, The short axis represents the diameter of the shortest point where the lesion intersects with the target tissue. Or, to simplify the statistics, the invasion data may include the proportion of the intersecting part of the lesion and the target tissue in the target tissue. Still taking the liver as an example, for example, the intersecting part of the lesion and the caudate lobe accounts for 1% of the caudate lobe, and the intersecting part of the lesion and the upper segment of the left outer lobe accounts for 5% of the upper segment of the left outer lobe. Give examples. Based on the invasion data above, the malignancy of the lesion can be evaluated, for example, the larger the volume, the higher the malignancy; or the larger the surface area, the higher the malignancy, and so on, and no more examples will be given here. On this basis, based on the degree of malignancy, an early warning prompt of a corresponding level can be output. For example, the higher the degree of malignancy, the higher the output warning level. For example, for high-level early warning prompts, eye-catching methods such as "dark red" and "bold" can be used to prompt, while for low-level early warning prompts, "light red" and other methods can be used to prompt. This is not limited. The above method detects the violation data of the lesion against the target tissue, and outputs an early warning based on the violation data, which can realize the automatic detection of the lesion violation, which is conducive to improving the user experience, and further based on the violation data, the malignancy of the lesion is obtained, and based on the malignancy Outputting the corresponding level of early warning prompts can help users understand the malignancy of the lesion more intuitively and quickly, and is conducive to further improving user experience.

In one implementation scenario, as mentioned above, the organization list can also be displayed, and the organization list includes identifiers of several medical organizations, and the several medical organizations can further include the aforementioned lesion and watershed segmentation. On this basis, it is possible to respond to user In order to select an identifier in the organization list, the medical organization corresponding to the selected identifier is used as the first target organization, and the first target organization is highlighted on the image display interface in a preset manner. In some embodiments, the The medical tissue corresponding to the unselected identifier is used as the second target tissue, and the second target tissue is hidden on the image display interface. The above method can support user-defined selection of watershed segments that are expected to be highlighted on the image display interface, which is conducive to supporting users to focus on the highlighted watershed segments, and eliminating interference from other watershed segments, which is conducive to improving user experience.

In one implementation scenario, as mentioned above, the image display interface includes a first display area and a second display area, the first display area is used to display the three-dimensional model of medical tissue, and the second display area is used to display , and the displayed tissue list includes the identifiers of several medical organizations, and the several medical organizations may further include the aforementioned lesion and watershed segmentation, then in response to the user's selection instruction for the identifier in the tissue list, the The medical organization corresponding to the selected identifier is used as the first target tissue, and the first target tissue is highlighted in the first display area in the first highlighting manner, and the first target tissue is highlighted in the second display area in the second highlighting manner. In some embodiments, the medical tissue corresponding to the unselected identifier can be used as the second target tissue, and the second target tissue is hidden in the first display area, and the second target tissue is displayed in the second display area in a conventional display manner . The above method can not only display watershed segments in different dimensions on the image display interface at the same time, which is conducive to improving the richness of image information displayed on the image display interface, but also can display user expectations in the first display area and the second display area. The prominent watershed segmentation enables users to visually connect the corresponding watershed segments in two different angles, 3D and 2D, to enhance user experience.

The image processing method of the embodiment of the present application can be applied to the scene of computer-aided surgery planning. The scheme of computer-aided surgery planning is: perform imaging analysis on human body image data, segment important organs, blood vessels, and lesions, and visualize each segmentation result Reconstruction, and then develop a detailed and precise surgical plan. Computer-aided surgical planning is divided into three parts: preoperative planning, intraoperative positioning and navigation, and postoperative evaluation; among them, preoperative planning includes the resection or inactivation area planning and preoperative evaluation of lesions; in order to prevent lesion resection or inactivation In clinical practice, a safe distance is introduced into the real target area, that is, the tissues within the safe distance of the lesion are resected or inactivated at the same time without endangering important organs and key blood vessels, so as to achieve the effect of completely killing the lesion. Therefore, the detection of lesion extension area and organ-at-risk is very important in preoperative planning and even in intraoperative navigation and warning, and the real-time and accuracy of the detection results are required to be high.

In related technologies, how to improve the timeliness and accuracy of the detection results for the detection of lesion extension area and endangered organs in the scheme of computer-aided surgery planning is an urgent technical problem to be solved. In view of this technical problem, using the image processing method of the embodiment of the present application, the lesion expansion area and the invasion data of the target tissue can be determined. An implementation scenario will be used to illustrate below.

In an implementation scenario, the image to be tested is a medical image, the target object is a lesion, and the target tissue is an organ. In the embodiment of the present application, the first pixel, the second pixel, and the third pixel can be initialized. For example, the original area where the lesion is located in the medical image is obtained, and the pixels in the original area are the first pixels. It is also possible to detect the violation data of each first pixel point on the target tissue and record the violation data, and the first identifier of the first pixel point is a natural number. For the pixels outside the original area, if it is determined that the 3x3x3 neighborhood of the pixel contains the first pixel, then it is determined that the pixels outside the original area are the second pixels, and the second identifier of the second pixels is a negative number. A pixel in the medical image other than the first pixel and the second pixel is used as a third pixel, and the third identifier of the third pixel is a negative number and is different from the second identifier.

After initializing the first pixel, the second pixel, and the third pixel, the second pixel in the second reference area can be determined based on the voxel coordinates in the medical image and using the image processing method of the foregoing embodiment The physical distance between the point and each first pixel point, the second reference area is the 3x3x3 neighborhood of the second pixel point; it can be determined whether to use the second pixel point as the minimum physical distance between the second pixel point and each first pixel point The new first pixel. In the case that the detection result includes the second pixel as the new first pixel, the third pixel that meets the preset condition can be used as the new second pixel, and the above-mentioned parallel processing of each second pixel can be performed again. Execute the step of expanding detection to obtain the detection result. In the case that it is detected that each second pixel point has been subjected to extension detection, the connected domain formed by the first pixel point in the original area and the first pixel point updated by the second pixel point can be obtained, and the The connected domain is used as the expansion area of the lesion. As mentioned above, the first pixel is assigned a first identifier whose value is a natural number, the second pixel is assigned a second identifier whose value is a negative number, and the third pixel is assigned a third identifier whose value is a negative number. On this basis Above, the connected domain formed by all the pixels corresponding to the identifier whose value is a natural number can be used as the lesion extension area. In the above method, the area expansion ends when all the second pixels have performed the expansion detection, and finally the connected domain formed by all the first pixels is used as the lesion expansion area, which is beneficial to accurately determine the lesion expansion area. area.

In an implementation scenario, the user loads liver CT data to the image analysis software, and all detected lesions can be displayed in the lesion list on the display interface when the image analysis software is running; when the user selects a lesion and clicks on the In the case of a button for expanding the lesion area, the image processing method of the embodiment of the present application can be used to calculate the area of the lesion expanding, and can generate a corresponding visual rendering effect in the window.

The image processing method of the embodiment of the present disclosure can meet the real-time requirements of lesion expansion area and organ-at-risk detection. For example, when the GPU core is used as the calculation kernel of lesion expansion detection, the lesion expansion area can be realized. The millisecond-level calculation improves the user experience to a certain extent.

Those skilled in the art can understand that in the above method of specific implementation, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The inner logic is OK.

Please refer to FIG. 6 , which is a schematic framework diagram of an image processing device provided by an embodiment of the present application. The image processing device 60 may include an original area acquisition module 61, a pixel extension detection module 62, and an outer expansion area acquisition module 63. The original area acquisition module 61 is configured to acquire the original area where the target object is located in the image to be tested, wherein the original area There are several first pixels inside, and there are several second pixels outside the original area; the pixel expansion detection module 62 is configured to perform an expansion detection on each second pixel in parallel to obtain a detection result, wherein the detection result includes whether the The second pixel point is used as a new first pixel point; the extended area obtaining module 63 is configured to obtain the extended area of the target object based on the original area and the detection result.

In some embodiments, the pixel extension detection module 62 is configured to perform the extension detection on each of the second pixels in parallel in a manner that each calculation core runs in parallel. Therefore, each computing core runs in parallel the extension detection performed on each second pixel point, which can further improve the calculation efficiency of region extension.

In some embodiments, the pixel extension detection module 62 includes a reference area acquisition submodule configured to acquire a second reference area surrounding the second pixel point; the pixel extension detection module 62 includes a physical distance calculation submodule configured for the second pixel point The second pixel point and each first pixel point in the reference area obtain the physical distance from each first pixel point to the second pixel point respectively; the pixel expansion detection module 62 includes a detection result acquisition sub-module configured to be based on the physical distance , get the detection result. Therefore, obtaining the second reference area surrounding the second pixel, and obtaining the detection result based on the physical distance from the first pixel to the second pixel in the second reference area, can reduce the possibility of the area expanding beyond the safety distance, It is conducive to improving the security of regional expansion.

In some embodiments, there are several third pixel points outside the original area, and there is no intersection between the first reference area where each third pixel point is located and the original area; the image processing device 60 also includes a duplication detection module configured to detect The result includes taking the second pixel as the new first pixel, taking the third pixel that meets the preset condition as the new second pixel, and re-executing the extension detection for each second pixel in parallel , the step of obtaining the detection result; wherein, the preset condition includes: the first reference area where the third pixel is located contains the first pixel updated by the second pixel.

In some embodiments, the expanded area acquiring module 63 is specifically configured to acquire the updated first pixel point and the second pixel point in the original area in response to detecting that each second pixel point has performed the expanded detection. A connected domain formed by a pixel, and the connected domain is used as an extended area. Therefore, when it is detected that each second pixel point has performed the extension detection, the connected domain formed by the first pixel point in the original area and the first pixel point updated by the second pixel point is obtained, and The connected domain is used as the expansion area, that is, the expansion of the area ends with the expansion detection of each second pixel, and the connected domain formed by all the first pixels is used as the expansion area, which is conducive to improving the expansion. area accuracy.

In some embodiments, the image to be tested is a medical image, the target object is a lesion, and the medical image also includes several target tissues; the image processing device 60 also includes an infringement detection module configured to detect the impact of each first pixel on several target tissues. Infringement data of the organization; wherein, the infringement data includes: the target organization infringed by the first pixel. Therefore, the image to be tested is a medical image, and the target object is a lesion. The medical image also includes several target tissues. Including the target tissue violated by the first pixel, that is, the infringement data of each first pixel on the target tissue can be recorded, so that reference information can be provided during the application process, which is conducive to improving user experience.

In some embodiments, multiple target objects are identified in the image to be tested; the image processing device 60 also includes a list display module configured to display a list of objects; wherein, the object list includes identifiers of multiple target objects; the image processing device 60 It also includes an external expansion interaction module configured to respond to the identifier being in the selected state and the external expansion instruction input by the user, combined with the pixel external expansion detection module 62 and the external expansion area acquisition module 63 for the target object corresponding to the identifier. The step of performing the expansion detection on the second pixel in parallel to obtain the detection result and the subsequent steps to obtain the expansion area. Therefore, a plurality of target objects are identified in the image to be tested. On this basis, an object list is displayed, and the object list includes identifiers of a plurality of target objects. , for the target object corresponding to the identifier, execute the expansion detection for each second pixel point in parallel, obtain the detection result and the subsequent steps, and obtain the expansion area, so it is possible to have multiple target objects in the image to be tested In this case, regional expansion is performed according to user instructions, which is conducive to improving user experience.

In some application embodiments, the image processing device 60 includes a triangulation module configured to triangulate the surface mesh of the target object based on the extended area; the image processing device 60 includes a model rendering module configured to use the rendering parameters of the target object Rendering the surface mesh of the target object to obtain an image model of the target object; the image processing device 60 includes a model display module configured to display the image model of the target object on an image display interface.

The above-mentioned original area acquisition module 61, pixel extension detection module 62, extension area acquisition module 63, repetition detection module, infringement detection module and triangulation module can all be implemented based on the processor of the electronic device.

Please refer to FIG. 7 . FIG. 7 is a schematic frame diagram of an electronic device provided by an embodiment of the present application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other, and the processor 72 is configured to execute program instructions stored in the memory 71 to implement the steps of any one of the above image processing method embodiments. In an implementation scenario, the electronic device 70 may include, but is not limited to: a microcomputer and a server. In addition, the electronic device 70 may also include mobile devices such as notebook computers and tablet computers, which are not limited here.

Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps of any one of the above image processing method embodiments. The processor 72 may also be called a central processing unit (Central Processing Unit, CPU). The processor 72 may be an integrated circuit chip with signal processing capability. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. In addition, the processor 72 may be jointly implemented by an integrated circuit chip.

The above scheme, on the one hand, because the expansion detection is performed based on each second pixel point, it can realize pixel-level area expansion, which is beneficial to improve the accuracy of area expansion; on the other hand, because the expansion detection is performed in parallel , so it is beneficial to increase the speed of regional expansion. Therefore, the accuracy and speed of area expansion can be improved.

Please refer to FIG. 8 . FIG. 8 is a schematic framework diagram of a computer-readable storage medium provided by an embodiment of the present application. The computer-readable storage medium 80 stores program instructions 801 that can be executed by the processor, and the program instructions 801 are used to implement the steps of any of the above image processing method embodiments.

In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the method embodiments above, and its specific implementation can refer to the description of the method embodiments above. For brevity, here No longer.

In the several embodiments provided in this application, it should be understood that the disclosed methods and devices may be implemented in other ways. For example, the device implementations described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

A unit described as a separate component may or may not be physically separated, and a component shown as a unit may or may not be a physical unit, that is, it may be located in one place, or may also be distributed to network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the methods in various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

Industrial Applicability

The embodiment of the present application discloses an image processing method, a related device, electronic equipment, a computer storage medium and a computer program. The image processing method includes: obtaining the original area where the target object is located in the image to be tested, wherein the original area contains a number of first pixels, and there are a number of second pixels outside the original area; parallelly performing external expansion detection on each second pixel , to obtain a detection result, wherein the detection result includes whether the second pixel is used as a new first pixel; based on the original area and the detection result, an extended area of the target object is obtained. The above solution can improve the accuracy and speed of area expansion.

Claims

An image processing method applied to electronic equipment, the method comprising:

Obtaining the original area where the target object is located in the image to be tested, wherein the original area contains a number of first pixels, and there are a number of second pixels outside the original area;

Executing extension detection in parallel on each of the second pixels to obtain a detection result, wherein the detection result includes whether the second pixel is used as the new first pixel;

Based on the original area and the detection result, an extended area of the target object is obtained.
The method according to claim 1, wherein the parallel execution of the expansion detection on each of the second pixels comprises: performing parallel execution on each of the second pixels in a manner in which each computing core runs in parallel Expansion detection.
The method according to claim 1 or 2, wherein the first reference areas where the respective second pixel points are respectively located overlap with the original area.
The method according to any one of claims 1 to 3, wherein, performing the extension detection on each of the second pixel points in parallel to obtain a detection result includes:

Acquiring a second reference area surrounding the second pixel, wherein at least one of the first pixel is contained in the second reference area;

Obtaining physical distances from each of the first pixels to the second pixels based on the second pixels and each of the first pixels in the second reference area;

Based on the physical distance, the detection result is obtained.
The method according to claim 4, wherein said obtaining said detection result based on said physical distance comprises:

When the minimum physical distance is lower than a preset threshold, determining the detection result includes using the second pixel as the new first pixel;

In a case where the minimum physical distance is not lower than the preset threshold, determining the detection result includes not using the second pixel as the new first pixel.
The method according to any one of claims 1 to 5, wherein there are several third pixel points outside the original area, and the first reference area where each of the third pixel points are respectively located is different from the original area. There is an intersection; before obtaining the extended area of the target object based on the original area and the detection result, the method further includes:

If the detection result includes the second pixel as the new first pixel, use the third pixel satisfying the preset condition as the new second pixel, and re-execute the The step of performing out-expansion detection in parallel on each of the second pixel points to obtain a detection result;

Wherein, the preset condition includes: the first reference area where the third pixel is located contains the first pixel updated by the second pixel.
The method according to any one of claims 1 to 6, wherein said obtaining the extended area of the target object based on the original area and the detection result comprises:

Responsive to detecting that each of the second pixel points has performed the expansion detection, acquiring the first pixel point in the original area formed by the first pixel point and the first pixel point updated by the second pixel point The connected domain of , and the connected domain is used as the extension area.
The method according to any one of claims 1 to 7, wherein the image to be tested is a medical image, the target object is a lesion, and the medical image further includes several target tissues; the method further comprises:

Detecting violation data of each of the first pixel points on the plurality of target tissues; wherein the violation data includes: the target tissue violated by the first pixel point.
The method according to any one of claims 1 to 8, wherein the image to be tested includes a plurality of target objects; the method further comprises:

displaying a list of objects; wherein the list of objects includes identifiers of the plurality of target objects;

In response to the identifier being in the selected state and the expansion instruction input by the user, for the target object corresponding to the identifier, perform the step of performing expansion detection on each of the second pixel points in parallel to obtain a detection result and subsequent steps to obtain the expanded area.
The method according to any one of claims 1 to 9, wherein, after obtaining the extended area of the target object based on the original area and the detection result, the method further comprises:

Triangulating to obtain a surface mesh of the target object based on the expanded area;

Rendering the surface mesh of the target object by using the rendering parameters of the target object to obtain an image model of the target object;

The image model of the target object is displayed in the image display interface.
An image processing device, comprising:

The original area acquisition module is configured to acquire the original area where the target object is located in the image to be tested, wherein the original area contains a number of first pixels, and there are a number of second pixels outside the original area;

The pixel expansion detection module is configured to perform expansion detection on each of the second pixels in parallel to obtain a detection result, wherein the detection result includes whether the second pixel is used as a new first pixel;

The expanded area acquisition module is configured to obtain the expanded area of the target object based on the original area and the detection result.
The device according to claim 11, wherein the pixel extension detection module is configured to perform the extension detection on each of the second pixels in parallel in a manner in which each calculation kernel runs in parallel.
The device according to claim 11 or 12, wherein the first reference areas where the respective second pixel points are respectively located overlap with the original area.
The device according to any one of claims 11 to 13, wherein the pixel extension detection module includes a reference area acquisition sub-module configured to acquire a second reference area surrounding the second pixel; the pixel outer The expanded detection module includes a physical distance calculation submodule configured to obtain the distance from each of the first pixel points to the second pixel points for the second pixel points and each of the first pixel points in the second reference area, respectively. physical distance; the pixel extension detection module includes a detection result acquisition submodule configured to obtain the detection result based on the physical distance; the second reference area contains at least one of the first pixel points.
The device according to claim 14, wherein the detection result acquisition submodule includes a first determination unit configured to determine the detection result when the minimum physical distance is lower than a preset threshold. The second pixel is used as the new first pixel; the detection result acquisition sub-module includes a second determination unit configured to determine that the minimum physical distance is not lower than the preset threshold The detection result includes not using the second pixel as the new first pixel.
The device according to any one of claims 11 to 15, wherein there are several third pixel points outside the original area, and neither the first reference area where each of the third pixel points is located nor the original area exists Intersection; the image processing device further includes a duplication detection module configured to, when the detection result includes the second pixel as the new first pixel, select the third pixel that meets the preset condition point as the new second pixel point, and re-execute the step of performing extension detection on each of the second pixel points in parallel to obtain the detection result; wherein, the preset condition includes: the third pixel The first reference area where the point is located includes the first pixel point updated by the second pixel point.
The device according to any one of claims 11 to 16, wherein the expansion area acquisition module is specifically configured to, in response to detecting that each of the second pixel points have performed the expansion detection, acquire the A connected domain formed by the first pixel in the original area and the first pixel updated by the second pixel is used as the extended area.
The device according to any one of claims 11 to 17, wherein the image to be tested is a medical image, the target object is a lesion, and the medical image further includes several target tissues; the image processing device further An infringement detection module is included, configured to detect the infringement data of each of the first pixel points on the plurality of target tissues; wherein the infringement data includes: the target tissue violated by the first pixel point.
The device according to any one of claims 11 to 18, wherein the image to be tested includes a plurality of target objects; the image processing device further includes a list display module configured to display an object list; wherein the object list including the identifiers of the plurality of target objects; the image processing device further includes the expansion interaction module configured to respond to the identifier being in a selected state and the expansion instruction input by the user, combined with pixel expansion detection The module and the extended area acquisition module perform the step of performing extended detection on each of the second pixel points in parallel for the target object corresponding to the identifier to obtain the detection result and subsequent steps to obtain the expanded area.
The device according to any one of claims 11 to 19, wherein the image processing device includes a triangulation module configured to triangulate the surface mesh of the target object based on the expanded area; the image The processing device includes a model rendering module configured to use the rendering parameters of the target object to render the surface mesh of the target object to obtain an image model of the target object; the image processing device includes a model display module configured to The image model of the target object is displayed in the image display interface.
An electronic device, comprising a memory and a processor coupled to each other, the processor is configured to execute program instructions stored in the memory, so as to realize the image processing method according to any one of claims 1 to 10.
The electronic device according to claim 21, wherein the electronic device is a terminal device or a server.
A computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.
A computer program, comprising computer-readable codes, when the computer-readable codes run in an electronic device, a processor in the electronic device executes the image processing method described in any one of claims 1 to 10 .