WO2023072146A1

WO2023072146A1 - Transluminal ultrasonic automatic inspection system, control method, computer-readable storage medium and electronic device

Info

Publication number: WO2023072146A1
Application number: PCT/CN2022/127689
Authority: WO
Inventors: 史轶伦; 史策; 李灏; 陈文波
Original assignee: 北京智愈医疗科技有限公司
Priority date: 2021-10-26
Filing date: 2022-10-26
Publication date: 2023-05-04
Also published as: CN113951935A

Abstract

A transluminal ultrasonic automatic inspection system and a control method, the system comprising: an ultrasonic imaging apparatus, wherein an ultrasonic probe can enter the body by means of a cavity, and a first ultrasonic image sequence and a second ultrasonic image sequence are generated; a motion control apparatus, which comprises a multi-degree-of-freedom driving mechanism, the multi-degree-of-freedom driving mechanism being connected to the ultrasonic probe; and a processor, which is connected to the ultrasonic imaging apparatus and the motion control apparatus, the processor obtaining the first ultrasonic image sequence and the second ultrasonic image sequence and processing the first ultrasonic image sequence and the second ultrasonic image sequence to generate a control signal; and the motion control apparatus receives the control signal, drives, on the basis of the control signal, the ultrasonic probe to move, and completes an automatic inspection process. Therefore, automatic and refined control of the ultrasonic inspection process may be achieved by the transluminal ultrasonic automatic inspection system and control method.

Description

A transcavity ultrasonic automatic inspection system, control method, computer-readable storage medium and electronic equipment

This application is based on the Chinese patent application CN202111245941.8 (application date: October 26, 2021), and enjoys its priority. This application incorporates this application in its entirety by reference.

technical field

The present application relates to a medical device, especially a transcavity ultrasonic automatic inspection system and its control method, a computer-readable storage medium, and an electronic device.

Background technique

The application of ultrasonic diagnostic instruments in clinical diagnosis has been very popular, and it has made great contributions to doctors' accurate understanding of patients' conditions and formulation of medical plans. For example, for intravaginal examinations (eg, to examine the uterus, ovaries, etc.), intrarectal examinations (eg, for the detection of rectal wall cancer, prostatic hyperplasia, prostate cancer, etc.), and intracavitary examinations.

Conventional ultrasound systems are commonly used in large medical facilities (eg, hospitals) and are operated by medical professionals, such as ultrasound technicians, who have experience with these systems. Ultrasound technicians typically receive years of hands-on training to learn how to properly use an ultrasound imaging system. For example, an ultrasound technician can learn how to properly position an ultrasound device on a subject to capture ultrasound images in various anatomical views. Additionally, ultrasound technicians can learn how to read captured ultrasound images to infer medical information about a patient. However, the operation of ultrasound technicians is limited by the experience and technical level of the technicians. It is difficult to guarantee the stability and convenience of the inspection, and it is also difficult to promote the application of ultrasound inspection more widely.

Contents of the invention

The present application proposes an automatic ultrasonic inspection system and a control method thereof. The automatic ultrasonic inspection system can realize automatic inspection in a cavity environment.

The embodiment of the present application proposes a transcavity ultrasonic automatic inspection system, including:

An ultrasonic imaging device, the ultrasonic imaging device includes an ultrasonic probe, the ultrasonic probe can enter the body through a cavity, and generate a two-dimensional first ultrasonic image sequence and a two-dimensional second ultrasonic image sequence;

A motion control device, the motion control device comprising a multi-degree-of-freedom drive mechanism connected to the ultrasonic probe;

a processor, connected to the ultrasonic imaging device and the motion control device, the processor acquires the first sequence of ultrasonic images and the second sequence of ultrasonic images, and performs an operation on the first sequence of ultrasonic images and/or The second ultrasound image sequence is processed to generate a control signal;

The motion control device receives the control signal, drives the ultrasonic probe to move based on the control signal, and completes the automatic inspection process.

According to the present invention, the ultrasonic probe can be controlled based on one or both of the two ultrasonic image sequences, which improves the reliability of the ultrasonic probe control.

Preferably, in the transcavity ultrasonic automatic inspection system proposed in the present application, the first ultrasonic image sequence is an ultrasonic tomographic image sequence, and the second ultrasonic image sequence is an ultrasonic sagittal image sequence.

Preferably, during the movement of the ultrasonic probe, the ultrasonic imaging device continuously acquires the first ultrasonic image sequence and the second ultrasonic image sequence according to a set step size.

Preferably, the processor extracts image contour information in the first and/or second ultrasound image sequence for judging whether the ultrasound probe reaches the end position. When the ultrasound probe reaches the end position, the processing The device generates a first control instruction to control the ultrasonic probe to stop moving.

Preferably, when the processor judges that the image quality of any one of the first ultrasonic image sequence and the second ultrasonic image sequence is sufficient to determine the movement end point of the ultrasonic probe, the first ultrasonic image sequence is used. One of the ultrasound image sequence and the second ultrasound image sequence is used to judge the movement end point of the ultrasound probe,

When the processor judges that the image quality of any one of the first ultrasound image sequence and the second ultrasound image sequence is not enough to judge the motion end point of the ultrasound probe, use the first ultrasound Both the image sequence and the second ultrasound image sequence are used to determine the motion end point of the ultrasound probe.

According to the present invention, according to the image quality of the first ultrasonic image sequence and the second ultrasonic image sequence, one or both of the first ultrasonic image sequence and the second ultrasonic image sequence can be flexibly selected to control the ultrasonic probe, thereby as far as possible It is possible to suppress the calculation load of the processor.

Preferably, when using both the first ultrasonic image sequence and the second ultrasonic image sequence to determine the movement end point of the ultrasonic probe, the processor The ultrasonic image sequence establishes a three-dimensional reconstruction coordinate system, and reconstructs the two-dimensional first ultrasonic image sequence and the two-dimensional second ultrasonic image sequence into the three-dimensional reconstruction coordinate system, thereby obtaining the fusion of the first a three-dimensional reconstructed image of the image information of the ultrasound image sequence and the image information of the second ultrasound image sequence,

And, performing interpolation processing on the 3D reconstructed image, and mapping the interpolated 3D reconstructed image into the two-dimensional first ultrasound image sequence and the two-dimensional second ultrasound image sequence, so as to obtain the repaired The first sequence of ultrasonic images and the second sequence of ultrasonic images after repairing, and using the first sequence of ultrasonic images after repairing and/or the second sequence of ultrasonic images after repairing to judge the movement end point of the ultrasonic probe.

According to the present invention, two ultrasonic image sequences are reconstructed into a three-dimensional reconstructed image, and interpolation processing is performed on the three-dimensional reconstructed image, thereby obtaining a three-dimensional reconstructed image with higher image quality. Furthermore, by mapping the interpolated three-dimensional reconstructed image to the original two-dimensional first ultrasonic image sequence and the two-dimensional second ultrasonic image sequence, the images of the first ultrasonic image sequence and the second ultrasonic image sequence can be enhanced quality.

Preferably, when the ultrasonic probe reaches the end position, the processor generates a second control command to control the ultrasonic probe to withdraw from the cavity.

Preferably, the processor can also generate a third control command, controlling the ultrasonic probe to move to a corresponding position in the cavity and then suspending the movement.

Preferably, the processor generates the third control instruction based on an input from an input device, or the processor generates the third control instruction based on a recognition result of the first or second ultrasound image sequence.

Preferably, the imaging parameters of the ultrasonic imaging device can be adjusted when the ultrasonic probe is suspended.

Preferably, the ultrasonic imaging device further includes a sheath, and the ultrasonic probe slides in the sheath.

The application also proposes a control method for a transcavity ultrasonic automatic inspection system, comprising the following steps:

Acquiring a first sequence of ultrasound images and a sequence of second ultrasound images according to a predetermined step during the movement of the ultrasound probe;

extracting image contour information in the first ultrasound image sequence and/or the second ultrasound image sequence;

According to the extracted image contour information, it is judged whether the ultrasonic probe has reached the terminal position;

When the ultrasonic probe reaches the terminal position, the ultrasonic probe is controlled to stop moving.

According to the control method of the ultrasonic automatic inspection system provided in the present application, preferably, it further includes the step of recording the starting position of the ultrasonic probe movement.

Preferably, after the ultrasonic probe stops moving, the step of controlling the ultrasonic probe to withdraw from the cavity is further included.

Preferably, during the process of the ultrasonic probe exiting the cavity, the step of controlling the movement of the ultrasonic probe to a corresponding position in the cavity and pausing the movement is further included.

Preferably, when the ultrasonic probe moves to a corresponding position in the cavity and stops moving, a step of adjusting ultrasonic imaging parameters is also included.

The present application also proposes a computer-readable storage medium on which a computer program is stored, wherein the program implements the method described in any embodiment of the present application when executed by a processor.

The present application also proposes an electronic device, including a memory, a processor, and a computer program stored on the memory and capable of running on the processor. When the processor executes the computer program, the computer program described in any embodiment of the present application is implemented method.

The above at least one technical solution adopted in the embodiment of the present application can achieve the following beneficial effects:

Under the control of the motion control device, the ultrasonic probe can complete the movement process of automatic inspection based on the control signal provided by the processor; it can judge the movement position of the ultrasonic probe based on the ultrasonic image sequence acquired during the movement of the ultrasonic probe, and according to the movement position By controlling the movement of the ultrasonic probe, the system provided by the present application can realize the automation of the transcavity ultrasonic examination process with less configuration. On the basis of the system and method provided in this application, the movement process of the ultrasonic probe can be controlled in real time, and further, during the movement process of the ultrasonic probe, the pause position can be selected according to the imaging needs, and the accuracy of the surrounding tissue at the pause position can be adjusted. High-quality ultrasonic imaging realizes the automation and fine control of the ultrasonic examination process.

Description of drawings

The drawings described here are used to provide a further understanding of the application and constitute a part of the application. The schematic embodiments and descriptions of the application are used to explain the application and do not constitute an improper limitation to the application. In the attached picture:

Fig. 1 is a schematic diagram of an embodiment of the ultrasonic automatic inspection system of the present application;

Fig. 2 is a schematic diagram of the connection relationship between various devices included in the ultrasonic automatic inspection system;

FIG. 3A is a schematic diagram of an embodiment of the control method of the ultrasonic automatic inspection system of the present application;

FIG. 3B is a detailed flow diagram of the control method of the ultrasonic automatic inspection system of the present application;

Fig. 4 is a schematic diagram of an embodiment of obtaining a sequence of tomographic images by an ultrasonic probe during movement in a cavity;

FIG. 5A is an example of an ultrasonic tomographic image of a target tissue acquired by an ultrasonic probe during movement;

FIG. 5B is another example of an ultrasonic tomographic image of a target tissue acquired by the ultrasonic probe during movement;

6 is a schematic diagram of an embodiment of an ultrasonic probe acquiring a sequence of ultrasonic sagittal images during movement in the cavity;

7 is a schematic diagram of a sliding window of an ultrasound sagittal plane image sequence;

8A-8E are schematic diagrams of the principle of detecting the end point position based on the ultrasonic sagittal image sequence.

Fig. 9 is a schematic diagram of an embodiment of an ultrasonic probe in the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments of the present application and corresponding drawings. Apparently, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

Different from the ultrasonic inspection system in the prior art, the embodiment provided in the present application can control the automatic movement of the ultrasonic probe used for the ultrasonic inspection in the lumen, so as to complete the automatic ultrasonic inspection of the lumen. As an example of the natural cavity of the human body, the embodiments provided in this application mainly describe the process of automatic ultrasonic examination of prostate tissue through the rectal cavity. However, those skilled in the art should understand that the system and control method of the present application are also It can be applied to the automatic ultrasonography of other cavity (for example: digestive tract, urinary tract, reproductive tract, nasal cavity, external auditory canal and nasolacrimal duct, etc.).

The technical solutions provided by various embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

As shown in FIG. 1 , the ultrasonic automatic inspection system described in this application includes an ultrasonic imaging device, a motion control device, and a processor. Wherein, the ultrasonic imaging device and the motion control device are connected with the processor to transmit image data and control signals. The motion control device is connected with the ultrasonic imaging device to control the movement of the ultrasonic probe in the ultrasonic imaging device.

As shown in Figure 2, in some embodiments, the motion control device of the ultrasonic automatic inspection system described in the present application may include at least one mechanical arm, one end of which is connected to the ultrasonic imaging probe of the ultrasonic imaging device, the The other end of the arm is connected to a fixed base. The motion control device includes a control unit, and the mechanical arm can receive a control signal from the control unit and move to drive the ultrasonic imaging probe to move in the cavity.

Since the human body cavity is a slender channel, such as the rectum, the ultrasonic probe used in the ultrasonic imaging device described in this application is a slender structure. Figure 9 provides an embodiment of an ultrasonic probe used in this application, such as As shown in the figure, the ultrasonic probe main body 103 is an elongated structure with a certain rigidity, the end of the probe main body 103 is coupled to the front end of the mechanical arm, and the front end of the probe main body 103 is equipped with a first ultrasonic probe 101 and a second ultrasonic probe 102, Both the first ultrasonic probe 101 and the second ultrasonic probe 102 are installed in the installation groove provided by the ultrasonic probe main body 103 . Through the joint use of the first ultrasonic probe 101 and the second ultrasonic probe 102, multiple ultrasonic imaging images can be obtained, and all-round image acquisition of the target tissue to be inspected can be realized.

During the advancement of the ultrasonic probe body 103 along the cavity, the first ultrasonic probe 101 is used to acquire ultrasonic tomographic images of the target tissue. The second ultrasonic probe 102 is used to acquire an ultrasonic sagittal image of the target tissue. The first ultrasonic probe 101 and the second ultrasonic probe 102 are different types of probes. As an example, the first ultrasonic probe 101 is a convex array ultrasonic probe, and the second ultrasonic probe 102 is a linear array ultrasonic probe. It can be understood that the first ultrasonic probe 101 and the second ultrasonic probe 102 may also be the same type of ultrasonic probes, for example, both are linear array ultrasonic probes. It is only necessary to set the first ultrasonic probe as a probe capable of acquiring a tomographic image of the target tissue, and the second ultrasonic probe as a probe capable of acquiring a sagittal plane image of the target tissue.

Simultaneously start the first ultrasonic probe 101 and the second ultrasonic probe 102 to continuously collect ultrasonic images according to the predetermined step length during the movement of the ultrasonic probe along the cavity, and obtain ultrasonic tomographic images at each step position through the first ultrasonic probe to form Ultrasonic tomographic image sequences S1, S2, S3...Sn, ultrasonic sagittal plane images are acquired at each step position by the second ultrasonic probe to form ultrasonic sagittal plane image sequences T1, T2, T3...Tn.

The first ultrasonic probe used to acquire ultrasonic tomographic images may be a ring-shaped probe disposed on the main body of the ultrasonic probe, so as to ensure the acquisition of image information of a full field of view. The first ultrasonic probe can also be a convex array probe or a linear array probe or other known probes. When using this type of probe, the first ultrasonic probe can be rotated while keeping the axis of the main body of the ultrasonic probe stationary to ensure its scanning The field of view can cover the entire target tissue.

Based on the ultrasonic tomographic image sequences S1, S2, S3...Sn, the three-dimensional reconstruction of the ultrasonic images can be realized, and the reconstructed three-dimensional images can be displayed in real time through the display device. Based on the analysis of the ultrasonic tomographic image sequence and/or the ultrasonic sagittal image sequence, data for judging the movement process of the ultrasonic probe can also be obtained through calculation. At the same time, based on the analysis of the ultrasonic tomographic image sequence and the ultrasonic sagittal image sequence, the calculated data will be more accurate, which is conducive to more accurate judgment and control of the movement process of the ultrasonic probe.

Preferably, the ultrasonic imaging device further includes a sheath. In order to ensure the safety of the medical environment, the sheath can be disposable, used in each automatic ultrasonic inspection, and discarded after the inspection is completed. The sheath used in the embodiment of the present application has a certain hardness to ensure that the sheath can be pre-inserted into the rectum and maintain a relatively stable position. The sheath is an axially elongated shell-like part whose shape roughly matches the ultrasound probe. Before inserting the elongated ultrasound probe body into the rectum, a sheath may be inserted in the lumen and the ultrasound probe is then slid along the inner wall of the sheath. In this way, the sheath provides a more stable environment for the motion of the ultrasound probe, which is beneficial to the precise calculation and control of the motion data of the ultrasound probe. As a preference, a couplant can also be filled in the sheath to isolate the air and improve the accuracy of the ultrasonic probe inspection, and, in the case of filling the couplant, the couplant also provides lubrication for the ultrasonic probe, which can reduce the impact of the ultrasonic probe. The friction between the probe and the surrounding environment during the insertion process makes the insertion process of the ultrasonic probe smoother, and the movement control of the ultrasonic probe will be more precise.

As shown in Figure 2, in some embodiments, the ultrasonic automatic inspection system described in the present application includes a processor, the processor is connected to an ultrasonic imaging device, and can obtain the results of the first ultrasonic probe and the second ultrasonic probe from the ultrasonic imaging device. The acquired ultrasound image data includes an ultrasound tomographic image sequence and an ultrasound sagittal image sequence. The processor performs processing and analysis on the acquired ultrasonic image data, that is, the ultrasonic tomographic image sequence and the ultrasonic sagittal image sequence, including extracting contours, and generates a control signal according to the processing and analysis results. The processor is also connected to the motion control device to send control signals to the control unit of the motion control device, and the control unit controls the movement process of the mechanical arm and the ultrasonic probe.

Preferably, the processor is also connected with a display device, which is used for real-time display of ultrasonic images acquired during the movement of the ultrasonic probe, can display the reconstructed three-dimensional image, and mark the extracted tissue contour information, the display device The relevant parameters during the operation of the ultrasonic imaging device can also be displayed, so that operators including doctors or technicians can grasp the parameters and data of the ultrasonic examination in real time.

As preferably, the processor is also connected with an input device, the input device can be various known devices capable of inputting information to the processor such as a touch screen, keyboard, mouse, etc., through the input device, it can be convenient for doctors or technicians The operator selects an image of a specific position and a specific frame in the acquired ultrasonic image sequence, or on the reconstructed three-dimensional image, and the processor can determine its position information according to the image selected by the operator, and generate a control signal to drive the The ultrasonic imaging probe is moved to a position corresponding to the image. After moving to this position, the operator can further adjust the imaging parameters of the ultrasonic imaging device through the input device, so as to facilitate more detailed observation.

Wherein, the motion control device may be a known mechanical arm and its control unit. Driven by the control unit, the mechanical arm can at least perform a translational movement along the line where the cavity or the sheath is located. Preferably, the robotic arm can have multiple joints to achieve more complex movements.

When the ultrasonic automatic inspection system described in this application is used in the ultrasonic automatic inspection scene, the mechanical arm is the active arm, but the device in the system described in the present application can also be used in the manual inspection scene, and the mechanical arm can also be the passive arm. The mechanical arm is connected to the fixed base, which is the connection mechanism for fixing the probe adapter and the drive module to the fixed base, such as 6-axis and 7-axis manipulators, fixed brackets, etc., and provides a reference for the probe adapter and drive module Location.

FIG. 3A provides a control method of the ultrasonic automatic inspection system, and FIG. 3B is a detailed flowchart design diagram of the control method. The control method of the ultrasonic automatic inspection system described in this application will be described in detail below with reference to FIG. 3A and FIG. 3B .

Before the automatic inspection starts, the mechanical arm in the motion control device is coupled and fixed to the end of the main body of the ultrasonic imaging probe, and the mechanical arm is driven to move the ultrasonic imaging probe to the vicinity of the cavity entrance of the patient to be examined. And the axial direction of the main body of the ultrasonic imaging probe is aligned with the axial direction of the cavity to be inspected or the sheath that has been inserted into the cavity.

Start the ultrasonic imaging device to obtain ultrasonic images. The time at this time is recorded as the initial time t1. At this time, the position of the ultrasonic probe in the cavity is recorded as the initial position p1. At this time, the image acquired by the convex array probe of the ultrasonic imaging probe is the ultrasonic The tomographic image S1, the image acquired by the linear array probe of the ultrasound imaging probe is the ultrasound sagittal plane image T1, and the above-mentioned time or position data is associated with the above-mentioned image data.

The motion control device drives the mechanical arm to drive the ultrasonic probe to advance along the cavity or the sheath, and the ultrasonic probe collects and obtains ultrasonic image data according to a predetermined step during the advancing process. This predetermined step size is closely related to the accuracy required for inspection, and the step size can be set according to time or distance. This embodiment provides automatic ultrasonic examination in the rectum. As a preference, the predetermined step length is set to a distance of 1 mm, and the ultrasonic image data is collected and recorded every time the ultrasonic probe moves a distance of 1 mm.

During the movement of the ultrasonic probe, the ultrasonic image data is collected and recorded according to the predetermined step distance, wherein, at the position p2, the ultrasonic tomographic image S2 and the ultrasonic sagittal plane image T2 are acquired; at the position p3, the ultrasonic tomographic image S3, ultrasonic tomographic image Sagittal plane image T3; ... and so on, at position pn, an ultrasonic tomographic image Sn and an ultrasonic sagittal plane image Tn are acquired. The ultrasonic image sequence collected in the above process can be expressed as {p1, S1, T1}, {p2, S2, T2}, {p3, S3, T3}, ... {pn, Sn, Tn}, store the above ultrasonic image sequence data.

In some embodiments, the step size can also be set according to the time. During the movement of the ultrasonic probe, the ultrasonic image data is collected and recorded according to the predetermined step time: at time t2, the ultrasonic tomographic image S2 and the ultrasonic sagittal image T2 are acquired ; At time t3, the ultrasound tomographic image S3 and the ultrasound sagittal plane image T3 are acquired; ... and so on, at time tn, the ultrasound tomographic image Sn and the ultrasound sagittal plane image Tn are acquired. The ultrasonic image sequence collected in the above process can be expressed as {t1, S1, T1}, {t2, S2, T2}, {t3, S3, T3}, ... {tn, Sn, Tn}, store the above ultrasonic image sequence data.

In the above manner, during the movement of the ultrasound probe along the cavity or the sheath, a sequence of ultrasound tomographic images and a sequence of ultrasound sagittal plane images are acquired according to a predetermined step length, and the ultrasound imaging device sends the acquired ultrasound image sequences to the processor in real time.

When the ultrasonic probe advances along the cavity, the processor processes and analyzes each frame of image data in the received ultrasonic image sequence, including operations such as contour extraction. According to the result of processing and analysis, it is judged whether the ultrasonic probe moves to the end position, if no, no instruction is issued, the ultrasonic probe continues to advance along the cavity along the original path, and continues to acquire ultrasonic images according to the predetermined step length; if yes, A first control instruction is then generated, which instructs the ultrasound probe to stop advancing. The processing and analysis process is preferably real-time, so as to make judgments and generate motion control instructions based on image analysis results in a timely manner.

As shown in FIG. 4 , the ultrasonic probe stops moving forward according to the first control instruction, and at this time, the ultrasonic probe stops at the end position. The advancing process of the ultrasonic probe along the cavity is completed. Through the entire advancing process, the ultrasonic probe has acquired all the ultrasonic image data sequences including all regions of the target tissue. At this time, the processor can perform three-dimensional reconstruction to form a holistic 3D image of the target tissue.

After a certain time after the ultrasonic probe stops moving (this time can be set according to needs, for example, it can be approximately equivalent to the time required for the processor to complete the three-dimensional reconstruction and display), the processor generates a second control command, and the second control command instructs the ultrasonic probe to Change the direction of motion and start to retreat along the lumen.

During the process of the ultrasonic probe being retracted along the cavity, the processor can also generate a third control instruction, the third control instruction instructs the ultrasonic probe to move to the specified pause position p', and the ultrasonic probe moves to the pause position according to the third control instruction After p', stop the movement for a certain period of time. During the time when the ultrasound probe stops at the pause position p', the operator can modify or adjust the ultrasound imaging parameters, select the ultrasound imaging mode at the pause position p', modify the displayed information, or perform other operations. The pause time of the ultrasonic probe at the pause position p' can be set according to the required operation time. After pausing for a period of time, the ultrasonic probe continues to retreat along the cavity until it returns to the initial position p1.

The pause position p' can be determined based on the input parameters of the input device. For example, after the ultrasound probe completes the advancing process, the operator can see the entire ultrasound image sequence and/or the reconstructed three-dimensional ultrasound image through the display device. For a certain part in the ultrasound image sequence or in the three-dimensional ultrasound image, the operator It may be considered that it is necessary to focus on the observation, the operator can select the image S' or T' of the position of interest by selecting the touch screen, or clicking the mouse, or other similar methods. As mentioned above, ultrasound image sequences are stored in the processor in a manner associated with positions, for example, the storage format is: {p1, S1, T1}, {p2, S2, T2}, {p3, S3, T3} , ... {pn, Sn, Tn}, by searching the stored data of the ultrasound image sequence, the position p' corresponding to the image S' or T' of the position of interest can be determined.

The pause position p' can also be determined on the basis of an automatic analysis of the images marked. For example, for the processing and analysis results of each frame of image data in the ultrasound image sequence, if there is difficulty in contour extraction, unclear display of target tissue, suspicious objects, etc. during image processing, the frame of image is marked as The image S' or T' of the position of interest, and by searching the stored data of the ultrasound image sequence, the position p' corresponding to the image S' or T' of the position of interest can be determined.

Fig. 4 and Fig. 5A-Fig. 5B show a schematic diagram of the principle of performing contour extraction on the slice tomographic image of the prostate according to an embodiment, and determining the end position of the ultrasonic probe movement according to the contour extraction results during automatic transrectal ultrasonography . As shown in Figure 4, the ultrasonic probe is driven by the motion actuator (such as the mechanical arm) to advance along the cavity, from the motion starting position p1 to the motion end position pn, and the tomographic images collected according to the predetermined step length are as follows: S1, S2, S3, ... Sn. The ultrasonic tomographic image of the motion starting position p1 is shown in FIG. 5A . The target tissue at this position has not yet entered the field of view, and a clear image outline cannot be extracted. The ultrasonic tomographic image collected at position p2 may be similar to that in FIG. 5A , and a clear image outline cannot be extracted. During the forward process, the contours of the tomographic images are changed from being unable to extract, to extracting smaller contours, to extracting larger contours (as shown in Figure 5B), and then extracting smaller contours until the contours cannot be extracted, the image sequence S1, S2, S3, ... Sn will experience the above-mentioned change process. According to whether the prostate contour can be accurately positioned in the ultrasound tomographic image, it is judged whether the image is a process position slice image. And, when the extracted contour undergoes the above-mentioned change process until the contour cannot be extracted, it is judged that this position is the end position pn, and a first control instruction for stopping the advancement of the ultrasonic probe is generated. In order to reduce the interference of noise and ensure the stability of the system, according to an embodiment, it is set to continuously detect the segmentation information of the target detection object in the multiple slice images, and then continuously detect the non-target in the multiple slice images When detecting the segmentation information of the object, determine the end position pn, and generate a first control instruction for stopping the advance of the ultrasonic probe.

FIGS. 6-8E show schematic diagrams of extracting the contour of the sagittal ultrasonic image of the prostate and determining the end position of the ultrasound probe movement according to an embodiment according to an embodiment of automatic transrectal ultrasonic examination. As shown in Figure 6, the ultrasonic probe goes through four stages during its advancement: T1 position, the target detection object prostate has not yet entered the sagittal ultrasonic field of view; T2 position, the target detection object enters the sagittal ultrasonic field of view; after that, T… …at the Tn position, the target detection objects all exist in the sagittal plane ultrasound field of view.

Relying on the sagittal image can better obtain all the outline information or all the main structure information of the target detection object, and the overall situation can be seen in real time. As shown in FIG. 6 , the contour of the prostate can be segmented from the T2 position to the Tn position, but only the Tn position can be used as the end position. Therefore, it will become unreliable to judge whether to reach the end position based on whether the contour can be segmented, and therefore, other judgment methods need to be provided.

According to one embodiment, the terminal position judgment based on the sagittal plane image sequence is provided as follows:

Referring to Fig. 7, the rectangular frame under the XY coordinate system in the figure is the sliding window of the ultrasonic imaging area. The ultrasonic probe moves along the negative direction of the X-axis in the figure, that is, the sliding window slides along the negative direction of the X-axis. The shaded part in the figure is The target detection object image in the sagittal ultrasound imaging field of view, the length of the rectangular side of the sliding window along the x direction is defined as the window width L, the window width L can be set according to the accuracy requirements, the higher the accuracy, the smaller the window width L; define The length of the rectangular side of the sliding window along the y direction is the window height H. The window height H only needs to be set to be large enough to cover the entire height of the target detection object, or the window height H is the same as the height of the entire sagittal plane image .

Define the area where the sliding window overlaps with the target detection object image as w, and the w area is used as the judgment part to judge whether the target detection object has fully entered the ultrasonic detection area. During the movement of the ultrasonic probe in the negative direction along the X-axis, the sliding field of view completely detects the target detection object once.

As shown in FIG. 8A , the crosshairs represent the contour area of the prostate in the sagittal plane, and the black area on the left side of the crosshairs region in the figure is the bladder.

As shown in FIG. 8B , the white area is the prostate area segmented by the image algorithm.

Figure 8C-8E is a schematic diagram of the detection process, the ultrasound image is an image covering the entire area around the prostate, assuming that the gray sliding window is the detection area of the linear array ultrasound probe that collects the sagittal plane ultrasound image, it can be judged by the w area The end position of the ultrasound probe movement. In Fig. 8C, the contour of the prostate cannot be segmented in the sliding window, which is near the initial position at this time; it can be seen in Fig. 8D that the window is sliding, and w is 1. As can be seen in Figure 8E, the window slides out and w is 0. Through the above-mentioned manner, the entire process of detecting the starting position and the ending position is completed.

According to an embodiment, the ultrasonic automatic inspection system acquires the ultrasonic tomographic image sequence and the ultrasonic sagittal image sequence at the same time. In the case of poor ultrasonic image information, in order to improve the accuracy of judgment, the judgment of the end position of the ultrasonic probe movement The condition is set as follows: according to the contour extraction results of the ultrasonic tomographic image frame corresponding to each step position and the contour extraction result of the ultrasonic sagittal plane image frame, determine the end point of the ultrasonic probe movement; only when the two judgment conditions are satisfied at the same time, the processing The processor considers that the terminal position has been reached, and sends out the first control instruction accordingly.

According to yet another embodiment, the ultrasonic automatic inspection system simultaneously acquires the ultrasonic tomographic image sequence and the ultrasonic sagittal image sequence. In some cases, for example, if the command to stop the movement is issued too late, it will cause damage to the sensitive area. The judgment condition of the ultrasonic probe movement end position is set as: according to the contour extraction result of the ultrasonic tomographic image frame corresponding to each step position and the contour extraction result of the ultrasonic sagittal plane image frame, determine the ultrasonic probe movement end point; when the two When one of the judging conditions is satisfied, the processor considers that the end position has been reached, and sends out the first control instruction accordingly.

In the present invention, the ultrasonic tomographic image sequence and the ultrasonic sagittal image sequence are acquired simultaneously, and in the determination of the movement end point of the ultrasonic probe, one of the ultrasonic tomographic image sequence and the ultrasonic sagittal image sequence can be used for determination , it is also possible to use both the ultrasonic tomographic image sequence and the ultrasonic sagittal image sequence for determination. Specifically, the processor judges the image quality of each of the ultrasonic tomographic image sequence and the ultrasonic sagittal image sequence, and when the image quality of a single ultrasonic image sequence is sufficient to determine the end point of the ultrasound probe, the ultrasonic tomographic One of the image sequence and the ultrasound sagittal image sequence is used to judge the motion end point of the ultrasound probe. When the image quality of a single ultrasound image sequence is not enough to judge the motion end point of the ultrasound probe, the ultrasound tomographic image sequence and the ultrasound sagittal image sequence The end point of the movement of the ultrasonic probe is judged on both sides of the surface image sequence. The processor can judge the image quality of the ultrasound image sequence by pre-setting the image quality threshold and the like inside the processor.

In the case of using both the ultrasonic tomographic image sequence and the ultrasonic sagittal image sequence to make the judgment, for example, when both the judgment based on the ultrasonic tomographic image sequence and the judgment results based on the ultrasonic sagittal image sequence have reached the end position Next, it is judged that the ultrasonic probe has reached the end point of motion.

On the other hand, either one of the ultrasonic tomographic image sequence and the ultrasonic sagittal image sequence may be affected by intra-image artifacts or the like. For example, the following situation may occur: in the ultrasonic tomographic image, artifacts and other interferences cause poor image quality, and the target detection object cannot be recognized and segmented on the image, while in the ultrasonic sagittal image, nearby Artifacts are less, and it is easier to identify and segment; or, in the ultrasonic sagittal image, the interference such as artifacts seriously causes poor image quality, and the target detection object cannot be recognized and segmented on the image, while in the ultrasonic tomographic image, the space Nearby locations have less artifacts and are easier to identify segmentations.

The ultrasonic tomographic image sequence and the ultrasonic sagittal plane image sequence may also have the following situation: on a certain frame of the ultrasonic tomographic image, there are too many artifacts to be separated, but it can be judged by referring to several adjacent frames of images; , in the ultrasonic sagittal image, when the ultrasonic probe is facing the metal surgical instruments in the human body cavity, a large number of artifacts will be generated due to the received reflected ultrasonic waves. However, the ultrasonic waves emitted by the ultrasonic probe have a certain width. Rotating the ultrasound probe at a certain angle can achieve the effect of seeing surgical instruments and reducing artifacts.

Therefore, by acquiring the ultrasonic tomographic image sequence and the ultrasonic sagittal image sequence at the same time, the two-dimensional image sequence and the corresponding spatial position relationship of the respective probes of the dual-plane probes can be accurately and easily obtained, so the image data of the two planes can be integrated Information and several adjacent image information of an ultrasound image, combined with the method of linear array probe rotation or sector scanning and convex array probe parallel scanning, and the ultrasound image sequence acquired at multiple positions, at the same time with the position and location provided by the ultrasonic stepper Angle information to achieve the effect of improving the accuracy of recognition in ultrasound images.

In addition, the traditional method of obtaining ultrasound spatial position information is 3D reconstruction and segmentation. Generally, only one probe’s 2D ultrasound image sequence is used for 3D reconstruction, and then based on the 3D segmentation network training, the 3D voxel of the division. However, 3D segmentation networks generally consume a large amount of computing resources. When training with a small amount of image data, the risk of overfitting is much higher than that of 2D image segmentation networks. It is more difficult to train and requires more clinical data and annotations. In this regard, in the present invention, a segmentation method based on the fusion of spatial information of multiple probe two-dimensional image sequences and adjacent image spatial information is proposed to achieve the effect of improving the accuracy of segmentation in ultrasonic images.

Specifically, as shown in FIG. 10, in step S11, the movement of the ultrasonic adapter and the acquisition of ultrasonic images are controlled to obtain a two-dimensional ultrasonic image sequence, including the first ultrasonic image sequence (such as an ultrasonic tomographic image sequence) of a biplane ultrasonic probe. and a second ultrasound image sequence (such as an ultrasound sagittal plane image sequence), each sequence includes two-dimensional image data and position information of the ultrasound adapter.

In step S12 , the position information when the two-dimensional image sequence is collected by the ultrasonic adapter, that is, the movement coordinates of the ultrasonic adapter, is used to construct the three-dimensional reconstruction coordinate system of the two-dimensional image sequence based on the moving coordinate system of the ultrasonic adapter.

In step S13, in the three-dimensional reconstruction coordinate system, the multi-plane ultrasound image information is fused based on voxels, that is, the pixels in the first and second ultrasound images are taken from the image coordinate system, and the position information of the ultrasound adapter is used Relationship, transformed into the 3D reconstruction coordinate system.

In step S14, through the gray value interpolation and fusion of the voxels in the three-dimensional reconstruction coordinate system, based on the three-dimensional reconstruction image corresponding to the first ultrasound image sequence and the three-dimensional reconstruction image corresponding to the second ultrasound image sequence, the two sides The 3D reconstructed image is inpainted. That is, the two 3D reconstructed images are repaired by complementing each other. During the interpolation and fusion process, there may be three situations on each voxel in the 3D reconstruction coordinate system:

(1) There is no gray value converted from a two-dimensional image sequence

(2) There is a gray value converted from a two-dimensional image sequence

(3) There are gray values converted from multiple two-dimensional image sequences

For (1), the gray value of the empty voxel can be obtained by interpolating adjacent voxels. For (2), the current gray value remains unchanged. For (3), the same voxel may contain two For multiple gray values of the probe, strategies such as voting and weighted average can be used to determine the final gray value. Thus, the data fusion of multiple ultrasound probes is completed.

Next, in step S15, the reconstructed three-dimensional data is mapped to the original two-dimensional image sequence data. At this time, in the original two-dimensional image (ultrasound tomographic image sequence or ultrasonic sagittal plane image sequence) of a probe Contains spatial image data information of other probes.

It is worth noting that whether to perform mutual patching based on the two 2D images can be judged according to the image quality. When the image quality of a single ultrasound image sequence is sufficiently high, mutual inpainting based on two two-dimensional images may not be performed.

In addition, instead of performing mutual inpainting based on two two-dimensional images, the two-dimensional image may be inpainted by using a two-dimensional image inside a two-dimensional image sequence. Specifically, the inter-frame reference can be performed according to several adjacent image information of an ultrasound image, so as to repair the image, and the intra-frame reference can also be made according to the pixels in an ultrasound image, so as to repair the image.

In the ultrasonic automatic inspection system described in this application, the ultrasonic probe slides and moves in the cavity and completes the inspection. In the process, every time it reaches a position, the ultrasonic automatic inspection system acquires images and simultaneously records the movement position information of the moving mechanism at that position. And the query data is formed by using the sliding position as an index, so that the image information of the moving position can be quickly searched through the index in the future.

In the ultrasonic automatic inspection system and control method provided by the present application, the ultrasonic probe can complete the movement process of the automatic inspection based on the control signal provided by the processor under the control of the motion control device; The ultrasonic image sequence judges the moving position of the ultrasonic probe, and controls the movement of the ultrasonic probe according to the moving position. The system provided in the present application can realize the automation of the transcavity ultrasonic examination process with less configuration.

On the basis of the system and method provided in this application, the movement process of the ultrasonic probe can be controlled in real time, and further, during the movement process of the ultrasonic probe, the pause position can be selected according to the imaging needs, and the accuracy of the surrounding tissue at the pause position can be adjusted. High-quality ultrasonic imaging realizes the automation and fine control of the ultrasonic examination process.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

Therefore, the present application also proposes a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any embodiment of the present application is implemented.

Furthermore, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, and when the processor executes the computer program, it implements any of the embodiments of the present application. the method described.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory. Memory may include non-permanent storage in computer-readable media, in the form of random access memory (RAM) and/or nonvolatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of computer readable media.

It should also be noted that the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes Other elements not expressly listed, or elements inherent in the process, method, commodity, or apparatus are also included. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The above descriptions are only examples of the present application, and are not intended to limit the present application. For those skilled in the art, various modifications and changes may occur in this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

A transcavity ultrasonic automatic inspection system, comprising:

An ultrasonic imaging device, the ultrasonic imaging device includes an ultrasonic probe, the ultrasonic probe can enter the body through a cavity, and generate a two-dimensional first ultrasonic image sequence and a two-dimensional second ultrasonic image sequence;

A motion control device, the motion control device comprising a multi-degree-of-freedom drive mechanism connected to the ultrasonic probe;

a processor, connected to the ultrasonic imaging device and the motion control device, the processor acquires the first sequence of ultrasonic images and the second sequence of ultrasonic images, and performs an operation on the first sequence of ultrasonic images and/or The second ultrasound image sequence is processed to generate a control signal;

The motion control device receives the control signal, drives the ultrasonic probe to move based on the control signal, and completes the automatic inspection process.
The transcavity ultrasonic automatic inspection system according to claim 1, characterized in that,

During the movement of the ultrasonic probe, the ultrasonic imaging device continuously acquires the first ultrasonic image sequence and the second ultrasonic image sequence according to a set step size.
The transcavity ultrasonic automatic inspection system according to claim 2, characterized in that,

The processor extracts image contour information in the first ultrasound image sequence and/or the second ultrasound image sequence for judging whether the ultrasound probe has reached the terminal position,

When it is determined that the ultrasonic probe reaches the end position, the processor generates a first control instruction to control the ultrasonic probe to stop moving.
The transcavity ultrasonic automatic inspection system according to any one of claims 1-3, characterized in that,

The first ultrasonic image sequence is an ultrasonic tomographic image sequence, and the second ultrasonic image sequence is an ultrasonic sagittal image sequence.
The transcavity ultrasonic automatic inspection system according to claim 3, characterized in that,

When the processor judges that the image quality of one of the first ultrasonic image sequence and the second ultrasonic image sequence is sufficient to determine the end point of the ultrasound probe movement, the first ultrasonic image sequence is used and a certain one of the second ultrasound image sequence to judge the movement end point of the ultrasound probe,

When the processor judges that the image quality of any one of the first ultrasound image sequence and the second ultrasound image sequence is not enough to judge the motion end point of the ultrasound probe, use the first ultrasound Both the image sequence and the second ultrasound image sequence are used to determine the motion end point of the ultrasound probe.
The transcavity ultrasonic automatic inspection system according to claim 5, characterized in that,

In the case of using both the first ultrasonic image sequence and the second ultrasonic image sequence to determine the movement end point of the ultrasonic probe, the processor for the first ultrasonic image sequence and the second ultrasonic image sequence Establishing a three-dimensional reconstruction coordinate system, and reconstructing the two-dimensional first ultrasonic image sequence and the two-dimensional second ultrasonic image sequence into the three-dimensional reconstruction coordinate system, thereby obtaining the fusion of the first ultrasonic image sequence a three-dimensionally reconstructed image of the image information and the image information of the second ultrasound image sequence,

And, performing interpolation processing on the 3D reconstructed image, and mapping the interpolated 3D reconstructed image into the two-dimensional first ultrasound image sequence and the two-dimensional second ultrasound image sequence, so as to obtain the repaired The first sequence of ultrasonic images and the second sequence of ultrasonic images after repairing, and using the first sequence of ultrasonic images after repairing and/or the second sequence of ultrasonic images after repairing to judge the movement end point of the ultrasonic probe.
The transcavity ultrasonic automatic inspection system according to any one of claims 1-3, characterized in that,

When the ultrasonic probe reaches the end position, the processor generates a second control command to control the ultrasonic probe to withdraw from the cavity.
The transcavity ultrasonic automatic inspection system according to claim 7, characterized in that,

The processor may also generate a third control instruction, controlling the ultrasonic probe to move to a corresponding position in the cavity and suspending the movement.
The transcavity ultrasonic automatic inspection system according to claim 8, characterized in that,

The processor generates the third control instruction based on an input from an input device, or the processor generates the third control instruction based on a recognition result of the first or second ultrasound image sequence.
The transcavity ultrasonic automatic inspection system according to claim 1, characterized in that,

The ultrasonic imaging device further includes a sheath, and the ultrasonic probe slides in the sheath.
A method for controlling a transcavity ultrasonic automatic inspection system, comprising the following steps:

Acquiring a first sequence of ultrasound images and a sequence of second ultrasound images according to a predetermined step during the movement of the ultrasound probe;

extracting image contour information in the first ultrasound image sequence and/or the second ultrasound image sequence;

According to the extracted image contour information, it is judged whether the ultrasonic probe has reached the terminal position;

When the ultrasonic probe reaches the terminal position, the ultrasonic probe is controlled to stop moving.
The control method of the transcavity ultrasonic automatic inspection system according to claim 11, characterized in that,

After the ultrasonic probe stops moving, it also includes the step of controlling the ultrasonic probe to withdraw from the cavity.
The control method of the transcavity ultrasonic automatic inspection system according to claim 12, characterized in that,

During the process of the ultrasonic probe withdrawing from the cavity, it also includes the step of controlling the movement of the ultrasonic probe to a corresponding position in the cavity and pausing the movement.
A computer-readable storage medium on which a computer program is stored, characterized in that,

When the program is executed by the processor, the method according to any one of claims 11-13 is realized.
An electronic device comprising a memory, a processor, and a computer program stored on the memory and operable by the processor, wherein the processor implements any one of claims 11-13 when executing the computer program the method described.