KR102233585B1 - Image registration apparatus and method using multiple candidate points - Google Patents

Abstract

Disclosed is an image processing apparatus that generates output image data representing at least a portion of an object by using a reference point sensed on an object. The image processing apparatus acquires a first fiducial points set, which defines a plurality of points sensed on an object on a first coordinate system, and image data corresponding to the object is defined in a second coordinate system. , A plurality of candidate correspondence points are obtained based on a second reference point mapped to any one of a plurality of first reference points indicated by the first reference point set, and on the object based on the first reference point set and the plurality of candidate correspondence points. Determines a matching parameter that converts position information on the first coordinate system of the sensed point into position information on the second coordinate system, and indicates the relative positions between the point sensed through the sensor and the object based on the determined matching parameter. And a processor that generates output image data.

Description

Image registration apparatus and method using a plurality of candidate correspondence points {IMAGE REGISTRATION APPARATUS AND METHOD USING MULTIPLE CANDIDATE POINTS}

The present disclosure relates to an image matching apparatus and method, and more particularly, to an apparatus and method for matching image data using a plurality of candidate correspondence points corresponding to one reference point.

Minimally invasive surgery, which minimizes the patient's incision during surgery, is widely used. Minimally invasive surgery has the advantage of minimizing the incision site and minimizing blood loss and recovery period accordingly, but some surgeries have risk factors such as meningeal damage and eye damage due to the limited vision of the doctor. . As one of the tools for overcoming the shortcomings of minimally invasive surgery in which a doctor's field of view is limited, a medical navigation device (or a surgical navigation device) has been used. The medical navigation device provides real-time tracking of location information of an instrument inserted in a patient's affected part based on a previously secured medical image of a patient. In addition, such a medical navigation device may be used in combination with an endoscope.

An optical or electromagnetic positioning device may be used for real-time positioning of a surgical instrument inserted in a medical navigation device. As an example for tracking the location of the surgical instrument, an optical location tracking device including an infrared emission device and a passive image sensor may be used. The optical position tracking device emits reference light through an infrared emission device, and collects images reflected by a plurality of markers through an image sensor. The corresponding location tracking device may acquire location information of a procedure device based on the locations of a plurality of markers. Meanwhile, as another example for tracking the location of the surgical instrument, an electromagnetic location tracking device including a magnetic field generator and a conductive metal object may be used. The electromagnetic position tracking device may obtain positional information of a surgical instrument by measuring an eddy current generated in a conductive metal object in a magnetic field generated by the magnetic field generator. In order to accurately display the positional relationship between the procedure device and the body part in the position tracking device, a registration process may be required to define the positional relationship between the medical data on the patient's body part and the procedure device.

1 shows an embodiment of an output image of a medical navigation device. The medical navigation device may display at least one of images of a horizontal plane, a sagittal plane, and a coronal plane for a body part. The operator (or doctor) interprets each image to determine the three-dimensional position of the surgical instrument, and identifies adjacent risk factors. However, these cross-sectional images cannot intuitively express the position of the surgical instrument within the body part. Accordingly, in order to determine the exact position of the surgical instrument by comparing a plurality of cross-sectional images, not only the operator's skill is required, but it may take a lot of time. In addition, if the time to watch the monitor of the medical navigation device is prolonged to determine the location of the procedure device, the overall treatment period may be prolonged, thereby increasing the fatigue of both the operator and the patient.

Unexamined Patent Publication No. 10-2014-0105101 Japanese Patent Publication No. 583157 Japanese Unexamined Patent Application Publication No. 2018-038609 Japanese Patent Publication No. 5461862

An image processing apparatus and method according to an embodiment of the present disclosure aims to provide output image data corresponding to an object. Specifically, an image processing apparatus and method according to an embodiment of the present disclosure aims to provide output image data matched based on image data corresponding to an object. In addition, an image processing apparatus and method according to an embodiment of the present disclosure aims to improve a matching error by reducing an error generated when specifying a reference point.

According to an embodiment of the present disclosure for solving the above problems, a first set of fiducial points, defining a plurality of points sensed on an object on a first coordinate system, is obtained, and the The image data corresponding to the object is defined in a second coordinate system, and a plurality of candidate correspondence points are obtained based on a second reference point mapped to any one of a plurality of first reference points indicated by the first reference point set, and the Based on a first set of reference points and the plurality of candidate corresponding points, a matching parameter for converting position information on the first coordinate system of a point sensed on the object to position information on the second coordinate system is determined, and the determined matching parameter is An image processing apparatus including a processor that generates output image data indicating relative positions between a point sensed through a sensor and the object as a basis may be provided.

In addition, the method according to an embodiment of the present disclosure includes: defining a plurality of points sensed on an object on a first coordinate system, obtaining a first set of reference points, and image data corresponding to the object is defined in a second coordinate system And obtaining a plurality of candidate correspondence points based on a second reference point mapped to any one of a plurality of first reference points indicated by the first reference point set, based on the first reference point set and the plurality of candidate correspondence points, Determining a matching parameter for converting position information on the first coordinate system of a point sensed on the object to position information on the second coordinate system, and between the point sensed through a sensor and the object based on the determined matching parameter It may include generating output image data indicating a relative position.

A computer-readable recording medium according to another aspect may include a recording medium recording a program for executing the above-described method on a computer.

According to the image processing apparatus and method according to an exemplary embodiment of the present disclosure, an error occurring when a reference point is designated may be reduced. In addition, the image processing apparatus and image processing method according to an exemplary embodiment of the present disclosure may provide output image data accurately indicating a position of a point sensed as a position of a medical device inserted into an object.

1 shows an embodiment of an output image of a medical navigation device.
2 is a schematic diagram illustrating an operation of an image processing apparatus according to an exemplary embodiment of the present disclosure.
3 is a diagram illustrating a reference point designation error and a reference point matching error according to an embodiment of the present disclosure.
4 is a flowchart illustrating a method of operating an image processing apparatus according to an exemplary embodiment of the present disclosure.
5 is a diagram illustrating an example of output image data generated according to an embodiment of the present disclosure.
6 is a diagram illustrating a plurality of candidate correspondence points according to an embodiment of the present disclosure.
7 is a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

The terms used in the present specification have been selected as currently widely used general terms as possible while taking functions in the present disclosure into consideration, but this may vary according to the intention, custom, or the emergence of new technologies of technicians engaged in the art. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in the corresponding description. Therefore, it should be noted that the terms used in the present specification should be interpreted based on the actual meaning of the term and the entire contents of the present specification, not a simple name of the term.

Throughout the specification, when a component is said to be “connected” with another component, this includes not only the case that it is “directly connected”, but also the case that it is “electrically connected” with another component in the middle. do. In addition, when a certain component "includes" a specific component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary. In addition, the limitations of "more" or "less than" based on a specific threshold value may be appropriately replaced with "excess" or "less than", respectively, depending on the embodiment.

Hereinafter, an image processing apparatus and an image processing method according to an exemplary embodiment of the present disclosure will be described with reference to each drawing. The image processing apparatus and image processing method according to an exemplary embodiment of the present disclosure may be applied to image data corresponding to an object including a human body and an animal body. Image data includes X-ray images, computed tomography (CT) images, positron emission tomography (PET) images, ultrasound images, and magnetic resonance imaging (MRI). The disclosure is not limited thereto. For example, image data corresponding to an object may mean image data obtained by photographing a corresponding object using the various photographing methods. In addition, in the present specification, image data is used in a broad sense including not only the image itself, but also various types of data generated by rendering the image. According to an embodiment, the image data may refer to data obtained by volume-rendering an image. Also, the image data may refer to a 3D data set composed of a group of 2D medical images. The value of the regular grid unit of the 3D data set constructed as described above is called a voxel.

The image processing apparatus and image processing method according to an exemplary embodiment of the present disclosure may provide output image data indicating a relative position between a point sensed through a sensor and an object. For example, the image processing apparatus may display the position of a medical device inserted in the object on image data of the object obtained in advance. Through this, the operator can grasp the position of the medical device in the object by using the position of the medical device displayed on the output image and identify the adjacent risk factors in the procedure.

As described above, when a position of a medical device is to be displayed on image data corresponding to an object, the image processing apparatus may need to convert position information of a point sensed as the position of the medical device on the object into position information on the image data. . This is because the location information of the point sensed through the sensor can be defined through a coordinate system different from the coordinate system representing the image data. In this case, the image processing apparatus may register location information of a point defined in the first coordinate system with location information defined in the second coordinate system. Here, registration may mean aligning with one coordinate system by using a relationship between position information defined on different coordinate systems. The first coordinate system may be a coordinate system indicating location information of a point sensed on the object. Also, the second coordinate system may be a coordinate system indicating a point on image data corresponding to an object. For example, the image processing apparatus may match position information on a first coordinate system indicating a position of a point sensed as a position of a medical device on an object with position information on a second coordinate system, and the position of the medical device on the image data corresponding to the object. Can be displayed.

Hereinafter, a method of matching a first coordinate system indicating a position of a point sensed through a sensor and a second coordinate system indicating image data corresponding to an object by the image processing apparatus according to an embodiment of the present disclosure will be described. This will be described with reference to FIG. 2. 2 is a schematic diagram showing an operation of the image processing apparatus 100 according to an exemplary embodiment of the present disclosure.

According to an embodiment, the image processing apparatus 100 may register the first coordinate system and the second coordinate system using point-based registration. Here, the point-based matching may mean matching different coordinate systems using fiducial points sets that are respectively defined in different coordinate systems.

For example, the image processing apparatus 100 may match the first coordinate system and the second coordinate system using a first set of reference points sensed on the object and a second set of reference points designated on image data corresponding to the object. In this case, a plurality of reference points included in the first reference point set and a plurality of reference points included in the second reference point set may have a one-to-one mapping relationship with each other. In addition, a fiducial point may mean a point used to match different coordinate systems through point-based registration.

In the embodiment of FIG. 2, the image processing apparatus 100 may acquire a first set of reference points including a plurality of reference points 21, 22, and 23 sensed on the object 300. Here, the first reference point set may be referred to as a scene point set. Also, the scene point set may include a patient landmark. Specifically, the first reference point 21, the second reference point 22, and the third reference point 23 included in the first reference point set may be 3D coordinates on the first coordinate system. The first coordinate system may be a coordinate system indicating position information of a point sensed on the object. In addition, the first reference point 21, the second reference point 22, and the third reference point 23 may represent location information of each point sensed on the object 300 by the location tracking device 200. 2, the image processing apparatus 100 may receive a first reference point 21, a second reference point 22, and a third reference point 23 from the location tracking device 200. For example, the image processing apparatus 100 may include a first reference point 21, a second reference point 22, and a third reference point ( 23) can be obtained.

Also, the image processing apparatus 100 may obtain a second set of reference points including a plurality of reference points 11, 12, and 13 designated on the image data 400 corresponding to the object. Here, the second reference point set may be referred to as a model point set. Additionally, the set of model points may include image landmarks. Specifically, the fourth reference point 11, the fifth reference point 12, and the sixth reference point 13 included in the second reference point set may be 3D coordinates on the second coordinate system. The second coordinate system may be a coordinate system representing image data corresponding to an object. Also, the image processing apparatus 100 may acquire the fourth reference point 11, the fifth reference point 12, and the sixth reference point 13 based on a user input. Specifically, the image processing apparatus 100 may acquire a designated second reference point set from 3D image data such as CT using visualization software.

Next, the image processing apparatus 100 may obtain a registration parameter based on the first reference point set and the second reference point set. For example, the matching parameter may be a first parameter that converts position information on a first coordinate system of a point sensed through a sensor into position information on a second coordinate system. Alternatively, the matching parameter may be a second parameter for converting position information on the second coordinate system into position information on the first coordinate system. In this case, the first parameter and the second parameter may have an inverse relationship. In the following of the present disclosure, the matching parameter may be described on the assumption that it is a first parameter for convenience of description, but is not limited thereto.

In the embodiment of FIG. 2, the first reference point 21 is at the fourth reference point 11, the second reference point 22 is at the fifth reference point 12, and the third reference point 23 is at the sixth reference point 13. Each is mapped to. In the present disclosure, a pair of reference points in a mapping relationship may be referred to as paired fiducial points. The image processing apparatus 100 may obtain a matching parameter based on a positional relationship between a plurality of paired reference points. The image processing apparatus 100 may convert the first reference point 21, the second reference point 22, and the third reference point 23 on the first coordinate system into a point on the second coordinate system. For example, the image processing apparatus 100 may rotate and translate a first reference point set on the first coordinate system into a point on the second coordinate system by rotating and moving it in parallel.

In addition, the image processing apparatus 100 includes the converted first reference point 21, the second reference point 22, and the fourth reference point 11, the fifth reference point 12, which are in a mapping relationship with the third reference point 23. An error between the sixth reference points 13 can be calculated. In this case, the error between the paired reference points after conversion may be referred to as a fiducial registration error (FRE) or a matching error. The image processing apparatus 100 may obtain a matching parameter that optimizes a matching error. Specifically, the image processing apparatus 100 includes the converted distance between the first reference point 21 and the fourth reference point 11, the distance between the second reference point 22 and the fifth reference point 12, and a third reference point ( The matching error may be determined based on the distance between 23) and the sixth reference point 13. Also, the image processing apparatus 100 may determine whether to update the matching parameter based on the matching error. For example, when the reference point matching error converges, the image processing apparatus 100 may not update the matching parameter. In this case, the image processing apparatus 100 may generate the above-described output image data based on the determined matching parameter. Meanwhile, when the reference point matching error does not converge, the image processing apparatus 100 may update the matching parameter.

The image processing apparatus 100 may use various optimization algorithms for optimizing a reference point matching error in order to obtain a matching parameter. Specifically, the image processing apparatus 100 may use an iterative closest points (ICP) method. The ICP method is an optimization algorithm that obtains a matching parameter that minimizes the sum of the distances between the paired reference points. The image processing apparatus 100 may obtain the matching parameter through an ICP method of updating the matching parameter until the reference point matching error becomes less than or equal to a preset threshold.

Meanwhile, in the embodiment of FIG. 2, it has been described that the image processing apparatus 100 matches the first coordinate system and the second coordinate system based on the paired three pairs of reference points, but the present disclosure is not limited thereto. For example, the image processing apparatus 100 may match the first coordinate system and the second coordinate system based on the six pairs of reference points shown in FIG. 2.

As described above, the image processing apparatus 100 may receive the first reference point set from the location tracking apparatus 200. In this case, the location tracking device 200 may include a probe that senses a specific location on the object 300. The location tracking device 200 may generate location information indicating a location of a probe that is in contact with a specific point on the object 300. Also, the location tracking device 200 may transmit the generated location information to the image processing device 100. The location tracking device 200 may transmit location information generated through a wired/wireless interface connected to the image processing device 100 to the image processing device 100.

The location tracking apparatus 200 according to an embodiment of the present disclosure may include a medical device inserted into the object 300. The location tracking device 200 may include various types of treatment functions. The location tracking device 200 may generate location information of a sensed point using an optical location tracking method or an electromagnetic location tracking method, but the present invention is not limited thereto. Specifically, the location tracking device 200 may include a magnetic tracking system. In this case, the location tracking device 200 may generate location information indicating the location of one contact point in a coordinate system defined in the magnetic tracking system. In addition, the location tracking apparatus 200 may include a magnetic field generator that generates a magnetic field and a probe that contacts the object. The location tracking device 200 may generate location information of a point in contact with the object 300 by measuring an eddy current generated by a probe contacting the object 300 within a magnetic field generated by the magnetic field generator.

Meanwhile, in FIG. 2, it has been described that the location tracking device 200 is a device separate from the image processing device 100, but is not limited thereto. For example, the image processing apparatus 100 may be equipped with a location tracking function. In more detail, the image processing apparatus 100 may acquire a first set of reference points sensed on the object by using a location tracking function.

As described above, when a position of a medical device inserted into an object is to be displayed on image data corresponding to the object, the image processing apparatus 100 may use point-based registration. In this case, the position of the medical device may be displayed differently from the actual one due to an error occurring while the user designates a reference point used for registration. An error that occurs while designating a reference point used for matching may be referred to as a fiducial localization error (FLE).

3 is a diagram illustrating a reference point designation error and a reference point matching error according to an embodiment of the present disclosure. Referring to block 31 of FIG. 3, when the image processing apparatus 100 acquires a reference point set based on a user input, the reference point set (circle indicated by a solid line) designated by the user is actually a reference point set (dotted line) It may be different from the marked circle). For example, the second reference point designated by the user on image data corresponding to the object may not be a point coincident with the first reference point sensed on the object based on the object. Conversely, when two reference points on the image data are previously designated, the first reference point sensed on the object may not be a point coincident with the second reference point. Due to this reference point designation error, a matching error may occur as in block 32 of FIG. 3.

Block 32 of FIG. 3 represents the reference point matching error FRE described above. 'X' of block 32 represents a transformed point in which the reference point on the first coordinate system is converted into a point on the second coordinate system. For example, when the image processing apparatus 100 acquires the above-described first reference point set and/or the second reference point set based on a user input, the image processing apparatus 100 is a matching parameter reflecting an error occurring when designating a reference point. Can be obtained. If the position of the medical device on the image data is displayed differently from the position in the actual object, a procedure error may occur or a procedure time may be delayed.

According to an embodiment of the present disclosure, the image processing apparatus 100 may match the first coordinate system and the second coordinate system using a plurality of candidate correspondence points mapped to one reference point on the first coordinate system sensed on the object. In this case, the plurality of candidate correspondence points may be reference points defined in the second coordinate system. In addition, the plurality of candidate correspondence points may further include at least one correspondence point adjacent to the reference point in addition to a reference point designated based on a user input for image data. That is, each reference point sensed on the object may be mapped to at least two or more candidate correspondence points. Through this, the image processing apparatus 100 may reduce a reference point designation error.

Hereinafter, a method of obtaining a matching parameter by using a plurality of candidate correspondence points by the image processing apparatus 100 according to an embodiment of the present disclosure will be described with reference to FIG. 4. 4 is a flowchart illustrating a method of operating the image processing apparatus 100 according to an exemplary embodiment of the present disclosure. 4 illustrates the operation of the image processing apparatus 100 divided by stages, but the present disclosure is not limited thereto. For example, each step-by-step operation of the image processing apparatus 100 disclosed in FIG. 4 may overlap or be performed in parallel. In addition, the image processing apparatus 100 may perform each step-by-step operation in an order different from the order disclosed in FIG. 4.

In step S402, the image processing apparatus 100 may obtain a first set of reference points defining a plurality of points sensed on the object on the first coordinate system. Here, the first coordinate system may be a coordinate system indicating the position of a point sensed on the object as described with reference to FIG. 2. That is, the first coordinate system may be a coordinate system indicating a position of a point sensed in a physical space. Specifically, the image processing apparatus 100 may receive a first set of reference points from the location tracking apparatus 200 described in FIG. 2.

In step S404, the image processing apparatus 100 may acquire a plurality of candidate correspondence points on the second coordinate system. Here, the second coordinate system may be a coordinate system representing image data corresponding to an object as described with reference to FIG. 2. According to an embodiment, the image processing apparatus 100 may acquire a plurality of candidate correspondence points based on a second reference point mapped to any one of a plurality of first reference points indicated by the first reference point set. The second reference point may be a reference point on the second coordinate system. In this case, the plurality of candidate correspondence points may include one second reference point and at least one correspondence point adjacent to the one second reference point. Specifically, the corresponding point adjacent to each of the plurality of second reference points may be at least one coordinate included in a preset area from the second reference point among coordinates located on the surface of the object in image data corresponding to the object. Corresponding points adjacent to each of the plurality of second reference points may be coordinates located on the second reference point and the iso-surface. For example, when any one of the plurality of second reference points represents the eyetail point of the patient, the image processing apparatus 100 may at least one of the coordinates located on the skin surface of the patient within a predetermined distance from the eyetail point. May be the coordinates of. For example, the image processing apparatus 100 may acquire a plurality of second reference points having a one-to-one mapping relationship with each of the plurality of first reference points. In this case, the image processing apparatus 100 may obtain at least one corresponding point for each of the plurality of second reference points for each of the plurality of second reference points. Also, the plurality of second reference points may be points previously designated on the image data. Alternatively, the plurality of second reference points may be points designated based on a user input for image data.

According to an embodiment, the image processing apparatus 100 may acquire a plurality of second reference points based on a user input. For example, the image processing apparatus 100 may acquire a plurality of second reference points based on a user input based on image data. The image processing apparatus 100 may acquire a plurality of points designated on the image data based on a user input. In this case, the image processing apparatus 100 may map each of the plurality of points to each of the plurality of first reference points. The image processing apparatus 100 may index a plurality of first reference points and provide index information allocated for each of the plurality of first reference points to a user. In this case, the image processing apparatus 100 may obtain a plurality of second reference points by mapping each of the plurality of points designated on the image data to each of the plurality of first reference points based on the index information and the user input. Specifically, the index may be an integer set based on the number of preset reference points. Alternatively, the index may be text indicating a feature corresponding to a corresponding point on the object. For example, the image processing apparatus 100 may use indexes such as'eyeLeftOuter','eyeLeftInner','NoseTip', and'mouthRight' to map the reference point.

Also, the image processing apparatus 100 may acquire a plurality of candidate correspondence points based on each of the plurality of second reference points. For example, one first reference point may be mapped with one second reference point and at least one corresponding point located within a preset distance based on the second reference point. According to an embodiment, when coordinates located at the right edge of the object in the image data are designated as the second reference point, the image processing apparatus 100 may acquire K surrounding 3D coordinates as candidate corresponding points. For example, when the first reference point set includes N first reference points, the image processing apparatus 100 may obtain K candidate corresponding points each mapped to the N first reference points on the image data. That is, the image processing apparatus 100 may match the first coordinate system and the second coordinate system using a total of KxN candidate correspondence points defined in the second coordinate system. For example, when the number of candidate correspondence points is 4 and the number of reference points sensed on the object is 4, the total number of candidate correspondence points may be 16. In this case, the number of candidate correspondence points may be a value set based on a user input or a default value. A method of obtaining a corresponding point adjacent to the second reference point by the image processing apparatus 100 will be described in detail with reference to FIG. 6.

Through the above-described method, the image processing apparatus 100 may use additional 3D coordinates as a reference point for matching a coordinate system in addition to a reference point designated based on a user's input. Through this, the image processing apparatus 100 may reduce an error that occurs when designating a reference point.

In step S406, the image processing apparatus 100 may determine a matching parameter based on the first reference point set and the candidate corresponding point. For example, the image processing apparatus 100 may determine an initial matching parameter based on a plurality of paired reference points. In this case, the paired reference point may represent a pair of first and second reference points in a mapping relationship among the plurality of first reference points and the plurality of second reference points. The image processing apparatus 100 may convert a plurality of first reference points into a plurality of transformed points on a second coordinate system based on the initial matching parameter.

Also, the image processing apparatus 100 may determine whether to update a matching parameter based on a matching error between a plurality of transform points and a plurality of candidate correspondence points. Here, the matching error may be determined based on a result of calculating a distance between one transform point and a plurality of candidate correspondence points mapped to one transform point for each of the plurality of transform points. In this case, a mapping relationship between one transform point and a plurality of candidate correspondence points may be determined based on a mapping relationship between the first reference point on the first coordinate system and the second reference point on the second coordinate system. For example, a first conversion point is converted from one specific first reference point among a plurality of first reference points, and a first candidate correspondence point and a second candidate correspondence point are mapped to one specific first reference point. When obtained, the first candidate correspondence point and the second candidate correspondence point may be mapped to the first transformation point.

Specifically, when there are two candidate correspondence points mapped to each transformation point, the image processing apparatus 100 includes a distance between the first candidate correspondence point mapped to the first transformation point and the first transformation point, and the second candidate mapped to the first transformation point. The matching error may be determined based on the distance between the corresponding point and the first transform point. In addition, the image processing apparatus 100 calculates a matching error based on the distance between the third candidate correspondence point and the second transformation point mapped to the second transformation point, and the distance between the fourth candidate correspondence point and the second transformation point mapped to the second transformation point. You can decide.

For example, the image processing apparatus 100 may obtain a first candidate correspondence point having a closer distance to the first transform point from among the first candidate correspondence point and the second candidate correspondence point mapped to the first transform point. Also, the image processing apparatus 100 may obtain a fourth candidate correspondence point having a closer distance to the second transform point from among the third candidate correspondence point and the fourth candidate correspondence point mapped to the second transform point. The image processing apparatus 100 may determine a matching error based on a distance between the first transform point and the first candidate correspondence point and the distance between the second transform point and the fourth candidate correspondence point. In this way, the image processing apparatus 100 may obtain a set of candidate correspondence points for minimizing the matching error. In this case, the candidate correspondence point set may include a candidate correspondence point selected from among a plurality of candidate correspondence points mapped for each transformation point.

According to an embodiment, when the matching error converges, the image processing apparatus 100 may not update the matching parameter. In this case, the image processing apparatus 100 may generate output image data based on a matching parameter in a corresponding round. Specifically, the image processing apparatus 100 may compare a difference between a first matching error in a previous loop and a second matching error in a current loop with a preset threshold. When the difference between the first matching error and the second matching error is smaller than a preset threshold, the image processing apparatus 100 may determine that the matching error converges. On the other hand, when the matching error does not converge, the image processing apparatus 100 may update the matching parameter. The image processing apparatus 100 may update a plurality of first reference points as transform points. In addition, the image processing apparatus 100 may repeat the above-described operation until the matching error converges.

According to an embodiment, the matching parameter is a rotation matrix (hereinafter referred to as'R') for rotating and moving position information on the first coordinate system and a translation matrix (hereinafter referred to as'R') for moving position information on the first coordinate system in parallel. T') may be included. The image processing apparatus 100 may determine a rotation matrix and a translation matrix based on the first set of reference points and a plurality of candidate corresponding points.

In an embodiment, the number of first reference points may be N, and the number of candidate correspondence points mapped to any one of the plurality of first reference points may be K. In this case, the image processing apparatus 100 may convert the plurality of first reference points into N transform points defined in the second coordinate system based on the rotation matrix and the translation matrix. Also, the image processing apparatus 100 may calculate a matching error based on a distance between each of the plurality of transformation points and the candidate corresponding points mapped to each of the plurality of transformation points. For example, for each of the N transform points, the image processing apparatus 100 may calculate a distance between any one of the N transform points and K candidate correspondence points mapped to any one of the N transform points. The image processing apparatus 100 may calculate a matching error between N transform points and a plurality of (N x K) candidate correspondence points based on the calculation result.

Also, the image processing apparatus 100 may update the rotation matrix and the translation matrix based on the calculated matching error. In this case, the image processing apparatus 100 may update the rotation matrix and the translation matrix in the manner described above in connection with the update of the matching parameter.

Specifically, the image processing apparatus 100 may determine a rotation matrix and a translation matrix based on a matching error obtained based on Equation 1.

는 복수의 제1 기준점 중에서 i번째 제1 기준점을 나타내고,

는 i번째 제1 기준점에 매핑되는 복수의 후보 대응점 중에서 k번째 후보 대응점을 나타낼 수 있다. 또한, 수학식 1에서 min(x)는 함수 'x'의 최소값을 나타낼 수 있다. 또한,

는 상기 i번째 제1 기준점에 로테이션 매트릭스를 적용하여 획득한 결과 값을 나타내고,

는 트랜스레이션 매트릭스를 나타낼 수 있다.In Equation 1, N denotes the number of a plurality of first reference points, K denotes the number of a plurality of candidate correspondence points mapped to any one of the N first reference points,

Represents the i-th first reference point among a plurality of first reference points,

May represent a k-th candidate correspondence point among a plurality of candidate correspondence points mapped to the i-th first reference point. In addition, in Equation 1, min(x) may represent the minimum value of the function'x'. Also,

Represents a result value obtained by applying a rotation matrix to the i-th first reference point,

May represent a translation matrix.

를 결정할 수 있다. 또한, 영상 처리 장치(100)는 결정된

및 수학식 1을 기초로 정합 오차를 획득할 수 있다. 또한, 영상 처리 장치(100)는 전술한 방법으로 정합 오차가 수렴할 때 까지

, 로테이션 매트릭스, 및 트랜스레이션 매트릭스를 갱신할 수 있다. 또한, 영상 처리 장치(100)는 정합 오차가 수렴하는 경우, 해당 라운드에서의

, 로테이션 매트릭스 및 트랜스레이션 매트릭스를 기초로 출력 영상 데이터를 생성할 수 있다.단계 S408에서, 영상 처리 장치(100)는 정합 파라미터를 기초로 출력 영상 데이터 생성할 수 있다. 예를 들어, 영상 처리 장치(100)는 정합 파라미터를 기초로 센서를 통해 센싱된 지점과 대상체 사이의 상대적인 위치(relative positions)를 나타내는 출력 영상 데이터를 생성할 수 있다. 이때, 영상 처리 장치(100)는 센서를 통해 센싱된 지점을 제1 좌표계 상에 정의하는 좌표를 획득할 수 있다. 또한, 영상 처리 장치(100)는 단계 S406에서 획득한 정합 파라미터를 기초로 센서를 통해 센싱된 지점과 대상체 사이의 상대적인 위치를 영상 데이터 상에 표시할 수 있다. 예를 들어, 영상 처리 장치(100)는 위치 추적 장치(200)에서 센싱된 좌표를 영상 데이터 상의 제2 좌표계 상의 점으로 표시할 수 있다.Specifically, the image processing apparatus 100 has the closest distance to the transformed point converted from the i-th first reference point among K candidate correspondence points mapped to the i-th reference point in Equation 1

Can be determined. In addition, the image processing device 100 is

And a matching error may be obtained based on Equation 1. In addition, the image processing apparatus 100 is used until the matching error converges in the above-described method.

, Rotation matrix, and translation matrix can be updated. In addition, when the matching error converges, the image processing apparatus 100

The output image data may be generated based on the rotation matrix and the translation matrix. In step S408, the image processing apparatus 100 may generate the output image data based on the matching parameter. For example, the image processing apparatus 100 may generate output image data indicating relative positions between a point sensed through a sensor and an object based on the matching parameter. In this case, the image processing apparatus 100 may acquire coordinates that define a point sensed through the sensor on the first coordinate system. Also, the image processing apparatus 100 may display a relative position between a point sensed through a sensor and an object on the image data based on the matching parameter acquired in step S406. For example, the image processing apparatus 100 may display coordinates sensed by the location tracking apparatus 200 as points on the second coordinate system on the image data.

According to an embodiment, the image processing apparatus 100 may apply the matching result to display the position of the medical device inserted into the object on the image data. The image processing apparatus 100 may acquire location information of a point sensed as a location of a medical device inserted into the object within the object. Also, the image processing apparatus 100 may generate output image data including a marker indicating the location of a medical device inserted into the object based on the location information and the matching parameter.

Specifically, as shown in FIG. 5, the image processing apparatus 100 may display the position of a medical device inserted in the object on a patient image (such as CT) previously acquired in an image guided surgery environment. For example, the image processing apparatus 100 may display a location of a medical device inserted in an object as a figure having specific properties. In this case, the specific attribute may mean a method of indicating the location of the medical device, such as the size, color, and pattern of a figure.

Hereinafter, a method of obtaining a candidate correspondence point by the image processing apparatus 100 according to an embodiment of the present disclosure will be described with reference to FIG. 6.

According to an embodiment, the image processing apparatus 100 may acquire a plurality of candidate correspondence points based on information on the number of candidate correspondence points indicating the number of candidate correspondence points. Here, the information on the number of candidate correspondence points may indicate the number of candidate correspondence points corresponding to one reference point sensed on the object. For example, the image processing apparatus 100 may determine the number of candidate correspondence points based on information related to image data corresponding to an object. In this case, the information related to the image data may include at least one of a resolution of the image data, an image size of the image data, and a compression method of the image data, but is not limited thereto. For example, the image processing apparatus 100 may determine the number of candidate correspondence points based on the resolution of the image data. This is because the lower the resolution of the image data, the larger the reference point designation error may be.

Also, the image processing apparatus 100 may determine the number of candidate correspondence points based on the matching processing performance of the image processing apparatus 100. This is because, as the number of candidate correspondence points increases, the amount of computation for matching processing may increase. Here, the matching processing performance of the image processing apparatus 100 may include a processing speed of a processor included in the image processing apparatus 100. This is because the resources that can be allocated to an operation for matching processing may be limited according to the processing speed of the processor. In addition, the matching processing performance may vary depending on the matching algorithm used by the processor. In addition, the matching processing performance of the image processing apparatus 100 may include a memory included in the image processing apparatus 100 or a computing power of a GPU. According to an embodiment, the image processing apparatus 100 may determine the number of all candidate correspondence points used for matching based on the computing power of a processor included in the image processing apparatus 100. In this case, the image processing apparatus 100 may determine the number of candidate correspondence points that can be allocated to one paired reference point based on the number of paired reference points. As described above, the number of paired reference points may be determined according to the resolution of image data corresponding to the object.

6 is a diagram illustrating a plurality of candidate correspondence points according to an embodiment of the present disclosure. In FIG. 6,'x' denotes a transform point, and'o' mapped to each transform point denotes a plurality of candidate correspondence points. 6 shows a relationship between each transform point and a plurality of candidate correspondence points on a second coordinate system 60 representing image data corresponding to an object. 6 illustrates a case where the number of reference points on the first coordinate system sensed on the object is '3' and the number of candidate correspondence points is '5', and the present disclosure is not limited thereto. In addition, in FIG. 6, each of the dots may be divided into a first group 61, a second group 62, and a third group 63.

In the embodiment of FIG. 6, the image processing apparatus 100 may acquire a plurality of candidate correspondence points mapped to each of a plurality of reference points on the first coordinate system based on information on the number of candidate correspondence points. In each of the first group 61, the second group 62, and the third group 63, an'o' indicated by a solid line is a reference point designated by a user on the second coordinate system 60 or pre-designated based on the feature points of the object. Can be Hereinafter, for convenience of description,'o' indicated by a solid line in each of the first group 61, the second group 62, and the third group 63 is designated as the first designated point 601 and the second designated point 602. And a third designated point 603.

According to an embodiment, the image processing apparatus 100 may acquire a plurality of candidate corresponding points based on a preset area from the location of the specified point in the second coordinate system 60. For example, in FIG. 6, the first designated point 601 may be a point designated by the user on the second coordinate system 60. In this case, the image processing apparatus 100 may acquire four corresponding points in the first group 61 located within a preset area based on the first designated point 601. Also, the image processing apparatus 100 may acquire five candidate correspondence points including the first designated point 601 and four corresponding points.

According to an embodiment, the preset area may be a circle or a sphere having a radius of a preset distance based on one designated point as shown in FIG. 6. Specifically, the preset area may be an area within a preset distance from one designated point based on at least one of a horizontal plane, a sagittal plane, and a coronal plane of the image data. The image processing apparatus 100 may acquire a plurality of candidate corresponding points based on an area within a preset distance based on one designated point.

According to an embodiment, the preset distance may be determined based on a distance between at least two designated points among a plurality of designated points. For example, the preset distance may be determined based on at least a distance between the first and second designated points 601 and 602. The image processing apparatus 100 may acquire a plurality of candidate correspondence points located in an area within a distance shorter than the distance between the first and second designated points 601 and 602 based on the first designated point 601. have. This is because, when designating a reference point, errors in designating a reference point may occur within a distance shorter than the distance between the designated points. Also, the preset distance may be determined based on information related to image data corresponding to the object. For example, the preset distance may be a distance determined based on an image size of image data. This is because the range of the reference point designation error may vary depending on the image size of the image data.

According to an exemplary embodiment, the image processing apparatus 100 may acquire a plurality of corresponding points positioned at a predetermined distance based on one designated point. For example, the image processing apparatus 100 may acquire a plurality of corresponding points located at the same predetermined distance from one designated point based on each of the horizontal plane, the sagittal plane, and the coronal plane of the image data. Also, the image processing apparatus 100 may acquire a plurality of candidate corresponding points based on a minimum unit distance that can be expressed in the second coordinate system 60 based on one designated point. For example, in FIG. 6, four points adjacent to the first designated point 601 may represent four coordinates located at a minimum unit distance from the first designated point 601. In addition, the plurality of candidate correspondence points may include a correspondence point selected from among a plurality of coordinates representing the surface of the object on image data corresponding to the object. In this case, the surface of the object may mean an isoplanar surface of the first designated point 601. For example, the image processing apparatus 100 may be configured from one designated point among a plurality of points representing the surface of the object on the image data. A plurality of candidate correspondence points may be obtained based on an area within a set distance. In FIG. 6, the second coordinate system 60 is expressed as a two-dimensional plane for convenience of description, but the present disclosure is not limited thereto. For example, the second coordinate system 60 may be a three-dimensional coordinate space.

According to an embodiment of the present disclosure, the image processing apparatus 100 may reduce a reference point designation error FLE by using a plurality of candidate correspondence points acquired through the above-described method. Through this, the image processing apparatus 100 may improve matching performance of accurately displaying the position of the medical device inserted in the object in a virtual space on the image data. For example, referring to FIG. 6, in the first group, the first transformed point is at the closest distance to the first corresponding point located on the left side of the first designated point 601. In addition, the second transformation point in the second group is at the closest distance to the second corresponding point located below the second designated point 602. In addition, in the third group, the third conversion point is at the closest distance to the third designated point 603. In this way, each of the first to third conversion points converted by applying the same matching parameter to each of the plurality of first reference points is the first to third designated points 601, 602, and 603 that are mapped to each of the first to third conversion points. It may not be located at the closest distance to the. Accordingly, the image processing apparatus 100 may reduce the reference point designation error FLE by using a plurality of candidate corresponding points.

Meanwhile, according to another embodiment of the present disclosure, the image processing apparatus 100 may obtain a reference point paired based on feature points predetermined for an object. For example, in steps S402 and S404 of FIG. 4, the image processing apparatus 100 may acquire a reference point in each of the first and second coordinate systems based on a preset feature point. In this case, the preset feature point may be a point previously designated based on a voxel value of the image data. For example, the image processing apparatus 100 may extract a feature point of an object in the image data based on a voxel value included in the image data. The image processing apparatus 100 may acquire a plurality of second reference points based on the extracted feature points. In this case, the image processing apparatus 100 may display a plurality of acquired second reference points through the output device. Through this, the image processing apparatus 100 may guide a user input for designating a plurality of first reference points. The image processing apparatus 100 may acquire a plurality of designated first reference points based on a user input according to the guide. Also, the image processing apparatus 100 may acquire a plurality of candidate correspondence points by the method described in step S404 of FIG. 4.

Hereinafter, a configuration of the image processing apparatus 100 according to an embodiment of the present disclosure will be described with reference to FIG. 7. 7 is a block diagram of an image processing apparatus 100 according to an exemplary embodiment of the present disclosure. As shown in FIG. 7, the image processing apparatus 100 according to an exemplary embodiment of the present disclosure may include a processor 110, an input unit 120, and an output unit 130. However, not all of the components shown in FIG. 7 are essential components of the image processing apparatus 100. For example, the image processing apparatus 100 may additionally include a component not shown in FIG. 7. In addition, at least some of the components of the image processing apparatus 100 illustrated in FIG. 7 may be omitted.

According to an embodiment, the input unit 120 may include an interface for receiving a user input for the image processing apparatus 100. For example, the input unit 120 may receive an input from a user specifying a reference point on the image data. For example, the input unit 120 includes a key pad, a dome switch, and a touch pad (contact type capacitance method, pressure type resistive film method, infrared detection method, surface ultrasonic conduction method, integral tension type). Measurement method, piezo effect method, etc.), a jog wheel, a jog switch, and the like. Also, the input unit 120 may receive image data corresponding to an object. For example, the input unit 120 may include a keyboard, a mouse, a camera, a microphone, a pointer, a USB, and a connection port for an external device, but the present disclosure is not limited thereto.

According to an embodiment, the output unit 130 may output output image data generated by the processor 110 to be described later. For example, the output unit 130 may include a display that outputs the generated image data. In addition, the output unit 130 may output an audio signal or a sound signal. The output unit 130 may include a wired/wireless port for outputting output image data through an external video/audio output device other than the image processing apparatus 100. In this case, the output unit 130 may be connected to an external video/audio output device through a wired/wireless port. Also, the output unit 130 may output output image data indicating the location of a medical device inserted into the object on the image data received from the input unit 120.

According to an embodiment, the processor 110 may include one or more processors to control the overall operation of the image processing apparatus 100. For example, the processor 110 may perform the operation of the image processing apparatus 100 described above with reference to FIGS. 2 to 6 by executing at least one program. The processor 110 may generate output image data including the object by using a reference point sensed on the object. The processor 110 may generate output image data based on image data corresponding to an object and a matching parameter.

According to an embodiment, the processor 110 may generate output image data based on image data corresponding to an object. In this case, the processor 110 may acquire image data corresponding to the object. For example, the processor 110 may use image data stored in a memory (not shown) in the image processing apparatus 100. Alternatively, the processor 110 may receive image data corresponding to an object from an external device through a communication unit (not shown). Alternatively, the processor 110 may receive image data through the input unit 120. Also, the processor 110 may acquire a plurality of image data corresponding to an object. For example, the processor 110 may acquire at least one of images of a horizontal plane, a sagittal plane, and a coronal plane corresponding to the object. When there are a plurality of image data corresponding to an object, the processor 110 may determine at least one of the plurality of image data as a processing target based on a user input. In addition, the processor 110 may generate image data to be processed using volume rendering in which a plurality of 2D images are projected onto a 3D sample data set.

Some embodiments may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and may include both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include a computer storage medium. Computer storage media may include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

In addition, in the present specification, the “unit” may be a hardware component such as a processor or a circuit, and/or a software component executed by a hardware configuration such as a processor.

The above description of the present disclosure is for illustrative purposes only, and those of ordinary skill in the art to which the present disclosure pertains will be able to understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present disclosure. will be. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present disclosure is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure. do.

Claims (13)

An image processing apparatus for matching a first coordinate system of a medical device used for an object with a second coordinate system of image data of the object,
Acquiring a set of a plurality of first reference points of the first coordinate system sensed on the object;
Converting each of the first reference points into a conversion point of the second coordinate system based on a matching parameter that converts a point of the first coordinate system to a point of the second coordinate system;
Obtaining a plurality of candidate correspondence points of the image data corresponding to each of the first reference points, wherein the plurality of candidate correspondence points are obtained based on a second reference point of the image data set for the first reference point;
Updating the matching parameter based on a matching error between the converted point of the first reference point and the plurality of candidate corresponding points;
Matching the first coordinate system with the second coordinate system using the updated matching parameter,
Including a processor,
The matching parameter includes a rotation matrix for rotating and moving a point of the first coordinate system and a translation matrix for moving a point of the first coordinate system in parallel,
The matching error is determined based on the following equation,

In the above equation, Rot() denotes the rotation matrix, Trans denotes the translation matrix,

silver

Represents the minimum value of,
N is the number of the plurality of first reference points, K is the number of candidate correspondence points corresponding to any one of the plurality of first reference points,

Is the i-th first reference point among the plurality of first reference points,

Denotes a k-th candidate correspondence point among the plurality of candidate correspondence points corresponding to the i-th first reference point, and the i-th first reference point

The conversion point of

+

An image processing device represented by. delete The method of claim 1,
The processor updates the matching parameter based on a set of candidate correspondence points corresponding to the set of first reference points, which minimizes a matching error of the equation. The method of claim 3,
The processor updates the matching parameter until the matching error becomes less than or equal to a preset threshold. The method of claim 1,
The candidate correspondence point includes a point located within a preset distance on an iso-surface with a second reference point of the image data. The method of claim 1,
The processor determines the number of candidate correspondence points corresponding to each first reference point based on the resolution of the image data. The method of claim 1,
The processor determines the number of candidate correspondence points corresponding to each first reference point based on the matching processing performance of the processor. The method of claim 1,
The processor outputs an output image of the image data generated based on a position in the second coordinate system corresponding to the position of the medical device in the first coordinate system. The method of claim 1,
The processor is an image processing apparatus that displays a position in the second coordinate system corresponding to the position of the medical device in the first coordinate system in the output image of the image data. In the image processing method of an image processing apparatus for matching a first coordinate system of a medical device used for an object with a second coordinate system of image data of the object,
Obtaining a set of a plurality of first reference points of the first coordinate system sensed on the object;
Converting each of the first reference points into a conversion point of the second coordinate system based on a matching parameter that converts a point of the first coordinate system to a point of the second coordinate system;
Obtaining a plurality of candidate correspondence points of the image data corresponding to each of the first reference points, the plurality of candidate correspondence points being obtained based on a second reference point of the image data set for the first reference point;
Updating the matching parameter based on a matching error between the converted point of the first reference point and the plurality of candidate corresponding points; And
Including the step of matching the first coordinate system with the second coordinate system using the updated matching parameter,
The matching parameter includes a rotation matrix for rotating and moving a point of the first coordinate system and a translation matrix for moving a point of the first coordinate system in parallel,
The matching error is determined based on the following equation,

In the above equation, Rot() denotes the rotation matrix, Trans denotes the translation matrix,

silver

Represents the minimum value of,
N is the number of the plurality of first reference points, K is the number of candidate correspondence points corresponding to any one of the plurality of first reference points,

Is the i-th first reference point among the plurality of first reference points,

Denotes a k-th candidate correspondence point among the plurality of candidate correspondence points corresponding to the i-th first reference point, and the i-th first reference point

The conversion point of

+

Image processing method expressed as. A computer-readable recording medium in which a program for executing the method of claim 10 on a computer is recorded. delete delete

Images