WO2022206406A1

WO2022206406A1 - Augmented reality system and method based on spatial position of corrected object, and computer-readable storage medium

Info

Publication number: WO2022206406A1
Application number: PCT/CN2022/081469
Authority: WO
Inventors: 孙非; 朱奕; 郭晓杰; 崔芙粒; 单莹
Original assignee: 上海复拓知达医疗科技有限公司
Priority date: 2021-04-01
Filing date: 2022-03-17
Publication date: 2022-10-06
Also published as: CN113509264A

Abstract

The present invention discloses an augmented reality system and method based on a spatial position of a corrected object. The method comprises: capturing a first object image in a space, and recognizing a first object recognition characteristic in the first object image, so as to obtain first object spatial position information; when a second object is at a specific position, capturing a second object image of the second object in the space, and recognizing a second object recognition characteristic in the second object image, so as to obtain second object spatial position information; correcting the second object spatial position information according to the first object spatial position information, and correcting the first object spatial position information according to the second object spatial position information; and according to the first object spatial position information and/or the second object spatial position information, displaying augmented reality information related to the position of the first object and/or the second object. By means of the method, a user can be helped to perform a precise and complete operation.

Description

An augmented reality system, method and computer-readable storage medium based on correcting the position of an object in space

technical field

The present invention relates to the technical field of image processing, and in particular, to an augmented reality system and method based on correcting the position of an object in space.

Background technique

Augmented reality technology usually captures images of real scenes through cameras, and needs to analyze and process the captured images of real scenes, and add additional information based on the real scenes to display to users, that is, augmentation of reality. The process of analyzing and processing images of real scenes often includes locating objects in the scene. Under certain specific requirements, the accuracy of object positioning in a scene is extremely high, and the accuracy of object positioning in a scene in the prior art cannot meet the requirements.

For example, when augmented reality technology is applied to surgical navigation scenarios, it is necessary to very accurately determine the positional relationship between medical devices, patients, and scenarios to ensure accurate navigation information is provided to users. For example, puncture navigation based on augmented reality technology can realize fast and accurate surgical navigation with the most simple, convenient, easy-to-learn and easy-to-use equipment. In the whole process, one of the cores of precise navigation: accurate spatial positioning of surgical instruments based on visible light patterns, and registration of virtual organs and real human bodies, all depend on the accurate spatial positioning of identifiable patterns on the object to be positioned. Due to the limitation of device design, identifiable patterns of different sizes and shapes have different spatial positioning accuracy due to the inherent laws of spatial distribution of their own pattern feature points or the characteristics of their production processes. If it is an identifier that is used repeatedly, it is possible to improve its recognition accuracy in advance through factory calibration before the first clinical use, but it is difficult to have similar pre-calibration opportunities for single-use and inconsistent product error distributions. How to quickly improve the pattern recognition accuracy in the field of use is a major difficulty in the practical application of this technology.

SUMMARY OF THE INVENTION

In view of the above defects or deficiencies, the purpose of the present invention is to provide an augmented reality system and method based on correcting the position of an object in space.

For achieving the above purpose, the technical scheme of the present invention is:

An augmented reality system based on correcting the position of an object in space, comprising: a first acquisition unit, a second acquisition unit, a correction unit and a display unit, wherein:

The first acquisition unit is configured to capture the image of the first object in space, and identify the first object recognition characteristic in the image of the first object to obtain the spatial position information of the first object;

The second acquisition unit is configured to capture a second object image of the second object in space when the second object is at a specific position, and identify the second object identification characteristics in the second object image to obtain a second object image. object space position information;

The correction unit includes a first correction unit and/or a second correction unit, wherein:

the first correction unit, configured to correct the second object space position information according to the first object space position information and the specific position;

the second correcting unit, configured to correct the spatial position information of the first object according to the spatial position information of the second object;

The display unit is configured to display augmented reality information related to the position of the first object or the second object.

The first object identification characteristic includes at least the first object body shape characteristic and/or the first object mark identification characteristic; the first object body shape characteristic at least includes the structure, shape or color of the first object body; the first object body The object mark identification feature at least includes a pattern, graphic or two-dimensional code set on the first object;

The second object identification characteristic at least includes the second object ontology morphological characteristic and/or the second object mark identification characteristic; the second object ontology morphological characteristic at least includes the structure, shape or color of the second object ontology; the second object ontology The object marking identification features at least include patterns, graphics or two-dimensional codes provided on the second object.

The first object space position information at least includes the first object space coordinates and/or the first object orientation; the second object space position information at least includes the second object space coordinates and/or the second object orientation.

The specific position is the position when the second object has a specific positional relationship with the first object, and the specific positional relationship includes the position between the second object and a preset point, line or surface on the first object Coincidence or partial coincidence.

The first correction unit is specifically used for:

Calculate the theoretical position information of the second object according to the spatial position information of the first object and the specific position relationship; and correct the spatial position information of the second object according to the theoretical position information of the second object.

The first correction unit is used for correcting the x and y coordinates of the second object.

The second correction unit is specifically used for:

Calculate the theoretical position information of the first object according to the spatial position information of the second object and the specific position relationship; and correct the spatial position information of the first object according to the theoretical position information of the first object.

The second correction unit is used for correcting the z coordinate of the first object.

The first object is a fixture in a surgical scene; the second object is an operating instrument in the surgical scene.

An augmented reality method based on correcting the position of objects in space, including:

capturing the first object image in space, and identifying the first object recognition characteristics in the first object image to obtain the spatial position information of the first object;

When the second object is at a specific position, capturing a second object image of the second object in space, and recognizing the second object identification characteristics in the second object image, to obtain the spatial position information of the second object;

Correcting the second object space position information according to the first object space position information and a specific position; and/or correcting the first object space position information according to the second object space position information;

Augmented reality information related to the location of the first object or the second object is displayed.

The specific position is a position when the second object has a specific positional relationship with a preset point, line or surface on the first object, and the specific positional relationship includes coincidence or partial coincidence of points, lines or surfaces.

The correcting the second object space position information according to the first object space position information and the specific position includes: calculating the second object theoretical position according to the first object space position information and the specific position relationship information; correcting the spatial position information of the second object according to the theoretical position information of the second object.

Preferably, correcting the spatial position information of the second object includes correcting the x and y coordinates of the second object.

Correcting the first object space position information according to the second object space position information includes: calculating the second object theoretical position information according to the first object space position information and the specific position relationship; The theoretical position information of the second object is used to correct the spatial position information of the second object.

Preferably, the calibrating the spatial position information of the second object includes calibrating the x and y coordinates of the second object.

The present invention also provides a computer-readable storage medium storing a non-transitory computer-executable program for instructing the computer to execute the method described in the present invention.

Compared with the prior art, the beneficial effects of the present invention are:

The present invention provides an augmented reality system and method based on correcting the position of an object in space. By using the identification characteristics of objects with different error characteristics in the same scene, and through the spatial association of the two corresponding objects, two different objects can be compared with each other. Carry out mutual correction of image acquisition and position, and realize the improvement of optical positioning accuracy of one or both parties. The method and system can be applied in various occasions, such as the positioning of medical device operations during surgery, applications in teaching simulation operations, and The application in the process of game activities, etc., accurate positioning and augmented reality of the location, can help users perform accurate and complete operations.

Description of drawings

1 is a structural block diagram of an augmented reality system based on the present invention for correcting the position of an object in space;

Fig. 2 is the embodiment example diagram in the specific embodiment of the present invention;

Fig. 3 is the flow chart of the augmented reality method of the present invention based on correcting the position of an object in space

FIG. 4 is a schematic diagram of mutual calibration based on the identification plate of the present invention.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In the precise operation scene, it is often necessary to accurately obtain the actual position of the object and the position in the image. Under certain specific requirements, the accuracy of the object positioning in the scene is extremely high. For example, in the medical process, it needs to be very precise. Only by accurately determining the positional relationship between the medical device, the patient, and the scene can provide accurate navigation information to the user. Help medical staff prepare to find the corresponding relationship between the operating position and the body. Based on this requirement, the present invention provides an augmented reality method based on correcting the position of an object in space, which can be applied to an operation scene, an operation scene in a simulated teaching process, or a game process. position.

Taking a surgical implementation scenario as an example, the embodiment of the present invention provides the user with the positioning of the instrument for the tissue and/or the instrument located in the object in the object. Among them, the user is the observer of the whole in vivo navigation process, and he is also the operator who probes the instrument into the body of the subject. Objects can be people or other animals that the user needs to operate on. The instrument can be any tool that can be penetrated into the body of the subject. The instrument may be, for example, a puncture needle, a biopsy needle, a radiofrequency or microwave ablation needle, an ultrasound probe, a rigid endoscope, an endoscopic oval forceps, an electric knife or a stapler and other medical instruments. Preferably, the first object is a fixture in a surgical scene; the second object is an operating instrument in a surgical scene.

As shown in Figure 1, an augmented reality system based on correcting the position of an object in space can be applied to surgical operations, simulated teaching operations, or game processes, and specifically includes: a first acquisition unit 1, a second acquisition unit 2, a correction Unit 3, and Display Unit 4, where:

The first acquisition unit 1 is configured to capture an image of a first object in space, and identify the first object recognition characteristic in the first object image to obtain spatial position information of the first object;

The second acquisition unit 2 is configured to capture a second object image of the second object in space when the second object is at a specific position, and identify the second object identification characteristics in the second object image to obtain the first object image. 2. The spatial position information of the object;

The correction unit 3 includes a first correction unit 31 and/or a second correction unit 32, wherein:

The first correcting unit 31 is configured to correct the spatial position information of the second object according to the spatial position information of the first object and a specific position;

The second correcting unit 32 is configured to correct the spatial position information of the first object according to the spatial position information of the second object;

The display unit 4 is configured to display augmented reality information related to the position of the first object or the second object.

In order to be able to position and calibrate the second object, first obtain the first object space position information of a fixed object, the first object space position information at least includes the first object space coordinates and/or the first object orientation, which can be The specific positioning of the spatial position of the fixed first object is performed.

In the present invention, the first object identification characteristic includes at least the first object body morphological characteristic and/or the first object mark identification characteristic. The morphological characteristic of the first object body at least includes the structure, shape, or color of the first object body, but in the specific implementation process, it is not limited to this, and may also be other identifiable characteristics of the object. Exemplarily, an object with a fixed shape can be fixed in the present invention. Before calibration, the shape of the structure of the object is recognized. During the recognition process, different display methods can be used to prompt the user whether the capture process and the recognition process are successful. The object is positioned and identified, and the accurate spatial position information of the object is obtained.

In addition, in the present invention, the first object mark identification characteristic includes at least a pattern, graphic or two-dimensional code set on the first object. The pattern, graphic or two-dimensional code can be set on the first object through a printing process, and the identifiable patterns have different spatial accuracy according to their own pattern rules and production characteristics. Make full use of the combination of recognizable patterns with different characteristics to achieve rapid spatial calibration of navigation instruments.

Exemplarily, in the present invention, as shown in FIG. 2 , a rectangular information board printed with a two-dimensional code can be used, the device for capturing the image of the first object is a device capable of image capturing, and the capturing angle is kept consistent with the user’s viewing direction. Consistent. When in use, the user may wear the image capture device on the body, such as the head. Optionally, the image capture device is a head-mounted optical camera. When the user uses it, no matter what posture he adopts, the acquisition angle of the head-mounted optical camera can be well kept consistent with its viewing direction. In this way, it is not only ensured that the angle at which the augmented reality information is displayed is the angle viewed by the user, and the accuracy is ensured, but also interference to various operations of the user during use is avoided. This significantly improves the user experience. Position the object in space according to the image collected by the camera, and obtain the position of the object in the xyz space coordinate system, where the z coordinate represents the coordinate along the depth direction captured by the camera, and the x and y coordinates are perpendicular to the z coordinate Coordinates in the axis direction. Obtain the first object image through the image acquisition device, find the first structure information corresponding to the first object in the database according to the first object image, identify the position and orientation of the first object, and set the current spatial coordinates for the first object, denoted as X1, Y1, Z1.

In a specific surgical scenario, instruments need to be used for operation. In the present invention, the second object is a moving instrument, and the second object space position information at least includes the second object space coordinates and/or the second object orientation.

The specific position is the position when the second object has a specific positional relationship with a preset point, line or surface on the first object, for example, the specific positional relationship may be the second object and the first object. Preset points, lines or surfaces on an object coincide or partially coincide within a preset range.

The first correction unit 31 is specifically configured to: calculate the theoretical position information of the second object according to the spatial position information of the first object and the specific position relationship; The spatial position information of the object is corrected; exemplarily, the first correction unit 31 is configured to correct the x and y coordinates of the second object.

The display unit 4 is configured to display the image of the second object, the information content associated with the position of the second object, or the position prompt information associated with the position of the second object.

The second object identification characteristic at least includes the second object body shape characteristic and/or the second object mark identification characteristic; the second object body shape characteristic at least includes the structure, shape or color of the second object body; the second object body The object marking identification features at least include patterns, graphics or two-dimensional codes provided on the second object. The second object space position information at least includes the second object space coordinates and/or the second object orientation.

A two-dimensional code is a black and white plane figure distributed on a plane, the points on it are very easy to identify, and the positioning of the two-dimensional code can be realized by identifying at least three points among them. Because the two-dimensional code is fixed to the object or device, the positioning of the object or device to which the two-dimensional code is fixed can be realized.

Optionally, the second object marker identification characteristic may also be other planar graphics such as a checkerboard. Using QR code or checkerboard as an identification makes positioning objects or instruments more accurate and fast. Thereby, fast moving instruments can be navigated more precisely.

Optionally, the logo fixed on the surface of the instrument can also be a three-dimensional figure. For example, during the design and production process of the instrument, the logo can be the handle of the instrument, or a structure fixed on the side of the handle. Although the calculation time required for recognition is longer than that of plane graphics, the spatial positioning accuracy of stationary or slow-moving objects is relatively high.

Exemplarily, as shown in FIG. 2 , the second object in the present invention is a puncture needle in an operation, and the end of the puncture needle is provided with an identification structure and a two-dimensional code is printed.

Based on the above content, the second obtaining unit 2 is specifically used for:

The first object is fixed in the space, and the second object is a moving object. When the second object moves to the specific position, the second object is identified according to the mark recognition characteristics of the second object, and the first object is obtained. The second object is oriented and/or the current second object space coordinate is set for the second object.

The second correction unit 32 is specifically configured to: calculate the theoretical position information of the first object according to the spatial position information of the second object and the specific position; The spatial position information of the first object is corrected; exemplarily, the second correction unit 32 is used to correct the z-coordinate of the first object.

The display unit 4 is configured to display the image of the first object, the information content associated with the position of the first object, or the position prompt information associated with the position of the first object.

In the present invention, the specific position is the position when the second object has a specific positional relationship with a preset point, line, or surface on the first object. For example, the specific positional relationship may be the second object. The object coincides with a preset point, line or surface on the first object or partially overlaps within a preset range.

When in use, the user can three-dimensionally display in-vivo organs, lesions, and parts of instruments within the subject's body that are not actually visible at the corresponding positions in the actual surgical scene. In other words, invisible internal organs, lesions, and parts of the instrument located within the body are aligned with the human body and the actual instrument to guide the user through the surgical procedure.

In this embodiment, the first object and the second object can be identified, and optical identification objects with different error characteristics can be used in the same scene. Through the spatial association of the two corresponding objects, the optical positioning accuracy of one or both parties can be improved. For the identification objects with different error characteristics, the correlation of the coordinates of different identification patterns in the same space is determined by matching the geometric structure of the instruments with spatial correlation. By using the known trusted values, the calibration of the spatial recognition positions of different recognition patterns is realized.

In addition, as shown in FIG. 3 , the present invention also provides an augmented reality method based on correcting the position of an object in space, including:

S1, capture the first object image in space, and identify the first object recognition characteristic in the first object image, and obtain the first object spatial position information;

In order to perform positioning and calibration on the second object, first obtain specific spatial position information of a fixed object, where the spatial position information at least includes the spatial coordinates of the first object and/or the orientation of the first object. The specific positioning of the spatial location.

In the present invention, the first object identification characteristic includes at least the first object body morphological characteristic and/or the first object mark identification characteristic. The morphological characteristics of the first object body at least include the structure, shape or color of the first object body, but in the specific implementation process, it is not limited to this, and may also be other identifiable characteristics of the object. Exemplarily, an object with a fixed shape can be fixed in the present invention. Before calibration, the shape of the structure of the object is recognized. During the recognition process, different display methods can be used to prompt the user whether the capture process and the recognition process are successful. The object is positioned and identified, and the accurate spatial position information of the object is obtained.

In addition, the first object mark identification characteristic includes at least a pattern, graphic or two-dimensional code set on the first object. The pattern, graphic or two-dimensional code can be set on the first object through a printing process, and the identifiable patterns have different spatial accuracy according to their own pattern rules and production characteristics. Make full use of the combination of recognizable patterns with different characteristics to achieve rapid spatial calibration of navigation instruments.

Exemplarily, as shown in FIG. 2 , a rectangular information board printed with a two-dimensional code may be used, the device for capturing the image of the first object is a device capable of image capturing, and the capturing angle is consistent with the user's viewing direction. When in use, the user may wear the image capture device on the body, such as the head. Optionally, the image capture device is a head-mounted optical camera. When the user uses it, no matter what posture the user adopts, the acquisition angle of the head-mounted optical camera can be well kept consistent with its viewing direction. In this way, it is not only ensured that the displayed angle is the angle viewed by the user, the accuracy of the display of the instrument is ensured, but also interference to various operations of the user during use is avoided. This significantly improves the user experience. The image of the first object is acquired by the image acquisition device, the identification characteristics of the first object are identified, the morphological characteristics of the body of the first object are obtained according to the identification characteristics of the first object, the orientation of the first object is obtained, and the current spatial coordinates of the first object are set for the first object , denoted as X1, Y1, Z1.

S2, when the second object is in a specific position, capture the second object image of the second object in space, and identify the second object recognition characteristic in the second object image, and obtain the second object space position information;

When the second object is at a specific position, capturing the second object image of the second object in space specifically includes:

The first object is fixed in the space, the second object is a moving object, and when the second object moves to a specific position, an image of the second object in the space is captured. In this process, the specific position can be set so that the second object moves to the preset coincidence with the first object, or, according to the needs of actual operation, when a certain position of the second object reaches a fixed position or completes a prescribed action, both can be positioned.

Specifically, the first object is fixed in the space, the second object is a moving object, and when the second object moves to the specific position, the second object is identified according to the identification characteristics of the second object. , obtain the orientation of the second object according to the morphological characteristics of the body of the second object, and set the current spatial coordinates of the second object for the second object, denoted as X2, Y2, and Z2. The specific position is a position when the second object has a specific positional relationship with a preset associated point, line, or surface on the first object, and the specific positional relationship includes coincidence of points, lines or surfaces , partially overlapped.

Exemplarily, the information board is used as the first object, and the puncture needle is used as the second object. When the user holds the puncture needle and makes point B of the needle point coincide with point A of the information board, the positions of the two objects are positioned and calibrated with each other.

S3. Correct the second object space position information according to the first object space position information and a specific position; and/or perform a correction on the first object space position information according to the second object space position information Correction.

S4. Display augmented reality information related to the position of the first object or the second object:

The process can be two processes. According to the actual situation, two objects are relatively corrected, for example, according to the spatial position information of the first object and the specific position, the theoretical position information of the second object is calculated;

Correcting the spatial position information of the second object according to the theoretical position information of the second object; and/or calculating the theoretical position information of the first object according to the spatial position information of the second object and the specific position;

According to the theoretical position information of the first object, the spatial position information of the first object is corrected.

For example, as shown in FIG. 2 , according to the first object image captured, the position information of the object in space is calculated. At this time, the coordinates of point A are the features of the first object captured (mainly on the panel). pattern features) calculated;

When the doctor holds the second object (puncture needle) and places point B of the needle tip at point A of the first object (identification board), at this time, according to the easily identifiable features set at the end of the identification puncture needle, the point B of the needle tip of the puncture needle can be calculated. coordinate;

It is known that the two points A and B are coincident at this time, but the coordinates of the two points A and B obtained through step 1 and step 2 respectively are not necessarily the same. According to the spatial geometric characteristics of the two objects, the accuracy of the x and y coordinates of point A on the first object is high but the accuracy of the z coordinate is relatively low, while the accuracy of the z coordinate of point B on the second object is relatively high. , so the X2 and Y2 coordinates of the second object are corrected according to the X1 and Y1 coordinates of the first object, and the Z1 coordinate of the first object is corrected with the Z2 coordinate of the second object. Then the corresponding positions of the two structures in the database are adjusted as follows:

X2=X1; Y2=Y1; Z1=Z2;

The specific mutual calibration method consists of the following two parts. The schematic diagram of the mutual calibration is shown in Figure 4. In the specific implementation, the first object is the identification plate, and the second object is the puncture needle:

(1) The coordinates of the needle tip point of the puncture needle in the needle identifier coordinate system are manually determined in advance.

(2) Process a hole on the identification plate so that it is parallel to the z-axis and perpendicular to the Oxy plane, and a point at the bottom of the hole is the calibration point. By designing the recognition plate phantom, it is necessary to determine the coordinate p _Q of the calibration point in the recognition plate coordinate system. When calibrating, insert the puncture needle into the hole and ensure that the needle point is at the calibration point. According to the feature that the coordinates of the calibration point in the camera coordinate system remain unchanged, through the coordinate transformation, the following relationship can be known. At this time, the calibration point has the following two expressions in the needle tip coordinate system: T _C←Q p _Q =T _{C← N} p _N

At this time, the calibration point has the following two expressions in the needle tip coordinate system:

(a) The coordinate system of the second object identified by the needle identifier and directly determined by manual point calibration:

(b) The coordinate system of the puncture needle identified by the identification plate and obtained by coordinate transformation:

The above two coordinates are the representation of the calibration point in the needle identifier coordinate system. Assuming that the expression (a) is more accurate for the z coordinate component, and the expression (b) is more accurate for the x and y coordinate components, the result after mutual calibration is

Among them, C: camera coordinate system

Q: Identify the board coordinate system

N: puncture needle coordinate system

T _B←A : Represents the coordinate transformation matrix from coordinate system A to coordinate system B

p _A : point p in coordinate system A

v _A : vector v in coordinate system A

Identifying the plate point calibration method, the camera can identify the positioning plate and the puncture needle, and then T _C←Q and T _C←N can be obtained. Place the puncture needle tip on a fixed point p on the identification plate. From the processing model of the identification plate, the coordinates of the fixed point in the identification plate coordinate system, that is, p _Q , can be determined. According to the feature that the coordinates of this point in the camera coordinate system remain unchanged, the following coordinate relationship can be obtained:

T _C←Q p _Q =T _C←N p _N

Therefore, the coordinates of the point in the puncture needle coordinate system are obtained, that is,

In addition, the present invention can also be calibrated by using direction calibration, which specifically includes:

(1) The direction vector v _N of the puncture needle in the needle identification object coordinate system is manually determined in advance.

(2) Process a hole on the identification plate so that it is parallel to the z-axis and perpendicular to the Oxy plane. A point at the bottom of the hole is the Calibration Point, and the direction of the hole is called the Calibration Direction. By designing the recognition plate phantom, it is necessary to determine the direction vector v _Q of the hole direction in the recognition plate coordinate system. When calibrating, insert the identification needle into the hole, and ensure that the needle point is located at the calibration point. According to the feature that the direction of the calibration direction remains unchanged in the camera coordinate system, through the coordinate transformation, the following relationship can be known:

T _C←Q v _Q =T _C←N v _N

At this time, the calibration direction has two expressions in the needle tip coordinate system:

(a) The direction vector of the puncture needle identified by the needle identifier and directly determined by the manual direction calibration:

(b) The direction vector of the puncture needle identified by the identification plate and obtained by coordinate transformation:

The above two vectors are the representation of the calibration direction in the coordinate system of the needle identifier. Assuming that the expression (a) is more accurate for the w coordinate component, and the expression (b) is more accurate for the u and v coordinate components, the result after mutual calibration is

The method for calibrating the orientation of the identification plate is shown in Figure 4. The camera recognizes the identification plate and the puncture needle, and T _C←Q and R _C←N can be obtained. Insert the tip of the puncture needle into a fixed hole on the identification plate. From the processing model of the identification plate, the direction vector of the hole in the identification plate coordinate system, ie v _Q , can be determined. Since the direction vector does not change in the camera coordinate system, the following conversion relationship can be obtained

T _C←Q v _Q =T _C←N v _N

Therefore, the representation of the direction vector in the puncture needle coordinate system is obtained, that is,

After the direction calibration, when the camera recognizes the needle identification object in real time, the needle tip direction can be calculated in real time according to the following formula:

v _C =T _C←N v _N

Among them, T _C←N is given by the camera after identifying the pin identification object, and v _N is the calibration result calculated by the mutual calibration or the orientation calibration of the positioning plate.

After the calibration is completed, the calibrated spatial position information of the first object and/or the second object is displayed, and augmented reality information related to the position is displayed, which may be that the content of the information is related to the position of the object, or the display position of the information. related to the location of the object.

For those skilled in the art, it is obvious that the above-mentioned specific examples are only the preferred solutions of the present invention. Therefore, the improvements and changes that those skilled in the art may make to certain parts of the present invention still embody the present invention. However, what is achieved is still the purpose of the present invention, which belongs to the scope of protection of the present invention.

Claims

An augmented reality system based on correcting the position of an object in space, comprising: a first acquisition unit, a second acquisition unit, a correction unit and a display unit, wherein:

The first acquisition unit is configured to capture the image of the first object in space, and identify the first object recognition characteristic in the image of the first object to obtain the spatial position information of the first object;

The second acquisition unit is configured to capture a second object image of the second object in space when the second object is at a specific position, and identify the second object identification characteristics in the second object image to obtain a second object image. object space position information;

The correction unit includes a first correction unit and/or a second correction unit, wherein:

the first correction unit, configured to correct the second object space position information according to the first object space position information and the specific position;

the second correcting unit, configured to correct the spatial position information of the first object according to the spatial position information of the second object;

The display unit is configured to display augmented reality information related to the position of the first object or the second object.
The augmented reality system based on correcting the position of an object in space according to claim 1, wherein the first object recognition characteristic at least includes the first object body shape characteristic and/or the first object mark recognition characteristic; the The morphological characteristics of the first object body at least include the structure, shape or color of the first object body; the first object mark identification characteristics at least include patterns, graphics or two-dimensional codes set on the first object;

The second object identification characteristic at least includes the second object ontology morphological characteristic and/or the second object mark identification characteristic; the second object ontology morphological characteristic at least includes the structure, shape or color of the second object ontology; the second object ontology The object marking identification features at least include patterns, graphics or two-dimensional codes provided on the second object.
The augmented reality system based on correcting the position of an object in space according to claim 1, wherein the first object space position information at least includes the first object space coordinates and/or the first object orientation; the second The object space position information includes at least the second object space coordinate and/or the second object orientation.
The augmented reality system based on correcting the position of an object in space according to claim 1, wherein the specific position is a position when the second object has a specific positional relationship with the first object.
The augmented reality system based on correcting the position of an object in space according to claim 4, wherein the first correcting unit is specifically configured to: according to the spatial position information of the first object and the specific position relationship, calculate second object theoretical position information; correcting the spatial position information of the second object according to the second object theoretical position information;

The second correction unit is specifically configured to: calculate the theoretical position information of the first object according to the spatial position information of the second object and the specific position relationship; correcting the spatial position information.
The augmented reality system based on correcting the position of an object in space according to claim 4, wherein the first correcting unit is used to correct the x and y coordinates of the second object; the second correcting The unit is used to correct the z coordinate of the first object.
The augmented reality system based on correcting the position of an object in space according to any one of claims 1 to 6, wherein the first object is a fixture in an operation scene; the second object is an operation Manipulators in the scene.
An augmented reality method based on correcting the position of an object in space, comprising:

capturing the first object image in space, and identifying the first object recognition characteristics in the first object image to obtain the spatial position information of the first object;

When the second object is at a specific position, capturing a second object image of the second object in space, and recognizing the second object identification characteristics in the second object image, to obtain the spatial position information of the second object;

Correcting the second object space position information according to the first object space position information and a specific position; and/or correcting the first object space position information according to the second object space position information;

Augmented reality information related to the location of the first object or the second object is displayed.
The augmented reality method based on correcting the position of an object in space according to claim 8, wherein the first object is a fixture in a surgical scene; the second object is an operating instrument in the surgical scene.
A computer-readable storage medium storing a non-transitory computer-executable program, characterized in that, the computer-executable program is used to instruct the computer to execute the method according to any one of claims 8-9.