CN117213468B - Method and device for inspecting outside of airplane and electronic equipment - Google Patents
Method and device for inspecting outside of airplane and electronic equipment Download PDFInfo
- Publication number
- CN117213468B CN117213468B CN202311445734.6A CN202311445734A CN117213468B CN 117213468 B CN117213468 B CN 117213468B CN 202311445734 A CN202311445734 A CN 202311445734A CN 117213468 B CN117213468 B CN 117213468B
- Authority
- CN
- China
- Prior art keywords
- inspection
- patrol
- glasses
- target
- virtual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 74
- 238000007689 inspection Methods 0.000 claims abstract description 229
- 239000011521 glass Substances 0.000 claims abstract description 143
- 230000033001 locomotion Effects 0.000 claims abstract description 41
- 238000013507 mapping Methods 0.000 claims abstract description 13
- 230000001360 synchronised effect Effects 0.000 claims abstract description 12
- 230000003190 augmentative effect Effects 0.000 claims abstract description 10
- 238000005259 measurement Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 abstract description 14
- 230000003993 interaction Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000007667 floating Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000011179 visual inspection Methods 0.000 description 1
Landscapes
- Processing Or Creating Images (AREA)
Abstract
The invention provides an off-board inspection method and device and electronic equipment of an airplane, and relates to the technical field of airplane inspection. The method comprises the following steps: after an inspection starting instruction of an inspector is received, displaying a virtual inspection map to the inspector through Augmented Reality (AR) glasses worn by the inspector, wherein the virtual inspection map comprises a standard point cloud map of an inspected target aircraft; acquiring the spatial position information of the AR glasses in the current environment in real time based on an information acquisition component on the AR glasses; based on a synchronous positioning and mapping SLAM algorithm, determining pose information of the AR glasses relative to the target aircraft according to the spatial position information and a standard point cloud map of the target aircraft; based on the pose information, updating a virtual cursor representing the patrol personnel and a patrol motion track representing a patrol route of the patrol personnel on the virtual patrol map in real time. The method can reduce the condition of missing inspection in the inspection process of inspection personnel.
Description
Technical Field
The invention relates to the technical field of aircraft inspection, in particular to an aircraft exterior inspection method, an aircraft exterior inspection device and electronic equipment.
Background
The outside inspection of the aircraft refers to that inspection personnel are required to perform fault inspection and evaluation on the external fuselage of the aircraft for daily pre-flight, short-stop and post-flight periods of the aircraft so as to ensure the flight safety of the aircraft. In general, an off-board inspection of an aircraft is performed by an inspector, and the inspection process is generally continued for about half an hour. The conventional inspection method is generally that an inspection personnel holds a paper or electronic inspection list, and inspects items of inspection points one by one around an aircraft according to the inspection points on the inspection list by visual inspection or by using some carried tools. In the inspection process, inspection personnel check and evaluate each inspection point according to the inspection progress, and the inspection result is transmitted to the tower and the crew member after the inspection is finished.
However, in the conventional inspection mode, inspection personnel inspect and evaluate each inspection point based on inspection points listed in an inspection list and personal experience, and some inspection points may be missed due to human factors.
Disclosure of Invention
The invention provides an off-board inspection method, an off-board inspection device and electronic equipment for solving the defect that inspection points can be omitted in the prior art, positioning the current inspection position of an inspection personnel through SLAM positioning technology, displaying a virtual inspection map capable of representing the current inspection position and the inspection movement track of the inspection personnel through AR glasses worn by the inspection personnel, and clearly knowing the current inspection progress and whether the inspection is omitted or not through the current position and the inspection movement track on the virtual inspection map, so that the inspection accuracy is improved.
The invention provides an off-board inspection method of an airplane, which comprises the following steps:
after receiving an inspection starting instruction of an inspector, displaying a virtual inspection map to the inspector through AR (Augmented Reality ) glasses worn by the inspector, wherein the virtual inspection map comprises a standard point cloud map of an inspected target aircraft;
acquiring the spatial position information of the AR glasses in the current environment in real time based on an information acquisition component on the AR glasses;
based on SLAM (Simultaneous Localization and Mapping) algorithm, determining pose information of the AR glasses relative to the target aircraft according to the space position information and a standard point cloud map of the target aircraft;
based on the pose information, updating the position of a virtual cursor representing the patrol personnel and the patrol motion trail representing the patrol route of the patrol personnel on the virtual patrol map in real time.
According to the method for inspecting the outside of the aircraft, the information acquisition component comprises a multi-view camera and an inertial sensor, and the spatial position information comprises spatial image information and inertial measurement data;
the real-time acquisition of the spatial position information of the AR glasses based on the information acquisition component on the AR glasses comprises the following steps:
Acquiring the space image information of the AR glasses in the current environment based on the multi-view camera;
the inertial measurement data of the AR glasses is acquired based on the inertial sensor.
According to the method for inspecting the outside of the airplane, the pose information comprises position information and pose information;
the determining pose information of the AR glasses relative to the target aircraft according to the spatial position information and the standard point cloud map of the target aircraft based on the synchronous positioning and mapping SLAM algorithm comprises the following steps:
based on SLAM algorithm, searching in the standard point cloud map by adopting the space image information, and determining the position information of the AR glasses;
pose information of the AR glasses is determined based on the inertial measurement data.
According to the method for inspecting the outside of the aircraft, which is provided by the invention, the method for inspecting the outside of the aircraft further comprises the following steps:
determining the position of the patrol personnel in the current environment and the head orientation of the patrol personnel based on the pose information and the standard point cloud map;
based on the position and the head orientation of the patrol personnel, displaying a virtual object corresponding to the current patrol target through the AR glasses, wherein the virtual object represents patrol information corresponding to the current patrol target.
According to the method for inspecting the outside of the aircraft provided by the invention, the virtual object corresponding to the current inspection target is displayed through the AR glasses based on the position and the head orientation of the inspection personnel, and the method comprises the following steps:
determining a current inspection target of the inspection personnel according to the position and the head orientation of the inspection personnel;
determining a target patrol block from a plurality of patrol blocks in the standard point cloud map based on the current patrol target, wherein the standard point cloud map is divided into a plurality of patrol blocks in advance;
and displaying the virtual object corresponding to the target patrol block through the AR glasses.
According to the method for inspecting the outside of the aircraft, which is provided by the invention, the method for inspecting the outside of the aircraft further comprises the following steps:
displaying a user main interface to the patrol personnel through the AR glasses, and setting at least one of the following based on an operation control instruction of the patrol personnel: the virtual object, the virtual patrol map and the user main interface corresponding to the current patrol target.
According to the method for inspecting the outside of the aircraft, which is provided by the invention, the method for inspecting the outside of the aircraft further comprises the following steps:
displaying a virtual inspection list to the inspector through the AR glasses;
And marking the check point in the virtual check list according to the check marking instruction of the patrol personnel.
According to the method for inspecting the outside of the aircraft, which is provided by the invention, the method for inspecting the outside of the aircraft further comprises the following steps:
generating a pose information record file after receiving an inspection starting instruction of the inspection personnel;
when the patrol personnel patrol, writing pose information corresponding to the patrol personnel into the pose information record file;
reading a target inspection file according to a historical inquiry instruction of an inquirer, and dynamically displaying a target inspection motion track on the virtual inspection map according to the target inspection file, wherein the target inspection file comprises at least one pose information record file or comprises a standard track record file and at least one pose information record file;
under the condition that the target inspection file comprises at least one pose information record file, the model numbers of target aircrafts corresponding to the pose information record files are consistent; and under the condition that the target inspection file comprises the standard track record file and at least one pose information record file, the model numbers of the target aircraft corresponding to the pose information record file and the standard track record file are consistent.
The invention also provides an aircraft exterior inspection device, which comprises:
the display module is used for displaying a virtual inspection map to the inspector through the augmented reality AR glasses worn by the inspector after receiving an inspection starting instruction of the inspector, wherein the virtual inspection map comprises a standard point cloud map of an inspected target aircraft;
the acquisition module is used for acquiring the space position information of the AR glasses in the current environment in real time based on the information acquisition component on the AR glasses;
the repositioning module is used for determining pose information of the AR glasses relative to the target aircraft according to the space position information and a standard point cloud map of the target aircraft based on a synchronous positioning and mapping SLAM algorithm; and the position of the virtual cursor representing the patrol personnel and the patrol motion trail representing the patrol route of the patrol personnel are updated on the virtual patrol map in real time based on the pose information.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the method for inspecting the outside of the airplane when executing the program.
The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements an off-board inspection method as described in any one of the above.
The invention also provides a computer program product comprising a computer program which when executed by a processor implements an off-board inspection method as described in any one of the above.
According to the method, the device and the electronic equipment for the aircraft exterior inspection, the virtual inspection map is displayed to the inspector after the inspection is started through the AR glasses, the space position information of the AR glasses worn by the inspector in the current environment is collected in the inspection process of the inspector through the information collection component on the AR glasses, the pose information of the AR glasses relative to the target aircraft can be determined through the SLAM algorithm according to the space position information and the standard point cloud map of the inspected target aircraft, the current inspection position of the inspector can be determined according to the pose information, and the position of a virtual cursor representing the inspector and the inspection motion trail representing the inspection route of the inspector can be updated on the virtual inspection map in real time, so that the visual azimuth and motion information are provided for the inspector, and the inspector can intuitively know whether the inspection is missed or not and the current inspection progress according to the virtual cursor and the inspection motion trail in the inspection process.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of an off-board inspection method of an aircraft according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of an off-board inspection device for an aircraft according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Aiming at the problem that the inspection quality is difficult to guarantee in the prior art, the embodiment of the invention provides an off-board inspection method for an airplane, which is described below with reference to fig. 1. Fig. 1 is a flow chart of an off-board inspection method of an aircraft according to an embodiment of the present invention, as shown in fig. 1, the off-board inspection method of the aircraft includes the following steps:
step 110: after an inspection starting instruction of an inspector is received, displaying a virtual inspection map to the inspector through Augmented Reality (AR) glasses worn by the inspector, wherein the virtual inspection map comprises a standard point cloud map of an inspected target aircraft;
specifically, the patrol personnel enter a patrol state after wearing and starting the patrol function of the AR glasses, and after receiving a patrol start command of the patrol personnel, the patrol personnel can be shown with a virtual patrol map through the AR glasses, wherein the virtual patrol map comprises a standard point cloud map of a target aircraft to be patrol, and the standard point cloud map virtualizes a 3D model of the target aircraft so as to simulate the current patrol environment. For example, the patrol personnel can issue the patrol start instruction by means of a button, a touch screen or a gesture.
Step 120: and acquiring the spatial position information of the AR glasses in the current environment in real time based on an information acquisition component on the AR glasses.
Specifically, when the patrol personnel patrol and examine, the AR glasses need to be worn, the information acquisition components are arranged on the AR glasses, and when the patrol personnel start patrol and examine, the information acquisition components on the AR glasses can be synchronously started when the patrol personnel receive a patrol and examine starting instruction, so that the space position information of the AR glasses in the current environment is acquired. It is understood that the AR glasses are worn by the inspector, and thus the spatial position information of the AR glasses is also the spatial position information of the inspector. The AR technology is a technology for skillfully fusing virtual information with the real world, and widely uses various technical means such as multimedia, three-dimensional modeling, real-time tracking and registering, intelligent interaction, sensing and the like, and applies virtual information such as characters, images, three-dimensional models, music, videos and the like generated by a computer to the real world after simulation, wherein the two information are mutually complemented, so that the enhancement of the real world is realized.
In one embodiment, the information acquisition component comprises a multi-view camera and an inertial sensor, and the spatial location information comprises spatial image information and inertial measurement data;
the real-time acquisition of the spatial position information of the AR glasses based on the information acquisition component on the AR glasses comprises the following steps:
Acquiring the space image information of the AR glasses in the current environment based on the multi-view camera;
the inertial measurement data of the AR glasses is acquired based on the inertial sensor.
Specifically, the information collection assembly provided on the AR glasses includes a multi-view camera and an inertial sensor, and the multi-view camera may be provided on both sides of the glasses body on the AR glasses, for example. The multi-view camera is a camera for synchronously shooting by a plurality of cameras, and the space image information of the AR glasses in the current environment can be obtained by splicing the image information shot by the plurality of cameras. The inertial sensor is a sensor mainly used for detecting and measuring acceleration, inclination, impact, vibration, rotation and multiple degree of freedom (DoF) motion, and is an important component for solving navigation, orientation and motion carrier control. In this embodiment, inertial measurement data of the AR glasses may be obtained by the inertial sensor.
According to the method for inspecting the aircraft outside of the embodiment, the data support can be provided for subsequent positioning based on the SLAM algorithm through the arranged multi-camera and the arranged inertial sensor. Furthermore, by arranging the multi-view camera on the AR glasses, compared with the case that the single-view camera can only obtain two-dimensional space image information, the embodiment can obtain three-dimensional space image information, and can improve the accuracy of pose information of the subsequently determined AR glasses relative to the target aircraft.
Step 130: based on a synchronous positioning and mapping SLAM algorithm, pose information of the AR glasses relative to the target aircraft is determined according to the space position information and a standard point cloud map of the target aircraft.
Specifically, for each model of aircraft, before the first aircraft outside inspection, a standard point cloud map of the model of aircraft needs to be acquired first, and the method for acquiring the standard point cloud map may be as follows: the user wears AR glasses to visually inspect each inspection point of the airplane around the airplane according to the standard route for the outside inspection of the airplane and the inspection flow, which are specified by the airline company; starting a target acquisition component on the AR glasses, and acquiring the spatial position information of the AR glasses along with the movement of a user by the target acquisition component; and generating a standard point cloud map of the aircraft according to all the spatial position information of the aircraft based on the SLAM algorithm. The standard point cloud map may be stored at the server side, and the AR glasses may call the standard point cloud map from the server side, or may directly store the standard point cloud map on the AR glasses, which is not specifically limited herein.
During inspection, based on SLAM algorithm, according to the space position information and the standard point cloud map of the inspected target aircraft, the pose information of AR glasses worn by inspection personnel relative to the target aircraft can be determined.
In one embodiment, the pose information includes position information and pose information;
the determining pose information of the AR glasses relative to the target aircraft according to the spatial position information and the standard point cloud map of the target aircraft based on the synchronous positioning and mapping SLAM algorithm comprises the following steps:
based on SLAM algorithm, searching in the standard point cloud map by adopting the space image information, and determining the position information of the AR glasses;
pose information of the AR glasses is determined based on the inertial measurement data.
Specifically, the pose information includes position information and pose information, where the position information indicates a current position of the AR glasses (i.e., the patrol personnel), and the pose information indicates a current pose of the AR glasses, such as a rotation vector of the AR glasses.
When the pose information of the AR glasses relative to the target aircraft is determined, the position information of the AR glasses, namely the position information of the AR glasses relative to the target aircraft, is determined by searching on a standard point cloud map of the target aircraft based on the space image information acquired by the multi-view camera through an SLAM algorithm. From the inertial measurement data, pose information of the AR glasses, such as the orientation of the AR glasses, may be determined. Alternatively, the SLAM algorithm may be a SLAM algorithm module integrated in AR glasses.
Step 140: based on the pose information, updating the position of a virtual cursor representing the patrol personnel and the patrol motion trail representing the patrol route of the patrol personnel on the virtual patrol map in real time.
Specifically, a virtual cursor representing the patrol personnel can be displayed on the virtual patrol map, the position of the virtual cursor on the virtual patrol map is the current position of the patrol personnel, and along with the movement of the patrol personnel, the virtual cursor can be updated in real time based on the determined pose information, so that the animation effect of the dynamic movement position of the patrol personnel is displayed on the virtual patrol map. Further, the patrol motion track representing the patrol route of the patrol personnel can be displayed on the virtual patrol map according to the movement of the patrol personnel. The virtual cursor and the patrol motion track can provide necessary direction guidance for patrol personnel.
For example, in the Unity virtual space, the virtual cursor can be deployed and aligned with the coordinate system of the virtual cursor and the coordinate system of the real space through the coordinate information provided by the standard point cloud map generated by the SLAM algorithm, so as to complete the mixed reality display and interaction.
According to the aircraft exterior inspection method provided by the invention, the virtual inspection map is displayed to the inspector after the inspection is started through the AR glasses, the spatial position information of the AR glasses worn by the inspector in the current environment is acquired in the inspection process of the inspector through the information acquisition component on the AR glasses, the pose information of the AR glasses relative to the target aircraft can be determined through the SLAM algorithm according to the spatial position information and the standard point cloud map of the inspected target aircraft, and then the current inspection position of the inspector can be determined according to the pose information, so that the position of the virtual cursor representing the inspector and the inspection motion track representing the inspection route of the inspector are updated on the virtual inspection map in real time, and visual azimuth and motion information are provided for the inspector, so that the inspector can intuitively know whether the inspection is missed or not and the current inspection progress according to the virtual cursor and the inspection motion track in the inspection process.
In one embodiment, the inspection method further comprises:
determining the position of the patrol personnel in the current environment and the head orientation of the patrol personnel based on the pose information and the standard point cloud map;
based on the position and the head orientation of the patrol personnel, displaying a virtual object corresponding to the current patrol target through the AR glasses, wherein the virtual object represents patrol information corresponding to the current patrol target.
Specifically, after the pose information of the AR glasses relative to the target aircraft is obtained, the position of the patrol inspector in the current environment and the head orientation of the patrol inspector can be determined according to the pose information and the standard point cloud map. And determining a current inspection target according to the position and the head orientation of the inspection personnel, and displaying a virtual object corresponding to the current inspection target through the AR glasses. It may be appreciated that the virtual object characterizes the inspection information corresponding to the current inspection target, and the inspection information may include, for example: the information of which content to be checked at the current inspection target, the inspection guide identifier, what is the last inspection point, what is the next inspection point, what faults occur in the history inspection process and the like is included, so that the inspection personnel can be subjected to inspection auxiliary reminding.
For example, a Unity application may be built in the AR glasses, and the Unity application may receive pose information sent by the SLAM algorithm module through the SDK (Software Development Kit, software development kit tool), and the Unity application may execute the above method steps.
The Unity application is pre-written, and the corresponding Unity application can be written for each model of aircraft, which is not specifically limited herein. The interaction toolkit (XR Interaction Toolkits) provided by the Unity authority may provide the necessary AR interaction components for the Unity application. By adding various components to the virtual object (including, but not limited to, components self-contained by the Unity engine, components provided by the plug-in, and custom components), the virtual object performs corresponding logical operations, and the Unity application then has corresponding functions. On the basis of the principle, a required art model can be imported, the relation among virtual objects is created and designed, components are added for each virtual object to be perfected, and then the Unity application program can be obtained.
In one embodiment, the displaying, by the AR glasses, the virtual object corresponding to the current inspection target based on the position and the head orientation of the inspection personnel includes:
Determining a current inspection target of the inspection personnel according to the position and the head orientation of the inspection personnel;
determining a target patrol block from a plurality of patrol blocks in the standard point cloud map based on the current patrol target, wherein the standard point cloud map is divided into a plurality of patrol blocks in advance;
and displaying the virtual object corresponding to the target patrol block through the AR glasses.
Specifically, the standard point cloud map of the target aircraft may be divided into areas in advance, that is, the standard point cloud map is divided into a plurality of patrol blocks. In the process of the inspection personnel, the current inspection target can be determined according to the position and the head direction of the inspection personnel, the inspection block to which the current inspection target belongs is determined from the standard point cloud map according to the current inspection target, the inspection block is determined to be a target inspection block, and all virtual objects corresponding to the target inspection block are displayed through the AR glasses.
According to the aircraft exterior inspection method in the embodiment, the virtual space and the real physical space are overlapped, the corresponding virtual object of the current inspection target inspection block of the inspector is rendered on the lenses of the AR glasses, and therefore various inspection information for auxiliary judgment can be provided for the inspector, the inspection information can assist the inspector to inspect various inspection points, and the inspector is helped to finish the inspection task in high quality.
Alternatively, the virtual object may be displayed in a window with a virtual inspection map, for example, the virtual inspection map may be a smaller virtual inspection map scaled down equally, and the virtual inspection map may be displayed near the edge of the window to avoid affecting inspection of the virtual object by inspection personnel.
Optionally, a floating window can be further set outside the window to which the virtual object belongs, and the floating window can display the inspection progress, the flight information of the target aircraft and the like, and the inspection progress can be visually displayed to an inspection personnel in a percentage mode.
In one embodiment, the inspection method further comprises:
displaying a user main interface to the patrol personnel through the AR glasses, and setting at least one of the following based on an operation control instruction of the patrol personnel: the virtual object, the virtual patrol map and the user main interface corresponding to the current patrol target.
Specifically, the inspection method further comprises the step of displaying a user main interface to the inspection personnel through the AR glasses, wherein the user main interface provides functions of setting a virtual object, setting a virtual inspection map, setting the user main interface and the like. Therefore, the patrol personnel can set at least one of the virtual object, the virtual patrol map and the user main interface corresponding to the current patrol target by inputting an operation control instruction on the user main interface.
After the inspection is started, the user main interface can be hidden, namely, a window of the virtual inspection map and/or a window corresponding to the virtual object is displayed after the inspection is started. When the patrol personnel needs to operate on the user main interface, the patrol personnel can switch from the current window to the user main interface window, and then perform setting operation on the user main interface.
The user main interface includes a control component for specially controlling the virtual inspection map, for example, a fixed anchor point can be set in the visual field range of the multi-camera for the virtual inspection map, so that when the inspection personnel moves, the virtual inspection map can move along with the fixed anchor point, i.e. can be always displayed in a window corresponding to the virtual object. The attribute of the virtual inspection map can be adjusted through the control component for controlling the virtual inspection map, such as rotating, zooming, dragging, hiding and the like, whether the virtual inspection map is started or not can be set, and the style of a virtual cursor and/or an inspection movement track on the virtual inspection map can be set. Optionally, the virtual tour map corresponds to a default attribute.
The user main interface also comprises a control component which is specially used for controlling the virtual objects, and whether each virtual object is loaded or disabled can be set for each virtual object. Alternatively, each virtual object may correspond to a default attribute.
On the user main interface, the attribute of the user main interface can be adjusted and set.
In one embodiment, the inspection method further comprises:
displaying a virtual inspection list to the inspector through the AR glasses;
and marking the check point in the virtual check list according to the check marking instruction of the patrol personnel.
Specifically, the method further comprises: the virtual checklist may be presented to the inspector through the AR glasses, including all checkpoints of the target aircraft. The virtual checklist can be hidden or displayed based on instructions of the patrol personnel, and a user can mark checkpoints in the virtual checklist according to own checking progress, so that the patrol personnel can determine checked checkpoints and non-checked checkpoints through the virtual checklist, and further the patrol personnel can complete control over the patrol process.
For example, when the AR glasses are AR glasses capable of gesture interaction, the AR glasses can hide or display the virtual checklist and mark the checkpoints through gesture interaction with the inspector.
By combining the patrol motion trail in the virtual patrol map and the virtual checklist, patrol personnel can more clearly know the current patrol progress and the positions with or without omission in the patrol process, and the patrol accuracy is further improved.
In one embodiment, the inspection method further comprises:
generating a pose information record file after receiving an inspection starting instruction of the inspection personnel;
when the patrol personnel patrol, writing pose information corresponding to the patrol personnel into the pose information record file;
reading a target inspection file according to a historical inquiry instruction of an inquirer, and dynamically displaying a target inspection motion track on the virtual inspection map according to the target inspection file, wherein the target inspection file comprises at least one pose information record file or comprises a standard track record file and at least one pose information record file;
under the condition that the target inspection file comprises at least one pose information record file, the model numbers of target aircrafts corresponding to the pose information record files are consistent; and under the condition that the target inspection file comprises the standard track record file and at least one pose information record file, the model numbers of the target aircraft corresponding to the pose information record file and the standard track record file are consistent.
Specifically, the method further comprises the step of generating a pose information record file for recording pose information for the current inspection after an inspection starting instruction of an inspection personnel is received. When the patrol personnel patrol, the pose information corresponding to the patrol personnel, namely the pose information of the AR glasses worn by the patrol personnel relative to the target aircraft, is written into a pose information record file for storage. Pose information may be written into a pose information record file, for example, at a refresh frame rate that is fixed every interval.
And after receiving the historical inquiry instruction of the inquirer, the target inspection file can be read, and the target inspection movement track corresponding to the target inspection file is displayed on the virtual inspection map according to the target inspection file. The target inspection file can comprise at least one pose information record file, namely an inquirer can instruct to read the pose information record file of one-time inspection and display a target inspection motion track in a virtual inspection map, or instruct to read the pose information record file of the same model or the same plane for multiple inspection and display the target inspection motion track in the virtual inspection map, and meanwhile display the target inspection motion track for multiple inspection, so that the inquirer can conveniently carry out comparison inspection on multiple tracks. The target inspection file can also comprise a standard track record file on the basis of comprising at least one pose information record file, wherein each pose information record file and each standard track record file belong to the same model of airplane or all belong to the same airplane. The standard track record file of the aircraft can be pre-stored in the AR glasses, the target inspection motion track corresponding to the pose information record file can be compared and checked through the target inspection motion track corresponding to the standard track record file, and the omission position in the history inspection process can be clearly found through the comparison and check.
The method for inspecting the aircraft outside of the embodiment provides the inquiry of the historical inspection data, and the inspection running track can be reappeared in a visual mode, so that the inquirer can trace back the historical inspection task conveniently.
The following describes the outboard inspection device of the aircraft provided by the invention, and the outboard inspection device of the aircraft described below and the outboard inspection method of the aircraft described above can be correspondingly referred to each other.
Fig. 2 is a schematic structural diagram of an outboard inspection device of an aircraft according to an embodiment of the present invention, and as shown in fig. 2, the outboard inspection device 200 of the aircraft includes:
the display module 210 is configured to display a virtual inspection map to an inspector through augmented reality AR glasses worn by the inspector after receiving an instruction to start inspection by the inspector, where the virtual inspection map includes a standard point cloud map of a target aircraft to be inspected;
the acquisition module 220 is configured to acquire, in real time, spatial position information of the AR glasses in the current environment based on an information acquisition component on the AR glasses;
a repositioning module 230, configured to determine pose information of the AR glasses relative to the target aircraft according to the spatial location information and a standard point cloud map of the target aircraft based on a synchronous positioning and mapping SLAM algorithm; and the position of the virtual cursor representing the patrol personnel and the patrol motion trail representing the patrol route of the patrol personnel are updated on the virtual patrol map in real time based on the pose information.
According to the aircraft exterior inspection device provided by the invention, the virtual inspection map is displayed to the inspector after the inspection is started through the AR glasses, the spatial position information of the AR glasses worn by the inspector in the current environment is acquired in the inspection process of the inspector through the information acquisition component on the AR glasses, the pose information of the AR glasses relative to the target aircraft can be determined through the SLAM algorithm according to the spatial position information and the standard point cloud map of the inspected target aircraft, and then the current inspection position of the inspector can be determined according to the pose information, so that the position of the virtual cursor representing the inspector and the inspection motion track representing the inspection route of the inspector are updated on the virtual inspection map in real time, and visual azimuth and motion information are provided for the inspector, so that the inspector can intuitively know whether the inspection is missed or not and the current inspection progress according to the virtual cursor and the inspection motion track in the inspection process.
In one embodiment, the information acquisition component comprises a multi-view camera and an inertial sensor, and the spatial location information comprises spatial image information and inertial measurement data;
the collection module 220 is specifically configured to:
Acquiring the space image information of the AR glasses in the current environment based on the multi-view camera;
the inertial measurement data of the AR glasses is acquired based on the inertial sensor.
In one embodiment, the pose information includes position information and pose information;
the relocation module 230 is specifically configured to:
based on SLAM algorithm, searching in the standard point cloud map by adopting the space image information, and determining the position information of the AR glasses;
pose information of the AR glasses is determined based on the inertial measurement data.
In one embodiment, the inspection device further comprises:
the virtual object module is used for determining the position of the patrol personnel in the current environment and the head orientation of the patrol personnel based on the pose information and the standard point cloud map;
based on the position and the head orientation of the patrol personnel, displaying a virtual object corresponding to the current patrol target through the AR glasses, wherein the virtual object represents patrol information corresponding to the current patrol target.
In one embodiment, the virtual object module is specifically further configured to:
determining a current inspection target of the inspection personnel according to the position and the head orientation of the inspection personnel;
Determining a target patrol block from a plurality of patrol blocks in the standard point cloud map based on the current patrol target, wherein the standard point cloud map is divided into a plurality of patrol blocks in advance;
and displaying the virtual object corresponding to the target patrol block through the AR glasses.
In one embodiment, the inspection device further comprises:
the main interface module is used for displaying a user main interface to the patrol personnel through the AR glasses and setting at least one of the following based on the operation control instruction of the patrol personnel: the virtual object, the virtual patrol map and the user main interface corresponding to the current patrol target.
In one embodiment, the inspection device further comprises:
the virtual list module is used for displaying a virtual inspection list to the patrol personnel through the AR glasses;
and marking the check point in the virtual check list according to the check marking instruction of the patrol personnel.
In one embodiment, the inspection device further comprises:
the history inquiry module is used for generating a pose information record file after receiving an inspection starting instruction of the inspection personnel;
When the patrol personnel patrol, writing pose information corresponding to the patrol personnel into the pose information record file;
reading a target inspection file according to a historical inquiry instruction of an inquirer, and dynamically displaying a target inspection motion track on the virtual inspection map according to the target inspection file, wherein the target inspection file comprises at least one pose information record file or comprises a standard track record file and at least one pose information record file;
under the condition that the target inspection file comprises at least one pose information record file, the model numbers of target aircrafts corresponding to the pose information record files are consistent; and under the condition that the target inspection file comprises the standard track record file and at least one pose information record file, the model numbers of the target aircraft corresponding to the pose information record file and the standard track record file are consistent.
Fig. 3 illustrates a physical schematic diagram of an electronic device, as shown in fig. 3, where the electronic device may include: processor 310, communication interface (Communications Interface) 320, memory 330 and communication bus 340, wherein processor 310, communication interface 320, memory 330 accomplish communication with each other through communication bus 340. The processor 310 may invoke logic instructions in the memory 330 to perform an off-board inspection method of an aircraft, the method comprising:
After an inspection starting instruction of an inspector is received, displaying a virtual inspection map to the inspector through Augmented Reality (AR) glasses worn by the inspector, wherein the virtual inspection map comprises a standard point cloud map of an inspected target aircraft;
acquiring the spatial position information of the AR glasses in the current environment in real time based on an information acquisition component on the AR glasses;
based on a synchronous positioning and mapping SLAM algorithm, determining pose information of the AR glasses relative to the target aircraft according to the spatial position information and a standard point cloud map of the target aircraft;
based on the pose information, updating the position of a virtual cursor representing the patrol personnel and the patrol motion trail representing the patrol route of the patrol personnel on the virtual patrol map in real time.
Further, the logic instructions in the memory 330 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being storable on a non-transitory computer readable storage medium, the computer program, when executed by a processor, being capable of executing the method for inspecting the outside of an aircraft provided by the above methods, the method comprising:
after an inspection starting instruction of an inspector is received, displaying a virtual inspection map to the inspector through Augmented Reality (AR) glasses worn by the inspector, wherein the virtual inspection map comprises a standard point cloud map of an inspected target aircraft;
acquiring the spatial position information of the AR glasses in the current environment in real time based on an information acquisition component on the AR glasses;
based on a synchronous positioning and mapping SLAM algorithm, determining pose information of the AR glasses relative to the target aircraft according to the spatial position information and a standard point cloud map of the target aircraft;
based on the pose information, updating the position of a virtual cursor representing the patrol personnel and the patrol motion trail representing the patrol route of the patrol personnel on the virtual patrol map in real time.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the method for aircraft exterior inspection provided by the above methods, the method comprising:
After an inspection starting instruction of an inspector is received, displaying a virtual inspection map to the inspector through Augmented Reality (AR) glasses worn by the inspector, wherein the virtual inspection map comprises a standard point cloud map of an inspected target aircraft;
acquiring the spatial position information of the AR glasses in the current environment in real time based on an information acquisition component on the AR glasses;
based on a synchronous positioning and mapping SLAM algorithm, determining pose information of the AR glasses relative to the target aircraft according to the spatial position information and a standard point cloud map of the target aircraft;
based on the pose information, updating the position of a virtual cursor representing the patrol personnel and the patrol motion trail representing the patrol route of the patrol personnel on the virtual patrol map in real time.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. An off-board inspection method for an aircraft, comprising:
after an inspection starting instruction of an inspector is received, displaying a virtual inspection map to the inspector through Augmented Reality (AR) glasses worn by the inspector, wherein the virtual inspection map comprises a standard point cloud map of an inspected target aircraft;
acquiring the spatial position information of the AR glasses in the current environment in real time based on an information acquisition component on the AR glasses;
based on a synchronous positioning and mapping SLAM algorithm, determining pose information of the AR glasses relative to the target aircraft according to the spatial position information and a standard point cloud map of the target aircraft;
based on the pose information, updating the position of a virtual cursor representing the patrol personnel and the patrol motion trail representing the patrol route of the patrol personnel on the virtual patrol map in real time;
the information acquisition component comprises a multi-view camera and an inertial sensor, and the spatial position information comprises spatial image information and inertial measurement data;
the real-time acquisition of the spatial position information of the AR glasses based on the information acquisition component on the AR glasses comprises the following steps:
acquiring the space image information of the AR glasses in the current environment based on the multi-view camera;
Acquiring the inertial measurement data of the AR glasses based on the inertial sensor;
the pose information comprises position information and pose information;
the determining pose information of the AR glasses relative to the target aircraft according to the spatial position information and the standard point cloud map of the target aircraft based on the synchronous positioning and mapping SLAM algorithm comprises the following steps:
based on SLAM algorithm, searching in the standard point cloud map by adopting the space image information, and determining the position information of the AR glasses;
pose information of the AR glasses is determined based on the inertial measurement data.
2. The method of claim 1, further comprising:
determining the position of the patrol personnel in the current environment and the head orientation of the patrol personnel based on the pose information and the standard point cloud map;
based on the position and the head orientation of the patrol personnel, displaying a virtual object corresponding to the current patrol target through the AR glasses, wherein the virtual object represents patrol information corresponding to the current patrol target.
3. The method according to claim 2, wherein the displaying, by the AR glasses, the virtual object corresponding to the current inspection target based on the position and the head orientation of the inspector, includes:
Determining a current inspection target of the inspection personnel according to the position and the head orientation of the inspection personnel;
determining a target patrol block from a plurality of patrol blocks in the standard point cloud map based on the current patrol target, wherein the standard point cloud map is divided into a plurality of patrol blocks in advance;
and displaying the virtual object corresponding to the target patrol block through the AR glasses.
4. The method of outboard of claim 2, further comprising:
displaying a user main interface to the patrol personnel through the AR glasses, and setting at least one of the following based on an operation control instruction of the patrol personnel: the virtual object, the virtual patrol map and the user main interface corresponding to the current patrol target.
5. The method of outboard of claim 2, further comprising:
displaying a virtual inspection list to the inspector through the AR glasses;
and marking the check point in the virtual check list according to the check marking instruction of the patrol personnel.
6. The method of outboard of claim 2, further comprising:
Generating a pose information record file after receiving an inspection starting instruction of the inspection personnel;
when the patrol personnel patrol, writing pose information corresponding to the patrol personnel into the pose information record file;
reading a target inspection file according to a historical inquiry instruction of an inquirer, and dynamically displaying a target inspection motion track on the virtual inspection map according to the target inspection file, wherein the target inspection file comprises at least one pose information record file or comprises a standard track record file and at least one pose information record file;
under the condition that the target inspection file comprises at least one pose information record file, the model numbers of target aircrafts corresponding to the pose information record files are consistent; and under the condition that the target inspection file comprises the standard track record file and at least one pose information record file, the model numbers of the target aircraft corresponding to the pose information record file and the standard track record file are consistent.
7. An off-board inspection device for an aircraft, comprising:
the display module is used for displaying a virtual inspection map to the inspector through the augmented reality AR glasses worn by the inspector after receiving an inspection starting instruction of the inspector, wherein the virtual inspection map comprises a standard point cloud map of an inspected target aircraft;
The acquisition module is used for acquiring the space position information of the AR glasses in the current environment in real time based on the information acquisition component on the AR glasses;
the repositioning module is used for determining pose information of the AR glasses relative to the target aircraft according to the space position information and a standard point cloud map of the target aircraft based on a synchronous positioning and mapping SLAM algorithm; based on the pose information, updating the position of a virtual cursor representing the patrol personnel and the patrol motion trail representing the patrol route of the patrol personnel on the virtual patrol map in real time;
the information acquisition component comprises a multi-view camera and an inertial sensor, and the spatial position information comprises spatial image information and inertial measurement data;
the acquisition module is specifically used for: acquiring the space image information of the AR glasses in the current environment based on the multi-view camera; acquiring the inertial measurement data of the AR glasses based on the inertial sensor;
the pose information comprises position information and pose information;
the repositioning module is specifically configured to: based on SLAM algorithm, searching in the standard point cloud map by adopting the space image information, and determining the position information of the AR glasses; pose information of the AR glasses is determined based on the inertial measurement data.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the aircraft outboard inspection method of any one of claims 1 to 6 when the program is executed by the processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311445734.6A CN117213468B (en) | 2023-11-02 | 2023-11-02 | Method and device for inspecting outside of airplane and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311445734.6A CN117213468B (en) | 2023-11-02 | 2023-11-02 | Method and device for inspecting outside of airplane and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117213468A CN117213468A (en) | 2023-12-12 |
| CN117213468B true CN117213468B (en) | 2024-04-05 |
Family
ID=89039244
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311445734.6A Active CN117213468B (en) | 2023-11-02 | 2023-11-02 | Method and device for inspecting outside of airplane and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117213468B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113222184A (en) * | 2021-01-29 | 2021-08-06 | 国网辽宁省电力有限公司大连供电公司 | Equipment inspection system and method based on augmented reality AR |
| WO2022021739A1 (en) * | 2020-07-30 | 2022-02-03 | 国网智能科技股份有限公司 | Humanoid inspection operation method and system for semantic intelligent substation robot |
| CN114636422A (en) * | 2021-11-26 | 2022-06-17 | 广西电网有限责任公司 | Positioning and navigation method for information machine room scene |
| CN115469744A (en) * | 2022-09-09 | 2022-12-13 | 南方电网调峰调频发电有限公司信息通信分公司 | Machine room equipment inspection operation method based on AR technology |
| CN115661966A (en) * | 2022-09-16 | 2023-01-31 | 武汉烽火技术服务有限公司 | Inspection system and method based on augmented reality |
| CN116824102A (en) * | 2023-07-05 | 2023-09-29 | 中国南方电网有限责任公司超高压输电公司贵阳局 | AR equipment-based intelligent equipment inspection method |
| CN116843867A (en) * | 2023-05-15 | 2023-10-03 | 北京航空航天大学 | Augmented reality virtual-real fusion method, electronic device and storage medium |
| CN116911823A (en) * | 2023-07-05 | 2023-10-20 | 中国南方电网有限责任公司超高压输电公司贵阳局 | AR (augmented reality) glasses-based substation inspection interaction method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9463574B2 (en) * | 2012-03-01 | 2016-10-11 | Irobot Corporation | Mobile inspection robot |
-
2023
- 2023-11-02 CN CN202311445734.6A patent/CN117213468B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022021739A1 (en) * | 2020-07-30 | 2022-02-03 | 国网智能科技股份有限公司 | Humanoid inspection operation method and system for semantic intelligent substation robot |
| CN113222184A (en) * | 2021-01-29 | 2021-08-06 | 国网辽宁省电力有限公司大连供电公司 | Equipment inspection system and method based on augmented reality AR |
| CN114636422A (en) * | 2021-11-26 | 2022-06-17 | 广西电网有限责任公司 | Positioning and navigation method for information machine room scene |
| CN115469744A (en) * | 2022-09-09 | 2022-12-13 | 南方电网调峰调频发电有限公司信息通信分公司 | Machine room equipment inspection operation method based on AR technology |
| CN115661966A (en) * | 2022-09-16 | 2023-01-31 | 武汉烽火技术服务有限公司 | Inspection system and method based on augmented reality |
| CN116843867A (en) * | 2023-05-15 | 2023-10-03 | 北京航空航天大学 | Augmented reality virtual-real fusion method, electronic device and storage medium |
| CN116824102A (en) * | 2023-07-05 | 2023-09-29 | 中国南方电网有限责任公司超高压输电公司贵阳局 | AR equipment-based intelligent equipment inspection method |
| CN116911823A (en) * | 2023-07-05 | 2023-10-20 | 中国南方电网有限责任公司超高压输电公司贵阳局 | AR (augmented reality) glasses-based substation inspection interaction method and system |
Non-Patent Citations (1)
| Title |
|---|
| 基于头戴式双目相机的智能变电站巡检巡视系统设计;刘广振等;《计算机测量与控制》(第2期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117213468A (en) | 2023-12-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20150062123A1 (en) | Augmented reality (ar) annotation computer system and computer-readable medium and method for creating an annotated 3d graphics model | |
| Dini et al. | Application of augmented reality techniques in through-life engineering services | |
| CN103456025B (en) | The positioning of sensor enhancing in virtual and actual environment | |
| US20180300551A1 (en) | Identifying a Position of a Marker in an Environment | |
| CN106662925B (en) | Multi-user gaze projection using head mounted display devices | |
| Zollmann et al. | Augmented reality for construction site monitoring and documentation | |
| CN105393284B (en) | Space engraving based on human body data | |
| EP2208021B1 (en) | Method of and arrangement for mapping range sensor data on image sensor data | |
| US8731276B2 (en) | Motion space presentation device and motion space presentation method | |
| US9235933B2 (en) | Wearable display system that displays previous runners as virtual objects on a current runner's path | |
| EP3892196A1 (en) | Improved registration for vehicular augmented reality using auto-harmonization | |
| US20170168306A1 (en) | Device and method for testing function or use of a head worn see through augmented reality device | |
| JP2019087241A (en) | Systems, methods and tools for spatially registering virtual content with physical environment in augmented reality platforms | |
| JP2021524014A (en) | Computerized inspection system and method | |
| EP3938870B1 (en) | Fixed holograms in mobile environments | |
| CN104049742A (en) | Three dimensional fingertip tracking | |
| CN105701261A (en) | Near-field aircraft automatic tracking and monitoring method | |
| CN110969687A (en) | Collision detection method, device, equipment and medium | |
| JP2020160944A (en) | Inspection work support device, inspection work support method, and inspection work support program | |
| CN103679792A (en) | Rendering method and system for three-dimensional models | |
| JP6061334B2 (en) | AR system using optical see-through HMD | |
| JP6345381B2 (en) | Augmented reality system | |
| CN117213468B (en) | Method and device for inspecting outside of airplane and electronic equipment | |
| CN115357500A (en) | Test method, device, equipment and medium for automatic driving system | |
| Waegel et al. | Filling the gaps: Hybrid vision and inertial tracking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |