CN114567535B - Product interaction and fault diagnosis method based on augmented reality - Google Patents
Product interaction and fault diagnosis method based on augmented reality Download PDFInfo
- Publication number
- CN114567535B CN114567535B CN202210231893.5A CN202210231893A CN114567535B CN 114567535 B CN114567535 B CN 114567535B CN 202210231893 A CN202210231893 A CN 202210231893A CN 114567535 B CN114567535 B CN 114567535B
- Authority
- CN
- China
- Prior art keywords
- information
- user
- product
- interaction
- platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000003993 interaction Effects 0.000 title claims abstract description 63
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000003745 diagnosis Methods 0.000 title claims abstract description 36
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 33
- 230000002452 interceptive effect Effects 0.000 claims abstract description 50
- 238000004891 communication Methods 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 7
- 239000011521 glass Substances 0.000 claims description 21
- 230000002093 peripheral effect Effects 0.000 claims description 10
- 238000001514 detection method Methods 0.000 claims description 8
- 230000005540 biological transmission Effects 0.000 claims description 7
- 230000000007 visual effect Effects 0.000 claims description 7
- 210000000988 bone and bone Anatomy 0.000 claims description 5
- 238000012216 screening Methods 0.000 claims description 4
- 241001270131 Agaricus moelleri Species 0.000 claims description 3
- 238000005259 measurement Methods 0.000 claims description 3
- 238000007781 pre-processing Methods 0.000 claims description 2
- 238000012423 maintenance Methods 0.000 abstract description 10
- 230000006870 function Effects 0.000 description 12
- 230000008569 process Effects 0.000 description 11
- 238000013473 artificial intelligence Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000007689 inspection Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 239000000969 carrier Substances 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000009529 body temperature measurement Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000000523 sample Substances 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Processing Or Creating Images (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a product interaction and fault diagnosis method based on augmented reality, which comprises the steps of establishing an interaction platform capable of watching a real scene and virtual information display; electronically converting information which corresponds to a product to be interacted and is required to be transmitted to a user into an information source; and label processing of keyword and image unitization is carried out for realizing quick search; pre-storing the information source and the information processed by the tag in an interactive platform or in a network or local storage device, and establishing a link channel capable of loading data information in real time through a communication medium and the interactive platform; when the demand instruction of the user is detected, the information data which is most suitable for the current scene is automatically replied and simultaneously called out and displayed to the user on the interaction platform so as to improve the interaction efficiency. The scheme has the advantages of freeing both hands, being more efficient, combining virtual and real, accurately reflecting equipment information, and being capable of carrying out fault diagnosis and recognition, guidance and maintenance on products.
Description
Technical Field
The invention belongs to the technical field of man-machine interaction, relates to an intelligent interaction technology for large equipment or complex machines, and particularly relates to a product interaction and fault diagnosis method based on augmented reality, which can enable a user to quickly master the functions of a product, use the product, disassemble, assemble and repair skills and quickly judge and lock the working state and fault point of the product.
Background
In the prior art, large equipment or machines are shipped with detailed specifications for users to understand and use, such as engine manuals, car manuals, precision lathe manuals, and the like. The corresponding machine operators, maintenance workers, secondary design developers and other users need to know the relevant content in the manuals to perform operation and use or maintenance design on the equipment. If the machine is not known, the machine is directly operated and used or maintained by hands, so that the damage to the machine is easily caused, and the functions and the working efficiency of the machine are affected.
However, the inventors of the present patent application found that detailed manuals for equipment delivery, including product specifications, service manuals, use specifications, user manuals, trouble-shooting instructions, disassembly manuals, and the like, were prepared in the prior art. Most of these manuals are paper, and the user wants to quickly find the related content in the manual is very inconvenient to operate. In order to solve the problem, some manufacturers can be provided with electronic documents on the basis of paper manuals or a special electronic tablet device so that users can quickly read corresponding product manuals. Although the searching efficiency is improved through the operation, the cup and the waterwheel salary are still available for large equipment in the practical operation level, and the improvement is not obvious. First, the large-scale equipment manuals are various, and users cannot completely and effectively find the content needed at the time in a short time. Secondly, through the text and two-dimensional picture description provided by the manual, the reading and understanding of the literal meaning are inefficient, and the incorrect operation is likely to be caused by the literal understanding error. Moreover, the manual is not separated from both hands, needs to be turned over by both hands, even a plurality of persons cooperate (such as one person turns over the record, one person controls the machine operation) in a narrow space, the contradiction is more prominent, and tools, manuals and the like can be placed in no place; learning time, cost and efficiency are all intolerable.
The inventor of the patent application finds through searching that, aiming at the problems existing in the solution, a solution for introducing the intelligent glasses is further proposed, namely, the equipped electronic manual or the tablet personal computer is updated into the intelligent glasses, so that a user can directly look up related information through the intelligent glasses. But this solution is still a simple stage of importing product information into existing commercial products. That is, although the consulting efficiency is further improved, the interaction efficiency is still low, and the real scene cannot be identified and more support is provided by combining the real scene of the product; the fault diagnosis and identification of the product can not be performed, and the self-service maintenance of the user can be guided. Meanwhile, the related function of carrying out safety early warning on the user is lacking, namely the user cannot be ensured not to be damaged by collision and the like between the product and other parts around the product when interacting or diagnosing with the product.
In addition, the inventors of the present patent application also found by searching that the patent application number is: 202110337423.2, patent name: an assembly guiding method, system and application based on augmented reality are disclosed; the patent application discloses an assembly guiding method, system and application based on augmented reality, wherein the assembly guiding method comprises the following steps: analyzing the contour information and the configuration information of the part model into structural data; identifying and capturing pose information of all parts, providing a three-dimensional tracking registration matrix of the parts for an AR scene, dynamically tracking and positioning the parts in a real scene, and constructing an augmented reality scene; extracting three-dimensional edge contour characteristic information of a part in real time, and distinguishing a basic part and other parts of the to-be-assembled body in the current state; highlighting the guiding information at the position to be assembled of the base member; carrying out envelope highlighting identification on the part to be assembled; assembling, dynamically identifying and judging the assembling condition by visual recognition to realize virtual-real fusion; repeating the steps, and finishing the installation of all parts. The inventors of the present patent application have found by comparison that this patent application discloses a solution for rapid assembly guidance by augmented reality technology. But the description of the use of the product, fault diagnosis, user safety precaution and the like are not related.
By further searching, the inventor of the present patent application found that the patent application number is: 202011420025.9, patent name: an invention patent application of a system for real-time control and visualization of a digital factory in an augmented reality environment; the patent application discloses a system and a device for controlling and visualizing a digital factory in real time in an augmented reality environment, at least comprising an augmented reality device, a server system and a plurality of controlled objects, wherein the server system at least comprises a two-dimensional module and a three-dimensional module, and the two-dimensional module displays a two-dimensional graphic component of the controlled objects in a two-dimensional mode; the three-dimensional module displays a three-dimensional graphic component of the controlled object in a three-dimensional mode; the attribute data of the controlled object can be displayed in parallel in a two-dimensional graph and a three-dimensional view mode in response to the operation instruction of the augmented reality device, and the two-dimensional module and/or the three-dimensional module can control the controlled object based on the operation instruction fed back by the augmented reality device. The inventor of the patent application finds through comparison that the patent application discloses a scheme of carrying out high-efficiency inspection through an augmented reality technology, and meanwhile, the scheme of carrying out prejudgment on local faults of a factory, assisting workers in the inspection route planning of bypassing crisis and the like are mentioned. The system is mainly used for inspection, namely, the system is mainly used for passively receiving inspection information, and is not used for efficient interaction.
In summary, a virtual-real combination is still lacking at present, the interaction mode is more efficient, fault diagnosis and identification can be performed on the product, self-service maintenance is guided for a user, and meanwhile, the safety of the user is still guaranteed to the greatest extent in the interaction and diagnosis process.
Disclosure of Invention
The invention aims to solve the technical problem of providing the product interaction and fault diagnosis method based on the augmented reality, which combines virtual and real, has more efficient interaction mode, can carry out fault diagnosis and identification on the product, guides a user to maintain by self-service, and simultaneously ensures the safety of the user to the greatest extent in the interaction and diagnosis process.
In order to solve the technical problems, the technical scheme provided by the invention is as follows: a product interaction and fault diagnosis method based on augmented reality comprises the following steps:
firstly, establishing an interaction platform taking intelligent wearable equipment which enables a user to watch a real scene and virtual information as a carrier; the interactive platform is connected with a camera, a microphone, a sound transmission device, a temperature sensor, a flaw detection sensor, a distance measurement sensor and other functional sensors for detecting the state of the product; the interaction platform is provided with a safety early warning system for detecting whether a user is threatened by safety when interacting with a product and diagnosing faults so as to ensure personal safety;
secondly, electronically converting information which corresponds to the product to be interacted and is required to be transmitted to a user into an information source; the information source comprises the use instruction information and fault diagnosis information of the product, and the forms comprise characters, voice, pictures and videos;
thirdly, carrying out label processing of keywords and image unitization on the information source in the second step for the purpose of realizing quick retrieval; meanwhile, pre-storing the information source and the information processed by the tag into an interaction platform; or pre-stored in a network or a local storage device, and establishes a link channel with the interactive platform, wherein the link channel can load data information in real time;
the fourth step, the control interaction platform acquires data information of a product in the current state, instruction information of a user and safety information around the environment where the user is located, wherein the data information, the instruction information of the user and the safety information are needed to be interacted in the current state of the product in front of the user in real time through all the carried visual angle cameras and sensors;
fifthly, after the instruction information of the user is detected, the interactive platform matches the user instruction, the acquired input source data and the stored information source data, and then the information closest to the current scene is called out and displayed to the user on the interactive platform, wherein the display content comprises but is not limited to characters, voice, images, videos and animations; if a plurality of similar information results exist, displaying a result profile list according to the weight, informing a user to select information, and then screening and displaying the information according to a selection instruction of the user; the user can watch the real scene of the product in front of eyes while watching the virtual information; when the safety early warning system of the interactive platform detects the safety threat, an early warning signal is immediately sent to the user, and the danger avoiding scheme is prompted.
Preferably, the interactive platform is provided with a multi-machine online communication function module, so that a multi-machine sharing view angle and shared data can be realized, and the communication mode comprises a physical wired connection mode and a wireless network connection mode; the functional sensors of the interactive platform comprise various sensors which can collect characteristic data of product images, gray scales, infrared rays and heat energy.
Preferably, the sound transmission system of the interactive platform comprises a loudspeaker, an earphone and a bone conduction microphone.
Preferably, the interactive platform is provided with a 360-degree scene camera for acquiring real-time images around the user.
Preferably, the intelligent wearable device comprises AR glasses.
Preferably, the interactive platform can directly communicate with the product to be interacted with and communicate control signals.
Preferably, the interactive platform can be connected with other intelligent wearable devices at the same time.
Preferably, the information source in the second step is updated online in real time.
Preferably, in the fifth step, when the information is displayed to the user on the interactive platform, after the information is preprocessed or the post-image is intelligently processed, the information display position is combined with the corresponding part to be described or maintained in the real product.
As a further preferred option, the interactive platform can detect the safety condition of the working environment around the platform in real time, synchronously combine the collected information of other peripheral sensors to judge whether the user is in a safe working environment, and provide an alarm signal or send out a control command for stopping peripheral dangerous equipment in an emergency.
After the scheme is adopted, the invention has the following beneficial effects: the augmented reality AR and artificial intelligence AI technology are utilized, virtual reality is combined, and in the aspects of interactive use and learning of products, a user can be helped to master and use large-scale equipment efficiently, the learning process is shortened, and the use efficiency is improved. The fault diagnosis process is simpler and more efficient in terms of fault diagnosis. The intelligent wearable device is used as a carrier, and hands are thoroughly liberated through various interaction means such as voice, gesture control and the like. All documents and materials related to the device are electronic and digitally virtualized. The search result can be searched and found on line and displayed in real time. The user can operate against the object by projecting the maintenance process of the three-dimensional display device component.
Various expansion functions are added in the preference, such as a multi-machine online communication function, a multi-machine shared visual angle camera can be realized, and the user can directly communicate with a product manufacturer or a maintenance expert in a connecting way. The bone conduction microphone is adopted in the microphone system, so that mutual interference of peripheral sounds can be reduced, and the situation that a user cannot hear alarm sounds of equipment or sounds among peripheral colleagues is avoided. 360 degree scene camera, collision early warning radar system, interactive platform can directly communicate and convey control signal with the product that needs to carry out the interaction, connects extension structures such as other intelligent wearable equipment and control method simultaneously, all is based on the extension of production safety, lets the user immerse in the interaction, the safety of the maximum guarantee user. Meanwhile, the interaction platform can directly communicate and control the interacted products, so that the interaction efficiency is further improved, the interaction platform is not only remained in learning and maintaining the products, but also can directly control the products to perform corresponding feedback or demonstrate corresponding states.
In summary, the present application provides an augmented reality-based product interaction and fault diagnosis method that can free the hands of a user, provide a more efficient interaction mode, combine virtual and real, accurately reflect equipment information, and perform fault diagnosis and identification on a product, and guide self-service maintenance of the user.
Drawings
Fig. 1 is a block schematic diagram of a method involved in the present invention when interacting with an augmented reality based product.
Fig. 2 is a block schematic diagram of a method for augmented reality-based product fault diagnosis in accordance with the present invention.
Fig. 3 is a schematic diagram of core modules when the product interaction and fault diagnosis method based on augmented reality is implemented.
Fig. 4 is a schematic diagram of the workflow of the core module of fig. 3.
Fig. 5 is a virtual information presentation example under a fault diagnosis embodiment.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
The invention discloses a product interaction and fault diagnosis method based on augmented reality, which comprises the following steps:
firstly, establishing an interaction platform taking intelligent wearable equipment which enables a user to watch a real scene and virtual information as a carrier; the interactive platform is connected with a camera, a microphone, a sound transmission device, a temperature sensor, a flaw detection sensor, a distance measurement sensor and other functional sensors for detecting the state of the product; the interaction platform is provided with a safety early warning system for detecting whether a user is threatened by safety when interacting with a product and diagnosing faults so as to ensure personal safety;
secondly, electronically converting information which corresponds to the product to be interacted and is required to be transmitted to a user into an information source; the information source comprises the use instruction information and fault diagnosis information of the product, and the forms comprise characters, voice, pictures and videos;
thirdly, carrying out label processing of keywords and image unitization on the information source in the second step for the purpose of realizing quick retrieval; meanwhile, pre-storing the information source and the information processed by the tag into an interaction platform; or pre-stored in a network or a local storage device, and establishes a link channel with the interactive platform, wherein the link channel can load data information in real time;
the fourth step, the control interaction platform acquires data information of a product in the current state, instruction information of a user and safety information around the environment where the user is located, wherein the data information, the instruction information of the user and the safety information are needed to be interacted in the current state of the product in front of the user in real time through all the carried visual angle cameras and sensors;
fifthly, after the instruction information of the user is detected, the interactive platform matches the user instruction, the acquired input source data and the stored information source data, and then the information closest to the current scene is called out and displayed to the user on the interactive platform, wherein the display content comprises but is not limited to characters, voice, images, videos and animations; if a plurality of similar information results exist, displaying a result profile list according to the weight, informing a user to select information, and then screening and displaying the information according to a selection instruction of the user; the user can watch the real scene of the product in front of eyes while watching the virtual information; when the safety early warning system of the interactive platform detects the safety threat, an early warning signal is immediately sent to the user, and the danger avoiding scheme is prompted.
Preferably, the interactive platform is provided with a multi-machine online communication function module, so that a multi-machine sharing view angle and shared data can be realized, and the communication mode comprises a physical wired connection mode and a wireless network connection mode; the functional sensors of the interactive platform comprise various sensors which can collect characteristic data of product images, gray scales, infrared rays and heat energy.
Preferably, the sound transmission system of the interactive platform comprises a loudspeaker, an earphone and a bone conduction microphone.
Preferably, the interactive platform is provided with a 360-degree scene camera for acquiring real-time images around the user.
Preferably, the intelligent wearable device comprises AR glasses.
Preferably, the interactive platform can directly communicate with the product to be interacted with and communicate control signals.
Preferably, the interactive platform can be connected with other intelligent wearable devices at the same time.
Preferably, the information source in the second step is updated online in real time.
Preferably, in the fifth step, when the information is displayed to the user on the interactive platform, after the information is preprocessed or the post-image is intelligently processed, the information display position is combined with the corresponding part to be described or maintained in the real product.
As a further preferred option, the interactive platform can detect the safety condition of the working environment around the platform in real time, synchronously combine the collected information of other peripheral sensors to judge whether the user is in a safe working environment, and provide an alarm signal or send out a control command for stopping peripheral dangerous equipment in an emergency.
In the above preferred embodiments, the interactive platform has a multi-camera online communication function, so that a multi-camera sharing view angle camera can be realized. The sound transmission system of the interactive platform adopts a bone conduction microphone. The interactive platform is provided with a 360-degree scene camera and is used for acquiring real-time pictures around a user. The interactive platform is provided with a collision early warning radar system. The interactive platform can directly communicate with the product to be interacted with and communicate control signals. The interaction platform can be connected with other intelligent wearable devices at the same time. The information source can be updated online in real time. When the information is displayed to a user through virtual projection on the interactive platform, the projected position is combined with a corresponding area which needs to be described or maintained in a real product after information preprocessing or post-image intelligent processing; while the projection area may be selected by the user himself. When the interaction platform performs information interaction, the collected peripheral information of the user can be synchronously combined to judge whether the user is in a safe working environment, and an alarm signal is provided or a control command for emergently stopping peripheral dangerous equipment is sent.
Several usage scenarios corresponding to the present application will be described in detail below in the form of specific embodiments.
Detailed description of the preferred embodiments
Further referring to fig. 1, basic product interaction functions are taken as an example; the interaction platform adopts intelligent glasses; the product is a multifunctional device with a plurality of robot devices. When a user inputs a certain keyword (voice or soft keyboard) to the intelligent glasses, the keyword is definitely a part of the current product, the intelligent glasses acquire the current camera scene, the computing unit performs recognition detection by using the prestored part model, and the detection result is identified in the scene through the frame body. If a plurality of frame results in the scene are detected, a user is informed to input a screening instruction, and after the user selects and determines a certain frame, the acquisition unit searches and retrieves information of a corresponding part and displays the information in a preset scene area 1. As a further improvement, if the frame body results are too many, a weighting algorithm is added, and intelligent sorting is performed first, namely, the interaction platform is used for intelligent sorting by combining the visual angle information of the user, the operation habit information of the user and the like, and the most likely results are arranged at the front position.
For example, in a voice mode, the user speaks "robot", and the host computer calls out the contents of the section "robot" of the user manual and displays the contents on the screen. The gesture mode is to click the search bar, pop up the soft keyboard, input "robot" in Chinese, similarly, the host computer will call out the content of a section of user manual "robot", and put in and display in the preset area 1. And next, the user can input a keyword command of a certain operation by voice, and the acquisition unit of the interactive platform detects that the part is preset with an operation model action or video, namely, the operation animation of the three-dimensional virtual model of the part or the video of the operation of the part can be put in and displayed in the preset area 2. The user can learn cognitively against the information data output from the acquisition unit and operate the actual device against the played three-dimensional animation or video. Wherein the calculation unit: the real-time video detection device is used for detecting real-time video of the camera, calculating the position and the size of the detected part position, and marking the frame in a real scene. The computing unit may be hardware such as a CPU, GPU, etc., and a software algorithm module running thereon. The unit is generally carried on the intelligent glasses, and a module with better performance can be designed by the user. Wherein the acquisition unit: the method is used for acquiring corresponding data information of the designated frame body component, a three-dimensional virtual model, operation animation, video and the like. The acquisition unit may be hardware such as a memory and a sensor, or may be a software module such as search calculation. The display unit is used for outputting the content of the acquisition unit in a specific form in a preset display area, and the unit is generally carried on the intelligent glasses and can also be used for designing a module with better performance by itself.
Through practice, the embodiment utilizes the augmented reality AR and artificial intelligence AI technology, and virtual reality is combined, so that a user can be helped to master and use large-scale equipment efficiently, the learning process is shortened, and the use efficiency is improved. Glasses are used as carriers, and hands are thoroughly liberated through various interaction means such as voice, gesture control and the like; all documents and materials related to the device are digitized and electronic. The search result can be searched and found on line and displayed in real time. The device components can be detected and identified, and the device attributes and data can be guided to be familiar to a user through a virtual frame, a virtual three-dimensional model, characters, pictures, videos and the like to be displayed in a specific area. The disassembly and assembly process of the three-dimensional immersive display equipment parts can be performed, and a user can operate according to the real objects.
Second embodiment
Further referring to fig. 2, a basic product failure diagnosis function is taken as an example; the interaction platform adopts intelligent glasses; the product is a multifunctional device with a plurality of robot devices. The fault of each part of the equipment is possible, the typical faults can be judged through the appearance, such as a switch, a signal lamp display, the relative positions of parts and the like, and the model can be stored through data processing and feature extraction in advance. When the glasses are in the diagnosis mode, videos are acquired in real time through the camera, the calculation unit calculates and compares the videos with a pre-stored model, and the positions and the ranges of the fault areas are acquired in real time and marked. The acquisition unit displays information for finding the corresponding failure in a predetermined area 1, and displays a corresponding maintenance video or virtual maintenance animation in a predetermined area 2. Wherein the calculation unit: the real-time video detection device is used for detecting real-time video of the camera, calculating the position and the size of the detected part position, and marking the frame in a real scene. The computing unit may be hardware such as a CPU, GPU, etc., and a software algorithm module running thereon. The unit is generally carried on the intelligent glasses, and a module with better performance can be designed by the user. Wherein the acquisition unit: the method is used for acquiring corresponding fault data information of the designated frame body component, such as fault text description, solution, image description and the like. The display unit is used for outputting the content of the acquisition unit in a specific form in a preset display area, and the unit is generally carried on the intelligent glasses and can also be used for designing a module with better performance by itself.
Practice proves that the embodiment utilizes the augmented reality AR and artificial intelligence AI technology, and virtual reality is combined, so that the fault diagnosis process is simpler and more efficient. Glasses are used as carriers, and hands are thoroughly liberated through various interaction means such as voice, gesture control and the like; all documents and materials related to the device are digitized and electronic. The search result can be searched and found on line and displayed in real time. The maintenance process of the equipment parts can be displayed in three dimensions, and a user can operate the equipment parts against the real objects.
Detailed description of the preferred embodiments
Further, referring to fig. 3 to fig. 5, with AR glasses as a main body interaction device, cameras are provided on the glasses to collect video of a current scene, and a display unit superimposes and displays the content to be displayed on a physical environment according to a specified mode (position coordinates, size and color), and a calculation unit and a storage unit are connected with the glasses main body in a wired or wireless mode. The AR glasses camera transmits the current video scene to a computing unit, the computing unit detects that two POI (point of interest) parts exist in the visual field range of a user through an AI (artificial intelligence) algorithm, and the display unit draws a mark frame according to the position and the size of the parts given by the computing unit. The operator selects one of the diagnostic target sites in the form of a pair of head rays, a pair of handle rays, eye tracking, and gestures. The calculating unit reads the target scene diagnosis information pre-stored in the storage unit and carries out the next step. For example, the computing unit reads the infrared temperature measurement, finds that the temperature is too high, the voice text prompt indicates that the equipment temperature is 48 degrees and the alarm temperature is 50 degrees, and simultaneously the computing unit detects that the fuse is reversed through the current panel image, marks a highlight frame on the fuse position, prompts an arrow, and the voice text prompt indicates that the fuse is reversed, and the equipment power is turned off to re-blow the insurance. And displaying the insurance three-dimensional model animation at the highlight frame side.
Regarding the aspect of user safety guarantee, in the implementation process, the three embodiments can detect the safety state of the environment where the user is located in real time through the interaction platform, including whether a movable object collides with the user, whether there is abnormal information that the parts on the product burst high temperature and high pressure or leak, or whether the user is overstressed due to the impact of virtual reality information or the sudden situation that the user is in a motion due to physical reasons, and the safety early warning system can detect and respond correspondingly to ensure the safety of the user. The security system of the interactive platform herein focuses on wearable devices such as a hand ring that is directly mounted on the platform itself, e.g., a smart helmet, or is directly signal-connected to the platform. The corresponding safety system comprises a mini radar probe similar to a vehicle-mounted anti-collision system and is used for detecting the distance change condition between the surrounding obstacles of the user and the body of the user, and if an object is detected to be fast close to the body of the user, an alarm is timely sent out to prompt the user to carry out emergency danger avoidance. The safety system can also comprise a bracelet device, so that the physical condition of a user can be detected in real time, and unmanned discovery is prevented when sudden situations such as too fast heart rate, faint and the like occur in some more stimulated augmented reality environments. Under the further improvement of the safety system, the safety system can also realize linkage control with products and equipment around the products, such as the emergency braking function of the products and the power supply system around the products; when a user is involved in a product gear under the influence of augmented reality in the interaction process with the product, or an electric shock accident occurs, a safety detection sensor worn by the user such as a bracelet, an intelligent camera and the like can timely detect the electric shock accident, or can timely receive dangerous information through a voice instruction sent by the user, and the operation of emergency braking or emergency power failure, starting fire alarm and the like is realized through linkage control, so that the safety guarantee is realized at the highest speed.
When the application is specifically implemented, the structural design, the hardware circuit, the chip and the control program and the algorithm of the specific product are different due to the corresponding interaction requirements of different products, and the production party is required to combine specific product characteristics and function requirements, and the application is operated under the guidance of the interaction method provided by the application. After knowing the product interaction and fault diagnosis method in the application, the technical obstacles on the implementation of the method level of a set of hardware structure and a specific algorithm are designed according to the method by those skilled in the industry, and the detailed description is omitted here.
The invention and its embodiments have been described above with no limitation, and the actual construction is not limited to the embodiments of the invention as shown in the drawings. In summary, if one of ordinary skill in the art is informed by this disclosure, a structural manner and an embodiment similar to the technical solution should not be creatively devised without departing from the gist of the present invention.
Claims (7)
1. A product interaction and fault diagnosis method based on augmented reality is characterized in that: it comprises the following steps:
firstly, establishing an interaction platform taking intelligent wearable equipment which enables a user to watch a real scene and virtual information as a carrier; the interactive platform is connected with a camera, a microphone, a sound transmission device, a temperature sensor, a flaw detection sensor, a distance measurement sensor and other functional sensors for detecting the state of the product; the interaction platform is provided with a safety early warning system for detecting whether a user is threatened by safety when interacting with a product and diagnosing faults so as to ensure personal safety;
secondly, electronically converting information which corresponds to the product to be interacted and is required to be transmitted to a user into an information source; the information source comprises the use instruction information and fault diagnosis information of the product, and the forms comprise characters, voice, pictures and videos;
thirdly, carrying out label processing of keywords and image unitization on the information source in the second step for the purpose of realizing quick retrieval; meanwhile, pre-storing the information source and the information processed by the tag into an interaction platform; or pre-stored in a network or a local storage device, and establishes a link channel with the interactive platform, wherein the link channel can load data information in real time;
the fourth step, the control interaction platform acquires data information of a product in the current state, instruction information of a user and safety information around the environment where the user is located, wherein the data information, the instruction information of the user and the safety information are needed to be interacted in the current state of the product in front of the user in real time through all the carried visual angle cameras and sensors;
fifthly, after the instruction information of the user is detected, the interactive platform matches the user instruction, the acquired input source data and the stored information source data, and then the information closest to the current scene is called out and displayed to the user on the interactive platform, wherein the display content comprises but is not limited to characters, voice, images, videos and animations; if a plurality of similar information results exist, displaying a result profile list according to the weight, informing a user to select information, and then screening and displaying the information according to a selection instruction of the user; the user can watch the real scene of the product in front of eyes while watching the virtual information; when the safety early warning system of the interactive platform detects the safety threat, an early warning signal is immediately sent to a user, and a risk avoiding scheme is prompted;
the interactive platform is provided with a multi-machine online communication function module, can realize multi-machine sharing view angle and sharing data, and the communication mode comprises a physical wired connection mode and a wireless network connection mode; the functional sensors of the interactive platform comprise various sensors which can collect characteristic data of product images, gray scales, infrared rays and heat energy;
in the fifth step, when information is displayed to a user on the interactive platform, after information preprocessing or post-image intelligent processing, the information display position is combined with a part of a real product, which is corresponding to the part to be described or maintained;
when the interactive platform performs information interaction, the safety conditions of the working environment around the platform can be detected in real time, the collected information of other peripheral sensors can be synchronously combined, whether a user is in a safe working environment is judged, and an alarm signal is provided or a control command for emergently stopping peripheral dangerous equipment is sent.
2. The augmented reality-based product interaction and fault diagnosis method according to claim 1, wherein: the sound transmission system of the interactive platform comprises a loudspeaker, an earphone and a bone conduction microphone.
3. The augmented reality-based product interaction and fault diagnosis method according to claim 1, wherein: the interactive platform is provided with a 360-degree scene camera and is used for acquiring real-time pictures around a user.
4. The augmented reality-based product interaction and fault diagnosis method according to claim 1, wherein: the intelligent wearable device comprises AR glasses.
5. The augmented reality-based product interaction and fault diagnosis method according to claim 1, wherein: the interactive platform can directly communicate with the product to be interacted with and communicate control signals.
6. The augmented reality-based product interaction and fault diagnosis method according to claim 1, wherein: the interaction platform can be connected with other intelligent wearable devices at the same time.
7. The augmented reality-based product interaction and fault diagnosis method according to claim 1, wherein: the information source in the second step can be updated online in real time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210231893.5A CN114567535B (en) | 2022-03-10 | 2022-03-10 | Product interaction and fault diagnosis method based on augmented reality |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210231893.5A CN114567535B (en) | 2022-03-10 | 2022-03-10 | Product interaction and fault diagnosis method based on augmented reality |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114567535A CN114567535A (en) | 2022-05-31 |
| CN114567535B true CN114567535B (en) | 2024-01-09 |
Family
ID=81717645
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210231893.5A Active CN114567535B (en) | 2022-03-10 | 2022-03-10 | Product interaction and fault diagnosis method based on augmented reality |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114567535B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116778119B (en) * | 2023-06-26 | 2024-03-12 | 中国信息通信研究院 | Man-machine cooperative assembly system based on augmented reality |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108879440A (en) * | 2018-06-20 | 2018-11-23 | 国网山东省电力公司济宁供电公司 | Intelligent examination and repair system and method based on wearable augmented reality display terminal and cloud platform |
| CN111638796A (en) * | 2020-06-05 | 2020-09-08 | 浙江商汤科技开发有限公司 | Virtual object display method and device, computer equipment and storage medium |
| CN114116100A (en) * | 2021-11-26 | 2022-03-01 | 内蒙动力机械研究所 | Missile weapon guarantee maintenance system and method based on AR interactive electronic manual |
-
2022
- 2022-03-10 CN CN202210231893.5A patent/CN114567535B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108879440A (en) * | 2018-06-20 | 2018-11-23 | 国网山东省电力公司济宁供电公司 | Intelligent examination and repair system and method based on wearable augmented reality display terminal and cloud platform |
| CN111638796A (en) * | 2020-06-05 | 2020-09-08 | 浙江商汤科技开发有限公司 | Virtual object display method and device, computer equipment and storage medium |
| CN114116100A (en) * | 2021-11-26 | 2022-03-01 | 内蒙动力机械研究所 | Missile weapon guarantee maintenance system and method based on AR interactive electronic manual |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114567535A (en) | 2022-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11100649B2 (en) | Fiducial marker patterns, their automatic detection in images, and applications thereof | |
| JP7442278B2 (en) | Drift correction for industrial augmented reality applications | |
| CN106340217B (en) | Manufacturing equipment intelligence system and its implementation based on augmented reality | |
| CN105182535B (en) | The method that automobile maintenance is carried out using intelligent glasses | |
| EP3422153A1 (en) | System and method for selective scanning on a binocular augmented reality device | |
| GB2607819A (en) | 3D mapping of a process control environment | |
| CN107301377B (en) | Face and pedestrian sensing system based on depth camera | |
| CN113222184A (en) | Equipment inspection system and method based on augmented reality AR | |
| CN110109535A (en) | Augmented reality generation method and device | |
| JP2003140736A (en) | Portable computer in process control environment | |
| US20210224752A1 (en) | Work support system and work support method | |
| US11016559B2 (en) | Display system and display control method of display system | |
| US11816887B2 (en) | Quick activation techniques for industrial augmented reality applications | |
| US20190299412A1 (en) | Augmented Reality System | |
| CN114567535B (en) | Product interaction and fault diagnosis method based on augmented reality | |
| US20220165044A1 (en) | Maintenance support system, maintenance support method, and program | |
| US20210201542A1 (en) | Building maintaining method and system | |
| EP3885972A1 (en) | Context based perception method and system for managing environmental safety in a computing environment | |
| CN110458052A (en) | Recongnition of objects method, apparatus based on augmented reality, equipment, medium | |
| CN208673383U (en) | Repairing auxiliary device and system | |
| CN113989625A (en) | Image identification method, device and equipment based on edge calculation and storage medium | |
| CN113674426A (en) | Equipment maintenance method and device using AR technology | |
| JP7094759B2 (en) | System, information processing method and program | |
| EP4130663A1 (en) | Positioning apparatus and method | |
| KR20200062002A (en) | Shopping aids method and apparatus for the vulnerable group of sight |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |