CN114567535A - Product interaction and fault diagnosis method based on augmented reality - Google Patents
Product interaction and fault diagnosis method based on augmented reality Download PDFInfo
- Publication number
- CN114567535A CN114567535A CN202210231893.5A CN202210231893A CN114567535A CN 114567535 A CN114567535 A CN 114567535A CN 202210231893 A CN202210231893 A CN 202210231893A CN 114567535 A CN114567535 A CN 114567535A
- Authority
- CN
- China
- Prior art keywords
- information
- user
- interaction
- product
- platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000003993 interaction Effects 0.000 title claims abstract description 74
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000003745 diagnosis Methods 0.000 title claims abstract description 38
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 36
- 230000002452 interceptive effect Effects 0.000 claims abstract description 56
- 238000004891 communication Methods 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 9
- 239000011521 glass Substances 0.000 claims description 22
- 230000002093 peripheral effect Effects 0.000 claims description 10
- 230000005540 biological transmission Effects 0.000 claims description 9
- 238000001514 detection method Methods 0.000 claims description 8
- 230000000007 visual effect Effects 0.000 claims description 8
- 210000000988 bone and bone Anatomy 0.000 claims description 5
- 238000012216 screening Methods 0.000 claims description 4
- 241001270131 Agaricus moelleri Species 0.000 claims description 3
- 210000001508 eye Anatomy 0.000 claims description 3
- 238000005259 measurement Methods 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 abstract description 8
- 230000006870 function Effects 0.000 description 11
- 230000008569 process Effects 0.000 description 11
- 238000013473 artificial intelligence Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000007689 inspection Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 239000000969 carrier Substances 0.000 description 2
- 238000011897 real-time detection Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 239000000523 sample Substances 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000009529 body temperature measurement Methods 0.000 description 1
- 210000005252 bulbus oculi Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 208000002173 dizziness Diseases 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000004936 stimulating effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Processing Or Creating Images (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a product interaction and fault diagnosis method based on augmented reality, which is characterized in that an interaction platform capable of viewing a real scene and virtual information display is established; the method comprises the steps that information which corresponds to a product needing interaction and needs to be transmitted to a user is electronically converted into an information source; and processing the keywords and the image unitized labels for the purpose of realizing quick retrieval; pre-storing the information source and the information processed by the label into an interactive platform or a network or local storage device, and then establishing a link channel capable of loading data information in real time with the interactive platform through a communication medium; when a demand instruction of the user is detected, the user can automatically answer the demand instruction, and simultaneously, information data which best accords with the current scene is called out and displayed to the user on the interaction platform so as to improve the interaction efficiency. The scheme has the advantages that both hands can be liberated, the method is more efficient, the virtuality and the reality are combined, the equipment information is accurately reflected, and the product can be subjected to fault diagnosis and recognition to guide maintenance.
Description
Technical Field
The invention belongs to the technical field of human-computer interaction, relates to an intelligent interaction technology aiming at large-scale equipment or complex machines, and particularly relates to a product interaction and fault diagnosis method based on augmented reality, which can enable a user to quickly master the functions, the using method, the dismounting and maintenance skills and quickly judge and lock the working state and the fault point of a product.
Background
In the prior art, large equipment or machines need to be provided with detailed specifications for users to know and use when leaving a factory, such as engine manuals, automobile manuals, precision lathe manuals and the like. Corresponding users such as machine operators, maintenance workers, secondary design developers and the like need to know relevant contents in the manuals so as to operate and use the equipment or maintain and design the equipment and the like. If not understood, the machine is directly operated or maintained by hands, so that the machine is easily damaged, and the function and the working efficiency of the machine are influenced.
However, the inventor of the present patent application has found that detailed manuals, including product specifications, service manuals, use manuals, user manuals, fault diagnosis manuals, disassembly manuals, and the like, are provided for equipment shipment in the prior art. Most of the manuals are made of paper, and the operation is very inconvenient when a user wants to quickly search related contents in the manuals. To solve this problem, some manufacturers may provide electronic documents on the basis of the paper manual, or provide a dedicated electronic tablet device, so that the user can quickly refer to the corresponding product manual. Although the searching efficiency is improved through the operation, the improvement is not obvious for large-scale equipment in the practical operation level. First, large device manuals are of many kinds, and users cannot find needed contents in a short time and effectively. Secondly, through the description of characters and two-dimensional pictures provided by the manual, the efficiency is low when the literal meaning is read and understood, and the wrong operation is probably caused by wrong literal understanding. Moreover, the manual can not be separated from two hands, and needs to be turned over by two hands, even multiple people cooperate (for example, one person turns over records and one person operates by contrast with a machine) in a narrow space, the contradiction is more prominent, and tools, manuals and the like can be placed everywhere; familiarity with learning is intolerable in terms of time, cost, and efficiency.
The inventor of this patent application finds through searching, to the problem that above-mentioned solution exists, someone has further proposed the scheme of introducing intelligent glasses, will be equipped with the electronic manual or panel computer upgrade to intelligent glasses promptly, lets the user directly look up relevant information through intelligent glasses. The solution remains in the simple phase of importing product information into existing commercial products. That is to say, although the search efficiency is further improved, the interaction efficiency is still very low, and the real scene cannot be identified and more support can be provided by combining the real scene of the product; and the product can not be diagnosed and identified more, and the user is guided to maintain by self. Meanwhile, the related functions of safety early warning for the user are lacked, namely, the user cannot be protected from being injured by collision of the product and other parts around the product when interacting with the product or diagnosing.
In addition, the inventor of the present patent application also finds out through search that the patent application numbers are: 202110337423.2, the patent names: an assembly guiding method, system and applied patent application based on augmented reality; the patent application discloses an assembly guiding method, system and application based on augmented reality, wherein the assembly guiding method comprises the following steps: analyzing the contour information and the matching information of the part model into structured data; identifying and capturing pose information of all parts, providing a three-dimensional tracking registration matrix of the parts for an AR scene, dynamically tracking and positioning the parts in a real scene, and building an augmented reality scene; extracting three-dimensional edge profile characteristic information of the parts in real time, and distinguishing a basic part and other parts of the to-be-assembled body in the current state; highlighting guide information at the position of the basic part to be assembled; carrying out envelope highlighting identification on a part to be assembled; assembling and dynamically visually recognizing and judging the assembling condition to realize the fusion of virtuality and reality; and repeating the steps to finish the installation of all parts. The inventor of the present patent application finds by contrast that the patent application discloses a solution for rapid assembly guidance by augmented reality techniques. But does not relate to the use instruction introduction, fault diagnosis, user safety early warning and other contents of the product.
Through further search, the inventor of the present patent application finds that the patent application numbers are: 202011420025.9, the patent names: the invention patent application of a system for controlling and visualizing a digital factory in real time in an augmented reality environment; the patent application discloses a system and a device for real-time control and visualization of a digital factory in an augmented reality environment, which at least comprise an augmented reality device, a server system and a plurality of controlled objects, wherein the server system at least comprises a two-dimensional module and a three-dimensional module, and the two-dimensional module displays a two-dimensional graphic component of the controlled objects in a two-dimensional mode; the three-dimensional module displays a three-dimensional graphic component of the controlled object in a three-dimensional mode; the attribute data of the controlled object can be displayed in a two-dimensional graph and a three-dimensional view mode in parallel in response to the operation instruction of the augmented reality device, and the two-dimensional module and/or the three-dimensional module can control the controlled object based on the operation instruction fed back by the augmented reality device. The inventor of the patent application finds through comparison that the patent application discloses a scheme for carrying out efficient routing inspection through an augmented reality technology, and simultaneously mentions the contents of prejudgment on local faults of a factory, routing inspection route planning for helping workers to bypass crisis and the like. However, the pre-judgment of the faults and the dangers is obtained based on the automatic analysis of data signals of various instruments and probes in an acquisition factory, the data signals are not directly detected by an interactive platform, information delay and transmission deviation exist, meanwhile, the system focuses on routing inspection, namely the system focuses on passively receiving routing inspection information, and efficient interaction is not achieved.
In summary, an augmented reality-based product interaction and fault diagnosis method which combines virtuality and reality, has a more efficient interaction mode, can diagnose and identify faults of products, guides users to maintain the products by themselves, and still guarantees the safety of the users to the greatest extent in the interaction and diagnosis process is still lacking at present.
Disclosure of Invention
The invention aims to solve the technical problem of providing an augmented reality-based product interaction and fault diagnosis method which combines virtuality and reality, has a more efficient interaction mode, can diagnose and identify faults of products, guides a user to self-help maintain and still ensures the safety of the user to the greatest extent in the interaction and diagnosis process.
In order to solve the technical problems, the technical scheme provided by the invention is as follows: a product interaction and fault diagnosis method based on augmented reality comprises the following steps:
the method comprises the steps that firstly, an interactive platform with intelligent wearable equipment which enables a user to view a real scene and virtual information as a carrier is established; the interactive platform is connected with a camera, a microphone, a sound transmission device, a temperature sensor, a flaw detection sensor, a distance measurement sensor and other functional sensors for detecting the state of a product; the interaction platform is provided with a safety early warning system for detecting whether a user is threatened by safety when interacting with a product and diagnosing faults so as to guarantee personal safety;
secondly, electronically converting information which is required to be transmitted to a user and corresponds to a product which needs to be interacted into an information source; the information source comprises the use instruction information and the fault diagnosis information of the product, and the forms comprise characters, voice, pictures and videos;
thirdly, processing the labels of the keywords and the images in a unitization way by using the information source in the second step for realizing quick retrieval; meanwhile, the information source and the information processed by the label are prestored in the interactive platform; or the link channel is pre-stored in a network or a local storage device and is established with the interactive platform to load data information in real time;
fourthly, controlling the interactive platform to acquire data information, instruction information of the user and safety information around the environment where the user is located of the product which needs to be interacted in front of the user in real time through the cameras and the sensors at the various visual angles which are carried by the interactive platform;
fifthly, after instruction information of the user is detected, the interaction platform carries out matching processing on the user instruction, the acquired input source data and the stored information source data, and then calls out information closest to the current scene to be displayed to the user on the interaction platform, wherein the displayed content comprises but is not limited to characters, voice, images, videos and animations; if a plurality of similar information results exist, displaying the result summary list according to the weight, informing a user to select information, and then screening and displaying the information according to a selection instruction of the user; the user can watch the real scene of the product before the eyes while watching the virtual information; when a safety early warning system of the interaction platform detects a safety threat, an early warning signal is immediately sent to a user, and a risk avoiding scheme is prompted.
Preferably, the interactive platform is provided with a multi-machine online communication function module, so that a multi-machine sharing visual angle and data sharing can be realized, and the communication mode comprises a physical wired connection mode and a wireless network connection mode; the functional sensors of the interactive platform comprise various sensors capable of collecting characteristic data of product images, gray scales, infrared rays and heat energy.
Preferably, the sound transmission system of the interactive platform comprises a loudspeaker, an earphone and a bone conduction microphone.
Preferably, the interactive platform is equipped with a 360-degree scene camera for acquiring real-time pictures around the user.
Preferably, the intelligent wearable device comprises AR glasses.
Preferably, the interaction platform can directly communicate with the product needing interaction and communicate the control signal.
Preferably, the interaction platform can be connected with other intelligent wearable devices at the same time.
Preferably, the information sources in the second step are updated online in real time.
Preferably, when the information is displayed to the user on the interactive platform in the fifth step, the position of the information display is combined with the part of the real product which needs to be explained or maintained correspondingly after the information is preprocessed or the image is intelligently processed in the later period.
Preferably, when the interactive platform performs information interaction, besides real-time detection of the safety condition of the working environment around the platform itself, the interactive platform also synchronously combines the collected information of other peripheral sensors to judge whether the user is in a safe working environment, and provides an alarm signal or sends a control command for emergently stopping the peripheral dangerous equipment.
After the scheme is adopted, the invention has the following beneficial effects: utilize augmented reality AR and artificial intelligence AI technique, virtual reality combines, uses and study the aspect in the product interaction, can help the user high-efficient master and use the main equipment, shortens the learning process, improves the availability factor. And the fault diagnosis process is simpler and more efficient in the aspect of fault diagnosis. The intelligent wearable device is used as a carrier, and hands are thoroughly liberated through various interaction means such as voice and gesture control. All documents and materials related to the equipment are electronized and digitally virtualized. The method can be used for on-line retrieval and search and displaying the search result in real time. Through the maintenance process of projecting the three-dimensional display equipment component, the user can operate according to the real object.
Various expansion functions are added in the preferred item, such as the function of multi-machine on-line communication, the multi-machine sharing visual angle camera can be realized, and the user can directly communicate with a product manufacturer or a maintenance expert on line. The bone conduction microphone is adopted by the sound transmission system, so that the mutual interference of peripheral sounds can be reduced, and the situation that a user cannot hear an alarm sound of equipment or sounds among peripheral colleagues is avoided. 360 degree scene camera, collision early warning radar system, interactive platform can directly communicate and convey control signal with the product that needs carry out the interaction, connect extension structures and control method such as other intelligent wearable equipment simultaneously, all are based on the extension of production safety, let the user immerse in the interaction simultaneously, and furthest ensures user's safety. Meanwhile, the interactive platform can directly communicate and control with the interacted product, interaction efficiency is further improved, the interactive platform can not only stay in learning and maintaining products, but also directly control the products to perform corresponding feedback or demonstrate corresponding states.
To sum up, this patent application provides one kind can liberate user's both hands, provides more efficient interactive mode, and virtual reality combines, accurate reflection equipment information to can carry out failure diagnosis and discernment to the product, instruct the self-service product interaction and the failure diagnosis method based on augmented reality of maintaining of user.
Drawings
Fig. 1 is a block schematic diagram of a method involved in the invention when interacting with an augmented reality based product.
Fig. 2 is a block diagram illustrating a method for diagnosing a failure of an augmented reality-based product according to the present invention.
FIG. 3 is a schematic diagram of core modules in implementing the augmented reality-based product interaction and fault diagnosis method of the present invention.
Fig. 4 is a schematic diagram of the core module workflow in fig. 3.
Fig. 5 is a virtual information display example of the fault diagnosis embodiment.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
The invention discloses a product interaction and fault diagnosis method based on augmented reality, which comprises the following steps:
the method comprises the steps that firstly, an interactive platform with intelligent wearable equipment which enables a user to view a real scene and virtual information as a carrier is established; the interactive platform is connected with a camera, a microphone, a sound transmission device, a temperature sensor, a flaw detection sensor, a distance measurement sensor and other functional sensors for detecting the state of a product; the interaction platform is provided with a safety early warning system for detecting whether a user is threatened by safety when interacting with a product and diagnosing faults so as to guarantee personal safety;
secondly, electronically converting information which is required to be transmitted to a user and corresponds to a product which needs to be interacted into an information source; the information source comprises the use instruction information and the fault diagnosis information of the product, and the forms comprise characters, voice, pictures and videos;
thirdly, processing the labels of the keywords and the images in a unitization way by using the information source in the second step for realizing quick retrieval; meanwhile, the information source and the information processed by the label are prestored in the interactive platform; or the link channel is pre-stored in a network or a local storage device and is established with the interactive platform to load data information in real time;
fourthly, controlling the interactive platform to acquire data information, instruction information of the user and safety information around the environment where the user is located of the product which needs to be interacted in front of the user in real time through the cameras and the sensors at the various visual angles which are carried by the interactive platform;
fifthly, after detecting instruction information of the user, the interaction platform carries out matching processing on the user instruction, the acquired input source data and the stored information source data, calls out information closest to the current scene and displays the information to the user on the interaction platform, wherein the display content comprises but is not limited to characters, voice, images, videos and animations; if a plurality of similar information results exist, displaying the result summary list according to the weight, informing a user to select information, and then screening and displaying the information according to a selection instruction of the user; the user can watch the real scene of the product in front of the eyes while watching the virtual information; when a safety early warning system of the interaction platform detects safety threat, an early warning signal is immediately sent to a user, and a risk avoiding scheme is prompted.
Preferably, the interactive platform is provided with a multi-machine online communication function module, so that a multi-machine sharing visual angle and data sharing can be realized, and the communication mode comprises a physical wired connection mode and a wireless network connection mode; the functional sensors of the interactive platform comprise various sensors capable of collecting characteristic data of product images, gray scales, infrared rays and heat energy.
Preferably, the sound transmission system of the interactive platform comprises a loudspeaker, an earphone and a bone conduction microphone.
Preferably, the interactive platform is equipped with a 360-degree scene camera for acquiring real-time pictures around the user.
Preferably, the intelligent wearable device comprises AR glasses.
Preferably, the interaction platform can directly communicate with the product needing interaction and communicate the control signal.
Preferably, the interaction platform can be connected with other intelligent wearable devices at the same time.
Preferably, the information sources in the second step are updated online in real time.
Preferably, when the information is displayed to the user on the interactive platform in the fifth step, the position of the information display is combined with the part of the real product which needs to be explained or maintained correspondingly after the information is preprocessed or the image is intelligently processed in the later period.
Preferably, when the interactive platform performs information interaction, besides real-time detection of the safety condition of the working environment around the platform itself, the interactive platform also synchronously combines the collected information of other peripheral sensors to judge whether the user is in a safe working environment, and provides an alarm signal or sends a control command for emergently stopping the peripheral dangerous equipment.
In the above preferred embodiment, the interactive platform has a multi-machine online communication function, and can realize multi-machine sharing of the viewing angle camera. And the sound transmission system of the interactive platform adopts a bone conduction microphone. The interactive platform is provided with a 360-degree scene camera and is used for acquiring real-time pictures around the user. And the interaction platform is provided with a collision early warning radar system. The interaction platform can directly communicate with the product needing interaction and convey control signals. The interaction platform can be connected with other intelligent wearable devices at the same time. The information source can be updated online in real time. When the virtual projection is displayed to a user on an interactive platform, the projected position is combined with a corresponding area needing explanation or maintenance in a real product after information preprocessing or later image intelligent processing; while the projection area can be selected by the user at his discretion. When the interactive platform carries out information interaction, whether the user is in a safe working environment or not can be judged by synchronously combining the collected user peripheral information, and an alarm signal is provided or a control command for emergently stopping peripheral dangerous equipment is sent out.
Several usage scenarios corresponding to the present invention application are described in detail below in the form of specific embodiments.
Detailed description of the preferred embodiment
Further referring to FIG. 1, the basic product interaction function is taken as an example; the interactive platform adopts intelligent glasses; the product is a multifunctional device with a plurality of robot devices. When a user inputs a certain keyword (voice or soft keyboard) into the intelligent glasses, the keyword is definitely a part of a current product, the intelligent glasses acquire a current camera scene, the computing unit performs recognition detection by using a pre-stored part model, and a detection result is identified in the scene through the frame body. If a plurality of frame results in the scene are detected, the user is informed to input a screening instruction, and after the user selects and determines a certain frame, the acquisition unit searches and retrieves information of a corresponding part to display in the scene preset area 1. As a further improvement, if the frame results are excessive, a weighting algorithm is added, and intelligent sorting is performed first, that is, the interactive platform performs intelligent sorting by combining the information of the user's perspective information and the user's operation habits, and the most probable results are arranged at the front position.
For example, in a voice mode, the user speaks "robot", and the host calls a section of the user manual "robot" and displays the section on the screen. And in the gesture mode, a search bar is clicked, a soft keyboard is popped up, and Chinese is input into the robot, and similarly, the host can call up the content of the section of the user manual robot and put and display the content in the preset area 1. Next, the user may input a keyword command of a certain operation by voice, and the obtaining unit of the interactive platform detects that the part is preset with a model action or a video of the operation, that is, an operation animation displaying the three-dimensional virtual model of the part or a video of the operation of the part may be launched in the scene preset area 2. The user can perform cognitive learning by contrasting the information data output by the acquisition unit and operate the actual equipment by contrasting the played three-dimensional animation or video. Wherein the computing unit: the real-time video detection device is used for detecting a real-time video of a camera, calculating the position and the size of a detected part and marking a frame in a real scene. The computing unit can be hardware such as a CPU, a GPU and the like and a software algorithm module running on the hardware. The unit is carried on common intelligent glasses, and a module with better performance can be designed by self. Wherein the acquisition unit: the method is used for acquiring corresponding data information, three-dimensional virtual models, operation animations, videos and the like of the specified frame parts. The acquisition unit may be hardware such as a memory and a sensor, or may be a software module such as a search calculation. The display unit is used for outputting the content of the acquisition unit in a specific form in a preset display area, and the unit is carried on common intelligent glasses and can also be designed into a module with better performance.
Practice proves that the embodiment utilizes the AR and AI technologies, combines virtual reality, can help a user to efficiently master and use large equipment, shortens the learning process and improves the use efficiency. The glasses are used as carriers, and hands are completely liberated through various interaction means such as voice and gesture control; all documents and materials associated with the device are electronically digitized. The method can be used for on-line retrieval and search and displaying the search result in real time. The device components can be detected and identified, and the user can be guided to be familiar with device attributes and data by displaying a virtual frame, a virtual three-dimensional model, characters, pictures, videos and the like in a specific area. The disassembly and assembly process of the three-dimensional immersive display equipment component can be realized, and a user can operate the three-dimensional immersive display equipment component in comparison with a real object.
Detailed description of the invention
Further referring to FIG. 2, the basic product failure diagnosis function is taken as an example; the interactive platform adopts intelligent glasses; the product is a multifunctional device with a plurality of robot devices. The faults of all parts of the equipment can occur, and the typical faults can be judged through expressions, such as switches, signal lamp display, relative positions of parts and the like, and can be stored through data processing and feature extraction and models in advance. When the glasses are in a diagnosis mode, videos are collected in real time through the camera, the calculation unit calculates and compares the videos with a prestored model, and the position and the range of a fault area are obtained in real time to be marked. The acquisition unit displays the information of searching the corresponding fault in a preset area 1, and displays a corresponding maintenance video or a virtual maintenance animation in a preset area 2. Wherein the computing unit: the real-time video detection device is used for detecting a real-time video of a camera, calculating the position and the size of a detected part and marking a frame in a real scene. The computing unit can be hardware such as a CPU, a GPU and the like and a software algorithm module running on the hardware. The unit is carried on common intelligent glasses, and a module with better performance can be designed by self. Wherein the acquisition unit: for obtaining corresponding fault data information, such as fault text description, solution, image description, etc., for a given frame component. The display unit is used for outputting the content of the acquisition unit in a specific form to a preset display area, and the unit is carried on common intelligent glasses and can also be used for designing a module with better performance.
Practice proves that the embodiment combines virtual reality by utilizing the augmented reality AR and artificial intelligence AI technologies, so that the fault diagnosis process is simpler and more efficient. The glasses are used as carriers, and hands are completely liberated through various interaction means such as voice and gesture control; all documents and materials associated with the device are electronically digitized. The method can be used for on-line retrieval and search and displaying the search result in real time. The maintenance process of the equipment component can be displayed in a three-dimensional mode, and a user can operate the equipment component in a real object mode.
Detailed description of the preferred embodiment
Further, with reference to fig. 3 to 5, AR glasses are used as main body interaction devices, a camera on the glasses collects a video of a current scene, a display unit displays contents to be displayed in a physical environment in a manner of superimposing a specified mode (position coordinate, size, color), and a calculation unit and a storage unit are connected to the glasses main body in a wired or wireless manner. The AR glasses camera transmits the current video scene to the computing unit, the computing unit detects that two POI (point of interest) parts exist in the visual field range of the user through an AI (artificial intelligence) algorithm, and the display unit draws a marking frame according to the position and the size of the parts given by the computing unit. The operator selects one of the target diagnosis parts, and the selection modes can be glasses head rays, handle rays, eyeball tracking and gestures. And the calculating unit reads the target scene diagnosis information prestored in the storage unit and carries out the next step. For example, the voice and text prompt 'please get close to the switch panel part of the engine', the computing unit reads the infrared temperature measurement, finds that the temperature is too high, and the voice and text prompt 'the equipment temperature is 48 degrees and the alarm temperature is 50 degrees', meanwhile, the computing unit detects the fuse reversal through the current panel image, marks a highlight frame on the fuse position, prompts by an arrow, and prompts 'the fuse reversal, please turn off the power supply of the equipment, and reopen the fuse'. And displaying the three-dimensional model animation with insurance at the highlight frame edge.
In terms of user safety guarantee, in the implementation process of the three embodiments, the safety state of the environment where the user is located is detected in real time through the interactive platform, including whether a moving object collides with the user, whether abnormal information that parts on a product suddenly suffer from high temperature and high pressure or leakage exists, or whether the user is excessively stressed due to the impact of virtual reality information or suddenly falls due to physical reasons and the like, the safety early warning system detects the abnormal information and correspondingly makes a response to guarantee the safety of the user. The safety system of the interactive platform focuses on the wearable devices such as a bracelet directly connected with the platform or a smart helmet directly mounted on the platform. The corresponding safety system comprises a vehicle-mounted anti-collision mini radar probe which is used for detecting the distance change condition between the obstacles around the user and the body of the user, and if an object is detected to be fast close to the body of the user, an alarm is sent out in time to prompt the user to avoid danger urgently. The safety system can also comprise a bracelet device, the physical condition of a user can be detected in real time, and the situation that the heart rate is too fast and the user feels dizzy under some stimulating augmented reality environments is prevented, so that the safety system can be found by no one. Under the further improvement of the safety system, the safety system can also realize linkage control with products and equipment around the products, such as linkage with the emergency braking function of the products and linkage with a power supply system around the products; when a user is in an interaction process with a product, the user is involved in a gear of the product under the influence of augmented reality or an electric shock accident occurs, a safety detection sensor worn by the user, such as a bracelet, an intelligent camera and the like, can detect in time or can receive dangerous information in time through a voice instruction sent by the user, and emergency braking or emergency power failure, operation of starting fire alarm and the like are realized through linkage control, so that safety guarantee is realized at the highest speed.
When the method is implemented specifically, the structural design, the hardware circuit, the chip and the control program and algorithm of a specific product are different, and the method needs to be operated by a producer according to specific product characteristics and functional requirements under the guidance of the interaction method provided by the patent application due to different interaction requirements corresponding to different products. After knowing the product interaction and fault diagnosis method in the present patent application, a set of hardware structures and specific algorithms are designed according to the method without technical obstacles in the implementation method level, and details are not repeated here.
The present invention and its embodiments have been described above, and the description is not intended to be limiting, and the drawings are only one embodiment of the present invention, and the actual configuration is not limited thereto. In summary, those skilled in the art should appreciate that they can readily use the disclosed conception and specific embodiments as a basis for designing or modifying other structures for carrying out the same purposes of the present invention without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A product interaction and fault diagnosis method based on augmented reality is characterized in that: it comprises the following steps:
the method comprises the steps that firstly, an interactive platform with intelligent wearable equipment which enables a user to view a real scene and virtual information as a carrier is established; the interactive platform is connected with a camera, a microphone, a sound transmission device, a temperature sensor, a flaw detection sensor, a distance measurement sensor and other functional sensors for detecting the state of a product; the interaction platform is provided with a safety early warning system for detecting whether a user is threatened by safety when interacting with a product and diagnosing faults so as to guarantee personal safety;
step two, electronically converting information which is required to be transmitted to a user and corresponds to a product which is required to be interacted into an information source; the information source comprises the use instruction information and the fault diagnosis information of the product, and the form comprises characters, voice, pictures and video;
thirdly, processing the labels of the keywords and the images in a unitization way by using the information source in the second step for realizing quick retrieval; meanwhile, the information source and the information processed by the label are prestored in the interactive platform; or the link channel is pre-stored in a network or a local storage device and is established with the interactive platform to load data information in real time;
fourthly, controlling the interactive platform to acquire data information, instruction information of the user and safety information around the environment where the user is located of the product which needs to be interacted in front of the user in real time through the cameras and the sensors at the various visual angles which are carried by the interactive platform;
fifthly, after instruction information of the user is detected, the interaction platform carries out matching processing on the user instruction, the acquired input source data and the stored information source data, and then calls out information closest to the current scene to be displayed to the user on the interaction platform, wherein the displayed content comprises but is not limited to characters, voice, images, videos and animations; if a plurality of similar information results exist, displaying the result introduction list according to the weight, informing a user to select information, and then screening and displaying the information according to a selection instruction of the user; the user can watch the real scene of the product in front of the eyes while watching the virtual information; when a safety early warning system of the interaction platform detects safety threat, an early warning signal is immediately sent to a user, and a risk avoiding scheme is prompted.
2. The augmented reality-based product interaction and fault diagnosis method of claim 1, wherein: the interactive platform is provided with a multi-machine online communication function module, can realize multi-machine sharing visual angle and data sharing, and adopts a communication mode comprising a physical wired connection mode and a wireless network connection mode; the functional sensors of the interactive platform comprise various sensors capable of collecting characteristic data of product images, gray scales, infrared rays and heat energy.
3. The augmented reality-based product interaction and fault diagnosis method of claim 1, wherein: the sound transmission system of the interactive platform comprises a loudspeaker, an earphone and a bone conduction microphone.
4. The augmented reality-based product interaction and fault diagnosis method of claim 1, wherein: the interactive platform is provided with a 360-degree scene camera and is used for acquiring real-time pictures around the user.
5. The augmented reality-based product interaction and fault diagnosis method of claim 1, wherein: the intelligent wearable device comprises AR glasses.
6. The augmented reality-based product interaction and fault diagnosis method of claim 1, wherein: the interaction platform can directly communicate with the product needing interaction and convey control signals.
7. The augmented reality-based product interaction and fault diagnosis method of claim 1, wherein: the interaction platform can be connected with other intelligent wearable devices at the same time.
8. The augmented reality-based product interaction and fault diagnosis method of claim 1, wherein: the information source in the second step can be updated online in real time.
9. The augmented reality-based product interaction and fault diagnosis method of claim 1, wherein: and fifthly, when the information is displayed to the user on the interactive platform, after the information is preprocessed or the image is processed intelligently in the later period, the position of the information display is combined with the part of the real product, which needs to be explained or maintained correspondingly.
10. The augmented reality-based product interaction and fault diagnosis method of any one of claims 2 to 7, wherein: when the interactive platform carries out information interaction, the safety condition of the working environment around the platform can be detected in real time, the acquired information of other peripheral sensors can be synchronously combined, whether a user is in a safe working environment or not is judged, and an alarm signal is provided or a control command for emergently stopping peripheral dangerous equipment is sent out.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210231893.5A CN114567535B (en) | 2022-03-10 | 2022-03-10 | Product interaction and fault diagnosis method based on augmented reality |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210231893.5A CN114567535B (en) | 2022-03-10 | 2022-03-10 | Product interaction and fault diagnosis method based on augmented reality |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114567535A true CN114567535A (en) | 2022-05-31 |
| CN114567535B CN114567535B (en) | 2024-01-09 |
Family
ID=81717645
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210231893.5A Active CN114567535B (en) | 2022-03-10 | 2022-03-10 | Product interaction and fault diagnosis method based on augmented reality |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114567535B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116778119A (en) * | 2023-06-26 | 2023-09-19 | 中国信息通信研究院 | Man-machine cooperative assembly system based on augmented reality |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108879440A (en) * | 2018-06-20 | 2018-11-23 | 国网山东省电力公司济宁供电公司 | Intelligent examination and repair system and method based on wearable augmented reality display terminal and cloud platform |
| CN111638796A (en) * | 2020-06-05 | 2020-09-08 | 浙江商汤科技开发有限公司 | Virtual object display method and device, computer equipment and storage medium |
| CN114116100A (en) * | 2021-11-26 | 2022-03-01 | 内蒙动力机械研究所 | Missile weapon guarantee maintenance system and method based on AR interactive electronic manual |
-
2022
- 2022-03-10 CN CN202210231893.5A patent/CN114567535B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108879440A (en) * | 2018-06-20 | 2018-11-23 | 国网山东省电力公司济宁供电公司 | Intelligent examination and repair system and method based on wearable augmented reality display terminal and cloud platform |
| CN111638796A (en) * | 2020-06-05 | 2020-09-08 | 浙江商汤科技开发有限公司 | Virtual object display method and device, computer equipment and storage medium |
| CN114116100A (en) * | 2021-11-26 | 2022-03-01 | 内蒙动力机械研究所 | Missile weapon guarantee maintenance system and method based on AR interactive electronic manual |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116778119A (en) * | 2023-06-26 | 2023-09-19 | 中国信息通信研究院 | Man-machine cooperative assembly system based on augmented reality |
| CN116778119B (en) * | 2023-06-26 | 2024-03-12 | 中国信息通信研究院 | Man-machine cooperative assembly system based on augmented reality |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114567535B (en) | 2024-01-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101990284B1 (en) | Intelligent cognitive technology based augmented reality system using speech recognition | |
| CN106340217B (en) | Manufacturing equipment intelligence system and its implementation based on augmented reality | |
| CN105182535B (en) | The method that automobile maintenance is carried out using intelligent glasses | |
| US10360452B2 (en) | Wearable device and vehicle diagnosis apparatus including the same | |
| US8125405B2 (en) | Wearable computer in a process control environment | |
| KR101636360B1 (en) | A virtual maintenance training system using a virtual reality | |
| US7245271B2 (en) | Portable computer in a process control environment | |
| US7814122B2 (en) | System and method for documentation processing with multi-layered structuring of information | |
| KR20190077898A (en) | Abnormal status monitoring system for equipment | |
| US20130010068A1 (en) | Augmented reality system | |
| JP2002538543A (en) | System and method for contextually assisting dialogue with enhanced reality technology | |
| CN113222184A (en) | Equipment inspection system and method based on augmented reality AR | |
| US20210224752A1 (en) | Work support system and work support method | |
| JP6270488B2 (en) | Operator monitoring control device and operator monitoring control method | |
| CN114567535A (en) | Product interaction and fault diagnosis method based on augmented reality | |
| CN112417142A (en) | Auxiliary method and system for generating word meaning and abstract based on eye movement tracking | |
| CN111552269B (en) | Multi-robot safety detection method and system based on attitude estimation | |
| JP7265667B2 (en) | Work support system and work support method | |
| CN112949457A (en) | Maintenance method, device and system based on augmented reality technology | |
| EP3885972A1 (en) | Context based perception method and system for managing environmental safety in a computing environment | |
| KR101576445B1 (en) | image evalution automation method and apparatus using video signal | |
| Bode | Evaluation of an augmented reality assisted manufacturing system for assembly guidance | |
| KR20210064830A (en) | Image processing system | |
| CN110458052A (en) | Recongnition of objects method, apparatus based on augmented reality, equipment, medium | |
| CN105643664A (en) | Vision recognition determination method of service robot and vision system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |