CN106339094B - Interactive remote expert cooperation examination and repair system and method based on augmented reality - Google Patents
Interactive remote expert cooperation examination and repair system and method based on augmented reality Download PDFInfo
- Publication number
- CN106339094B CN106339094B CN201610801442.5A CN201610801442A CN106339094B CN 106339094 B CN106339094 B CN 106339094B CN 201610801442 A CN201610801442 A CN 201610801442A CN 106339094 B CN106339094 B CN 106339094B
- Authority
- CN
- China
- Prior art keywords
- image
- video
- target
- sent
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 34
- 238000000034 method Methods 0.000 title claims abstract description 17
- 230000008439 repair process Effects 0.000 title claims abstract description 17
- 230000002452 interceptive effect Effects 0.000 title claims abstract description 11
- 238000012423 maintenance Methods 0.000 claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 29
- 238000001514 detection method Methods 0.000 claims abstract description 4
- 238000012706 support-vector machine Methods 0.000 claims description 39
- 238000004891 communication Methods 0.000 claims description 32
- 239000002245 particle Substances 0.000 claims description 31
- 230000015572 biosynthetic process Effects 0.000 claims description 23
- 238000003786 synthesis reaction Methods 0.000 claims description 23
- 230000004927 fusion Effects 0.000 claims description 12
- 230000005540 biological transmission Effects 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 5
- 238000002844 melting Methods 0.000 claims 2
- 230000008018 melting Effects 0.000 claims 2
- 238000003745 diagnosis Methods 0.000 abstract 1
- 238000012360 testing method Methods 0.000 description 6
- 238000012549 training Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 235000013399 edible fruits Nutrition 0.000 description 4
- 239000011521 glass Substances 0.000 description 4
- HUTDUHSNJYTCAR-UHFFFAOYSA-N ancymidol Chemical compound C1=CC(OC)=CC=C1C(O)(C=1C=NC=NC=1)C1CC1 HUTDUHSNJYTCAR-UHFFFAOYSA-N 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000035800 maturation Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000003014 reinforcing effect Effects 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B23/00—Testing or monitoring of control systems or parts thereof
- G05B23/02—Electric testing or monitoring
- G05B23/0205—Electric testing or monitoring by means of a monitoring system capable of detecting and responding to faults
- G05B23/0259—Electric testing or monitoring by means of a monitoring system capable of detecting and responding to faults characterized by the response to fault detection
- G05B23/0275—Fault isolation and identification, e.g. classify fault; estimate cause or root of failure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/012—Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
Abstract
The invention discloses interactive remote expert cooperation examination and repair systems and method based on augmented reality；The equipment that Measuring error is treated at on-site customer end carries out video acquisition or image taking, it is transferred to remote service end, remote service end receives on-site customer the end video or image sent, receives expert and delineates processing to video or image, the target area image delineated is sent to on-site customer end；Identification, positioning and the tracking of target area in target area image and live video image are completed after receiving by object tracking algorithm in on-site customer end, detector finds target area image, target is tracked by tracker, the detection accuracy of detector is continuously improved by P-N Learning, indicia patterns or text annotation are superimposed at the position in target in video image region at the scene, the status information of current device generates augmented reality image video；Augmented reality video is synthesized using augmented reality, the fault diagnosis and maintenance of field device are completed under the cooperation of expert.
Description
Technical field
The interactive remote expert that the present invention relates to a kind of based on augmented reality cooperates examination and repair system and method.
Background technique
Augmented reality (Augmented Reality, abbreviation AR) is increased by the information that computer system provides to be used
Family perceives real world, by virtual Information application to real world, and by the dummy object of computer generation, scene or is
System prompt information is added in real scene, to reach the reinforcing effect to reality.The target of this technology is on the screen
Virtual world is implanted into real world and is interacted.Currently, being MS using the product of AR Technical comparing maturation
Hololens glasses, but Hololens only support by computer generate effect be superimposed on real world, be supplied to list
A user checks or interacts therewith, and but can not achieve the interaction between multiple users.And existing utilization AR skill
The product of art is chiefly used in the consumer field such as life, social activity, seldom even without answering in terms of towards industrial system Measuring error
With.
At this stage, large scale industry system equipment is distributed each corner at home and abroad already, with 777 visitors of Boeing Co.
For machine, ends in March, 2012, produced more than 800 framves, be distributed in China, the U.S., Britain, Japan etc. all over the world.Aircraft
Engine, wing, fuselage, driver's cabin are designed manufacture by professional, are related to electronics, physics, dynamics, Information Center
The multi-subject knowledges such as skill, artificial intelligence, pattern-recognition.When aircraft goes wrong, it is necessary to reach scene by special expert and carry out
Measuring error, the time for not only increasing product repairing also considerably increase the operation cost of service provider.
Summary of the invention
The purpose of the present invention is to solve the above-mentioned problems, provides a kind of interactive remote based on augmented reality
Expert's cooperation examination and repair system and method, it utilizes augmented reality, by expert feedback information, on-site customer acquisition video or figure
The status information of picture and current device synthesizes augmented reality video, liberates client's both hands, and client is helped to be quickly found out problem.
To achieve the goals above, the present invention adopts the following technical scheme:
Interactive remote expert cooperation examination and repair system based on augmented reality, comprising: on-site customer end and long-range clothes
Business end；
The equipment that the on-site customer end is used to treat Measuring error carries out video acquisition or image taking, by the letter of acquisition
Breath is transferred to remote service end, and receives the target area image delineated at remote service end, the target area image that will be delineated
It is synthesized with the video of collection in worksite, at the scene the target position of client real-time display maintenance to be detected；
The remote service end receives expert to video or image for receiving the video or image that on-site customer end is sent
Delineate processing, the target area image delineated is sent to on-site customer end.
On-site customer end, comprising:
Video acquisition unit, the equipment for treating Measuring error carry out video acquisition or image taking；It obtains to be detected
The video flowing or image of the equipment of maintenance；The equipment video flowing of the maintenance to be detected of acquisition or image are sent to the first communication unit
Member；
Video flowing or image are sent to far for receiving video flowing or image by wireless network by the first communication unit
Journey server-side, and receive the feedback information at remote service end；The feedback information includes that voice messaging and band delineate markup information
Target area image；
AR processing unit, the acquisition information that feedback information, video acquisition unit for sending to remote service end are sent
Augmented reality video is synthesized with the status information of current device；The first display unit is sent to after synthesis；Video acquisition unit hair
The acquisition information come includes the equipment video flowing or image of maintenance to be detected；
First display unit is shown for the processing result to AR processing unit.
Remote service end, comprising:
Second communication unit, for receiving the video flowing or image of the transmission of the first communication unit；By received video flowing or
Image is sent to video image synthesis unit；Video image synthesis unit feedback information is received, feedback information is sent to first
Communication unit；The feedback information includes the target area image that voice messaging and band delineate markup information；
Video image synthesis unit synthesizes with image for that will delineate endorsement information, band is delineated endorsement information
Target area image is cut；Then the target area image of cutting is sent to the second communication unit；
Second display unit for showing video flowing or image that the first communication unit sends over, while being also used to show
Show the processing result of video image synthesis unit；
Second input unit delineates markup information and voice messaging for receiving, and will delineate markup information and voice letter
Breath is sent into video image synthesis unit.
Interactive remote expert cooperation repair method based on augmented reality, includes the following steps:
Step (1): the equipment for treating Measuring error carries out video acquisition or image taking, obtains maintenance of equipment view to be detected
Frequency stream or image；
Step (2): the maintenance of equipment video flowing or image to be detected of acquisition are sent to remote service end by on-site customer end；
Step (3): remote service end receives the video flowing or image and shows, receives expert to doubtful Measuring error portion
Position delineate the information of annotation；Receive the voice messaging of repair prompt；
Step (4): remote service end will delineate endorsement information and synthesize with image, and band is delineated to the target of endorsement information
Area image is cut；Then the target area image of cutting and voice messaging are sent to on-site customer end.
Step (5): on-site customer end receives the feedback information that remote service end sends over；The feedback information includes language
Message breath and band delineate the target area image of markup information；
Step (6): the target area image of markup information is delineated at on-site customer end to band, is completed by object tracking algorithm
Identification, positioning and the tracking of target area in target area image and live video image, at the scene target in video image area
It is superimposed indicia patterns or text annotation at the position in domain, and is superimposed the status information of current device, to generate augmented reality shadow
As video；
Step (7): on-site customer end shows augmented reality image video.
On-site customer end, comprising:
Video acquisition unit, the equipment for treating Measuring error carry out video acquisition or image taking；It obtains to be detected
The video flowing or image of maintenance of equipment；The maintenance of equipment video flowing or image to be detected of acquisition are sent to the first communication unit；
Video flowing or image are sent to far for receiving video flowing or image by wireless network by the first communication unit
Journey server-side, and receive the feedback information at remote service end；The feedback information includes that voice messaging and band delineate markup information
Target area image；
AR processing unit, the acquisition information that feedback information, video acquisition unit for sending to remote service end are sent
Augmented reality video is synthesized with the status information of current device；The first display unit is sent to after synthesis；Video acquisition unit hair
The acquisition information come includes the equipment video flowing or image of maintenance to be detected；
First display unit is shown for the processing result to AR processing unit.
The AR processing unit includes:
Particle filter tracker obtains position and the size of target area for tracking target；
Support vector machine classifier, for detecting all possible target；
Learn subelement, for using P-N Learning learning strategy complete support vector machine classifier it is online more
Newly, the nicety of grading of support vector machine classifier is stepped up；
Object module finds the highest sample of matching degree for being matched with sample to be sorted；
Subelement is merged, for by the tracking result of the classification results of support vector machine classifier and particle filter tracker
It blends, obtains the position of target.
Preferably, the video acquisition unit is video camera.
Further, on-site customer end, further includes: the first input unit.
Preferably, first input unit includes the first touch panel and the first voice module, the first voice mould
Block includes the first microphone and the first loudspeaker.
Further, on-site customer end, further includes: the first power supply unit.
Preferably, the on-site customer end is smart phone, tablet computer or wearable device.
Preferably, the wearable device is AR intelligent glasses.
Remote service end, comprising:
Second communication unit, for receiving the video flowing or image of the transmission of the first communication unit；By received video flowing or
Image is sent to video image synthesis unit；Video image synthesis unit feedback information is received, feedback information is sent to first
Communication unit；The feedback information includes the target area image that voice messaging and band delineate markup information；
Video image synthesis unit synthesizes with image for that will delineate endorsement information, band is delineated endorsement information
Target area image is cut；Then the target area image of cutting is sent to the second communication unit；
Second display unit for showing video flowing or image that the first communication unit sends over, while being also used to show
Show the processing result of video image synthesis unit；
Second input unit delineates markup information and voice messaging for receiving, and will delineate markup information and voice letter
Breath is sent into video image synthesis unit.The markup information of delineating refers to that expert delineates and builds to maintenance to punctuating for target area
The text of view annotates.
The video image synthesis unit includes:
Synthesizing subunit, for being synthesized to the target area image delineated with endorsement information；
Subelement is cut out, for being cut out to the image information of synthesis, obtains required target image information.
Further, remote service end, further includes: the second input unit；
Preferably, second input unit includes the second touch panel and the second voice module, the second voice mould
Block includes second microphone and the second loudspeaker.
Further, remote service end, further includes: the second power supply unit.
The repair method at on-site customer end, includes the following steps:
Step (a-1): the equipment for treating Measuring error carries out video acquisition or image taking, obtains maintenance of equipment to be detected
Video flowing or image；
Step (a-2): the video flowing of the maintenance of equipment to be detected of acquisition or image are sent to remote service end；
Step (a-3): the feedback information sended over by remote service end is received；The feedback information includes voice messaging
The target area image of markup information is delineated with band；
Step (a-4): target area in target area image and live video image is completed by object tracking algorithm
Identification, positioning and tracking are superimposed indicia patterns or text annotation at the scene, and are superimposed at the position in target in video image region
The status information of current device, to generate augmented reality image video；
Step (a-5): augmented reality image video is shown.
Step (a-4) the object tracking algorithm steps are as follows:
Step (a-4-1): the 1st frame of tracking process is identified in video image；It is loaded into the target that band delineates markup information
Area image is as tracking target P；
Step (a-4-2): initialization support vector machine classifier and particle filter tracker；
Step (a-4-3): in the subsequent frame processing of video, being loaded into the i-th frame video image, and the value range of i is 2~n,
N indicates the number of frame in video；All possible target P={ P is detected using support vector machine classifier1,P2,...,Pn,
Target is tracked using particle filter tracker, obtains position and the size of target area；
Step (a-4-4): by the tracking result phase of the classification results of support vector machine classifier and particle filter tracker
Fusion；The target position of detection positioning is obtained after fusion；Enter step (a-4-6)；
Step (a-4-5): the online updating of support vector machine classifier is completed using P-N Learning learning strategy, is returned
It returns step (a-4-3)；
Step (a-4-6): judging whether video has reached n-th frame, if so, terminating；Otherwise, i=i+1；It is transferred to step
Suddenly (a-4-3).
There are five types of possible altogether for step (a-4-4) fusion:
The first, support vector machine classifier and particle filter tracker have image block to be used as output, but supporting vector
Machine classifier has multiple similar image blocks to be determined out, the multiple to refer to greater than one；And particle filter tracker is only
A target position is found, arest neighbors classification, selection and previous frame target are carried out to testing result with normalized related coefficient
The highest image block of similarity is as last testing result；
Second, particle filter tracker does not have image block output, and support vector machine classifier has image block output, that
Testing result is clustered with normalized related coefficient to be divided again, using first cluster segmentation result as fusion knot
Fruit；
The third, particle filter tracker does not have image block output, and support vector machine classifier has image block output, such as
Fruit Clustering Decision-Making result is the largest the result of decision corresponding to related coefficient, but the result of decision is more than the threshold value model of setting
It encloses, then then initialization is re-started to particle filter tracker as fusion results using the result of decision, and lose and recognized originally
For correct sample；
4th kind, particle filter tracker has image block output, and support vector machine classifier does not export image block, that
Using the output result of particle filter tracker as fusion results；
5th kind, if support vector machine classifier and particle filter tracker are all exported without image block, then it is assumed that mesh
Mark disappears.
The online updating process of step (a-4-5) support vector machine classifier is:
In P-N Learning, P-expert and N-expert is defined；
P-expert find video sequence in time domain on Structural Characteristics: time structure, and assume target be along
Trajectory line movement；
P-expert, which is found out, to be classified device and is classified as negative but requires the classification should to be according to structural constraints
The sample of positive；
N-expert finds the Structural Characteristics in the spatial domain in video sequence: space structure, and assumes target one
It is only possible to appear in a position in a video frame；
N-expert, which is found out, to be classified device and is classified as positive but requires the classification should to be according to structural constraints
The sample of negative；
P-N Learning generates training sample by P-expert and N-expert, forms the new training sample marked
This collection, and the new training sample set marked is added in support vector machine classifier, update online object module and branch
Hold vector machine classifier.
The online object module constantly carries out the update of model over time, and training sample is continuously increased, mould
Type is constantly updated, and classification accuracy is higher and higher, and the target of acquisition is more accurate, and target position also determines that.
The image information or video information of the calibration of step (a-3) feedback include the doubtful Measuring error position of calibration
Image, video or speech prompt information.
The status information of step (a-4) current device includes: buying hour or maintenance record of equipment etc.；
The status information of step (a-4) current device, is on-site customer end and field device controller or intelligent network
The status information for the real-time device that pipe is obtained by wireless telecommunications.
The repair method at remote service end, includes the following steps:
Step (b-1): video flowing or image are received and is shown；
Step (b-2): the information that expert delineate to doubtful Measuring error position annotation is received；Receive repair prompt
Voice messaging；
Step (b-3): endorsement information will be delineated and synthesized with image, band is delineated to the target area image of endorsement information
It is cut；Then the target area image of cutting and voice messaging are sent to on-site customer end.
Beneficial effects of the present invention:
1 utilizes augmented reality, by the shape of expert feedback information, on-site customer acquisition video or image and current device
State information synthesizes augmented reality video, liberates client's both hands, and client is helped to be quickly found out problem.
2 clients while watching the video, can carry out real-time interactive voice with remote port expert, temporarily go out live
Existing problem, which timely feedbacks, gives remote port expert, and obtains the prompt and guidance of remote port expert in time.
Detailed description of the invention
Fig. 1 is the structural schematic diagram of inventive algorithm；
Fig. 2 is structural schematic diagram of the invention；
Fig. 3 is the structural schematic diagram at on-site customer end；
Fig. 4 is on-site customer end flow chart；
Fig. 5 is remote service end flow chart.
Specific embodiment
The invention will be further described with embodiment with reference to the accompanying drawing.
As shown in Figure 1, Fig. 1 is the structural schematic diagram of algorithm, the position for obtaining target is first had to, that is, in video tracking
The first frame of processing draws a circle to approve target, and with continually entering for video flowing, detector constantly detects target, and tracker constantly tracks
Target finally carries out tracker with the result of detector to merge acquisition by study so that the precision of detector is higher and higher
The position of target.
As shown in Fig. 2, Remote cooperation examination and repair system includes on-site customer end and remote service end, the floor trader
Family end and remote service end carry out data communication by wireless network.
The on-site customer end includes video camera, AR processing unit, wireless communication module, touch panel, display, language
Sound module (including client microphone and client loudspeaker), power supply module, as shown in figure 3, the on-site customer end can
To be smart phone, tablet computer or wearable device (such as AR intelligent glasses), the client communication module communication link
It is connected to on-site customer end, and server-side is connected to by telecommunication network.
The on-site customer end AR processing unit, calculating and processing for the augmented reality.
The on-site customer end communication unit receives institute for sending the video flowing or image to remote service end
State the expert feedback information of remote service end return.
The on-site customer end display unit, for show remote service end return expert calibration image information or
Video information.
The remote service end includes video image synthesis unit, server-side wireless communication module, display, voice mould
Block (including server-side microphone and server-side loudspeaker), the server-side communication module communication link then server-side host,
And client is connected to by telecommunication network.
The remote service end video image synthesis unit is hooked for expert at the doubtful Measuring error position of video image
Draw mark image synthesis.
The remote service end communication unit, for receiving the video flowing or image of client transmission, and to described existing
Field client sends expert feedback information.
The remote service end display unit, for showing the video or image information.
Implementation steps are as follows:
Firstly, service personnel carries out video acquisition or image to field scene using the video camera at on-site customer end at the scene
Shooting, on-site customer end sends the video of acquisition or image to remote service end by wireless telecommunications；
Secondly, the video that expert comes in the transmission of remote service end real time inspection on-site customer end, and using mouse in video
Or the doubtful maintenance position of image carries out delineating mark, and can carry out interactive voice by voice and field service personnel；
Finally, the feedback information that remote service end marks expert assists service personnel to complete back to on-site customer end
Service work to equipment.
Specific implementation step is as follows:
Step (1): as shown in the first frame of Fig. 4, service personnel treats setting for Measuring error using on-site customer end at the scene
It is standby to carry out video acquisition or image taking, video flowing or image needed for obtaining remote service end expert；The on-site customer end
Including the mobile wearable device such as smart phone, tablet computer and AR intelligent glasses；
Step (2): as shown in the second frame of Fig. 4, on-site customer end is remote by radioing to by the video flowing or image
Journey server-side；
Step (3): as shown in the first and second frame of Fig. 5, remote service end receives the video flowing or image and shows.Expert
Observation can be zoomed in and out to image or video as needed at remote service end, and feelings are understood by voice and Field Force
Condition, expert provide maintenance direction opinion according to priori knowledge.Illustrate without loss of generality, video that expert's viewing client-side is sent or
Image information carries out delineating mark at doubtful Measuring error position, can also carry out real-time language by microphone and on-site customer
Sound interaction, the work of real-time instruction Measuring error；
Step (4): as shown in Fig. 5 third frame, the video at the doubtful Measuring error position that expert is demarcated at remote service end
Or image and the speech prompt information are sent to on-site customer end；
Step (5): as shown in Fig. 4 third frame, on-site customer end receives the expert feedback information returned by remote service end,
The expert feedback information includes the image or video and specialist speech prompt information at the doubtful Measuring error position of expert's calibration；
Step (6): as shown in the 4th frame of Fig. 4, the augmented reality processing unit will receive expert feedback information and visitor
The image video at family end scene is completed the identification of object in expert's calibration object and live video by object tracking algorithm, is determined
Position and tracking, are superimposed the indicia patterns or text annotation of Remote calibration in image at the position of object, and are superimposed current
The status information of equipment, such as buying hour, the maintenance record of equipment, generate augmented reality image video；Described currently sets
Standby status information is on-site customer end and the real-time device that field device controller or intelligent network management are obtained by wireless telecommunications
Status information.
Step (7): as shown in the 5th frame of Fig. 4, finally augmented reality image audio video synchronization is shown and is worn in Field Force
In the on-site customer end equipment worn.
By above-mentioned implementation steps, to reach the mesh of the interactive remote expert cooperation maintenance based on augmented reality
's.
Step (6) the object tracking algorithm steps are as follows:
Step (6-1): the 1st frame of tracking process is identified in video image；It is loaded into the target area that band delineates markup information
Area image is as tracking target P；
Step (6-2): initialization support vector machine classifier and particle filter tracker；
Step (6-3): in the subsequent frame processing of video, it is loaded into the i-th frame video image, the value range of i is 2~n, n
Indicate the number of frame in video；All possible target P={ P is detected using support vector machine classifier1,P2,...,Pn,
Target is tracked using particle filter tracker, obtains position and the size of target area；
Step (6-4): the classification results of support vector machine classifier are mutually melted with the tracking result of particle filter tracker
It closes；The target position of detection positioning is obtained after fusion；Enter step (6-6)；
Step (6-5): completing the online updating of support vector machine classifier using P-N Learning learning strategy, returns
Step (a-4-3)；
Step (6-6): judging whether video has reached n-th frame, if so, terminating；Otherwise, i=i+1；It is transferred to step
(6-3)。
There are five types of possible altogether for step (6-4) fusion:
The first, support vector machine classifier and particle filter tracker have image block to be used as output, but supporting vector
Machine classifier has multiple similar image blocks to be determined out, the multiple to refer to greater than one；And particle filter tracker is only
A target position is found, arest neighbors classification, selection and previous frame target are carried out to testing result with normalized related coefficient
The highest image block of similarity is as last testing result；
Second, particle filter tracker does not have image block output, and support vector machine classifier has image block output, that
Testing result is clustered with normalized related coefficient to be divided again, using first cluster segmentation result as fusion knot
Fruit；
The third, particle filter tracker does not have image block output, and support vector machine classifier has image block output, such as
Fruit Clustering Decision-Making result is the largest the result of decision corresponding to related coefficient, but the result of decision is more than the threshold value model of setting
It encloses, then then initialization is re-started to particle filter tracker as fusion results using the result of decision, and lose and recognized originally
For correct sample；
4th kind, particle filter tracker has image block output, and support vector machine classifier does not export image block, that
Using the output result of particle filter tracker as fusion results；
5th kind, if support vector machine classifier and particle filter tracker are all exported without image block, then it is assumed that mesh
Mark disappears.
The online updating process of step (6-5) support vector machine classifier is:
In P-N Learning, P-expert and N-expert is defined.
P-expert find video sequence in time domain on Structural Characteristics, that is, time structure, and assume target be along
Trajectory line movement, i.e., the mobile very little between adjacent video frames, and there are certain correlations.
P-expert, which finds out those and is classified device, to be classified as negative but requires its classification to answer according to structural constraints
This is the sample of positive.
N-expert finds Structural Characteristics, that is, space structure in the spatial domain in video sequence, and assumes target one
It is only possible to appear in a position in a video frame.
N-expert, which finds out those and is classified device, to be classified as positive but requires its classification to answer according to structural constraints
This is the sample of negative.
P-N Learning generates effective training sample by P-expert and N-expert, forms new marked
Training sample set, and the sample set is added in support vector machine classifier, update online object module and support vector machines
Classifier.
Above-mentioned, although the foregoing specific embodiments of the present invention is described with reference to the accompanying drawings, not protects model to the present invention
The limitation enclosed, those skilled in the art should understand that, based on the technical solutions of the present invention, those skilled in the art are not
Need to make the creative labor the various modifications or changes that can be made still within protection scope of the present invention.
Claims (6)
1. on-site customer end, characterized in that include:
Video acquisition unit, the equipment for treating Measuring error carry out video acquisition or image taking；Obtain maintenance to be detected
The video flowing or image of equipment；The maintenance of equipment video flowing or image to be detected of acquisition are sent to the first communication unit；
Video flowing or image are sent to long-range clothes by wireless network for receiving video flowing or image by the first communication unit
Business end, and receive the feedback information at remote service end；The feedback information includes the mesh that voice messaging and band delineate markup information
Mark area image；
AR processing unit, the acquisition information and work as that feedback information, video acquisition unit for sending to remote service end are sent
The status information of preceding equipment synthesizes augmented reality video；The first display unit is sent to after synthesis；What video acquisition unit was sent
Acquisition information includes the equipment video flowing or image of maintenance to be detected；
The AR processing unit includes:
Particle filter tracker obtains position and the size of target area for tracking target；
Support vector machine classifier, for detecting all possible target；
Learn subelement, for completing the online updating of support vector machine classifier using P-N Learning learning strategy, makes
The nicety of grading of support vector machine classifier steps up；
Object module finds the highest sample of matching degree for being matched with sample to be sorted；
Subelement is merged, for mutually melting the classification results of support vector machine classifier with the tracking result of particle filter tracker
It closes, obtains the position of target；
First display unit is shown for the processing result to AR processing unit.
2. repair method used by on-site customer end as described in claim 1, characterized in that include the following steps:
Step (a-1): the equipment for treating Measuring error carries out video acquisition or image taking, obtains the view of maintenance of equipment to be detected
Frequency stream or image；
Step (a-2): the video flowing of the maintenance of equipment to be detected of acquisition or image are sent to remote service end；
Step (a-3): the feedback information sended over by remote service end is received；The feedback information includes voice messaging and band
Delineate the target area image of markup information；
Step (a-4): by object tracking algorithm complete the identification of target area in target area image and live video image,
Positioning and tracking are superimposed indicia patterns or text annotation at the scene, and are superimposed current at the position in target in video image region
The status information of equipment, to generate augmented reality image video；
Step (a-5): augmented reality image video is shown.
3. method according to claim 2, characterized in that step (a-4) the object tracking algorithm steps are as follows:
Step (a-4-1): the 1st frame of tracking process is identified in video image；It is loaded into the target area that band delineates markup information
Image is as tracking target P；
Step (a-4-2): initialization support vector machine classifier and particle filter tracker；
Step (a-4-3): in the subsequent frame processing of video, it is loaded into the i-th frame video image, the value range of i is 2~n, n table
Show the number of frame in video；All possible target P={ P is detected using support vector machine classifier1,P2,...,Pn, benefit
Target is tracked with particle filter tracker, obtains position and the size of target area；
Step (a-4-4): the tracking result of the classification results of support vector machine classifier and particle filter tracker is blended；
The target position of detection positioning is obtained after fusion；Enter step (a-4-6)；
Step (a-4-5): completing the online updating of support vector machine classifier using P-N Learning learning strategy, returns to step
Suddenly (a-4-3)；
Step (a-4-6): judging whether video has reached n-th frame, if so, terminating；Otherwise, i=i+1；It is transferred to step (a-
4-3)。
The examination and repair system 4. interactive remote expert based on augmented reality cooperates, characterized in that include: on-site customer end and
Remote service end；
The equipment that the on-site customer end is used to treat Measuring error carries out video acquisition or image taking, and the information of acquisition is passed
It is defeated by remote service end, and receives the target area image delineated at remote service end, by the target area image delineated and now
The video of field acquisition is synthesized, at the scene the target position of client real-time display maintenance to be detected；
The remote service end receives expert to the hook of video or image for receiving the video or image that on-site customer end is sent
Picture processing, is sent to on-site customer end for the target area image delineated；
On-site customer end, comprising:
Video acquisition unit, the equipment for treating Measuring error carry out video acquisition or image taking；Obtain maintenance to be detected
Equipment video flowing or image；The equipment video flowing of the maintenance to be detected of acquisition or image are sent to the first communication unit；
Video flowing or image are sent to long-range clothes by wireless network for receiving video flowing or image by the first communication unit
Business end, and receive the feedback information at remote service end；The feedback information includes the mesh that voice messaging and band delineate markup information
Mark area image；
AR processing unit, the acquisition information and work as that feedback information, video acquisition unit for sending to remote service end are sent
The status information of preceding equipment synthesizes augmented reality video；The first display unit is sent to after synthesis；What video acquisition unit was sent
Acquisition information includes the equipment video flowing or image of maintenance to be detected；
The AR processing unit includes:
Particle filter tracker obtains position and the size of target area for tracking target；
Support vector machine classifier, for detecting all possible target；
Learn subelement, for completing the online updating of support vector machine classifier using P-N Learning learning strategy, makes
The nicety of grading of support vector machine classifier steps up；
Object module finds the highest sample of matching degree for being matched with sample to be sorted；
Subelement is merged, for mutually melting the classification results of support vector machine classifier with the tracking result of particle filter tracker
It closes, obtains the position of target；
First display unit is shown for the processing result to AR processing unit.
5. examination and repair system as claimed in claim 4, characterized in that remote service end, comprising:
Second communication unit, for receiving the video flowing or image of the transmission of the first communication unit；By received video flowing or image
It is sent to video image synthesis unit；Video image synthesis unit feedback information is received, feedback information is sent to the first communication
Unit；The feedback information includes the target area image that voice messaging and band delineate markup information；
Video image synthesis unit synthesizes with image for that will delineate endorsement information, band is delineated to the target of endorsement information
Area image is cut；Then the target area image of cutting is sent to the second communication unit；
Second display unit for showing video flowing or image that the first communication unit sends over, while being also used to show view
The processing result of frequency image composing unit；
Second input unit delineates markup information and voice messaging for receiving, and will delineate markup information and voice messaging is sent
Enter video image synthesis unit.
6. repair method used by system as claimed in claim 4, characterized in that include the following steps:
Step (1): the equipment for treating Measuring error carries out video acquisition or image taking, obtains maintenance of equipment video flowing to be detected
Or image；
Step (2): the maintenance of equipment video flowing or image to be detected of acquisition are sent to remote service end by on-site customer end；
Step (3): remote service end receives the video flowing or image is simultaneously shown, receive expert to doubtful Measuring error position into
Row delineates the information of annotation；Receive the voice messaging of repair prompt；
Step (4): remote service end will delineate endorsement information and synthesize with image, and band is delineated to the target area of endorsement information
Image is cut；Then the target area image of cutting and voice messaging are sent to on-site customer end；
Step (5): on-site customer end receives the feedback information that remote service end sends over；The feedback information includes voice letter
Breath and band delineate the target area image of markup information；
Step (6): the target area image of markup information is delineated at on-site customer end to band, completes target by object tracking algorithm
Identification, positioning and the tracking of target area in area image and live video image, at the scene target in video image region
It is superimposed indicia patterns or text annotation at position, and is superimposed the status information of current device, to generate augmented reality image view
Frequently；Step (7): on-site customer end shows augmented reality image video.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610801442.5A CN106339094B (en) | 2016-09-05 | 2016-09-05 | Interactive remote expert cooperation examination and repair system and method based on augmented reality |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610801442.5A CN106339094B (en) | 2016-09-05 | 2016-09-05 | Interactive remote expert cooperation examination and repair system and method based on augmented reality |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106339094A CN106339094A (en) | 2017-01-18 |
| CN106339094B true CN106339094B (en) | 2019-02-26 |
Family
ID=57823708
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610801442.5A Active CN106339094B (en) | 2016-09-05 | 2016-09-05 | Interactive remote expert cooperation examination and repair system and method based on augmented reality |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106339094B (en) |
Families Citing this family (55)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110036356B (en) | 2017-02-22 | 2020-06-26 | 腾讯科技（深圳）有限公司 | Image processing in VR systems |
| CN107274431A (en) * | 2017-03-07 | 2017-10-20 | 阿里巴巴集团控股有限公司 | video content enhancement method and device |
| US10618137B2 (en) * | 2017-03-24 | 2020-04-14 | National Cheng Kung University | Automated constructing method of cloud manufacturing service and cloud manufacturing system |
| CN109218794B (en) * | 2017-06-30 | 2022-06-10 | 全球能源互联网研究院 | Remote operation guidance method and system |
| CN107333063A (en) * | 2017-07-14 | 2017-11-07 | 福建铁工机智能机器人有限公司 | A kind of method that utilization AR shoots film |
| CN107241579A (en) * | 2017-07-14 | 2017-10-10 | 福建铁工机智能机器人有限公司 | A kind of utilization AR realizes the method and apparatus of Telemedicine Consultation |
| CN107454365A (en) * | 2017-07-14 | 2017-12-08 | 福建铁工机智能机器人有限公司 | A kind of method that special reconnaissance is carried out using AR |
| CN107277429A (en) * | 2017-07-14 | 2017-10-20 | 福建铁工机智能机器人有限公司 | A kind of method that teleconference is carried out using AR |
| CN107277451A (en) * | 2017-07-14 | 2017-10-20 | 福建铁工机智能机器人有限公司 | A kind of utilization AR realizes the method and apparatus of remote guide scene investigation failure |
| CN107426508A (en) * | 2017-07-14 | 2017-12-01 | 福建铁工机智能机器人有限公司 | A kind of method that long-range security inspection is realized using AR |
| CN107240317A (en) * | 2017-07-14 | 2017-10-10 | 福建铁工机智能机器人有限公司 | A kind of utilization AR realizes the method and apparatus of long-distance education |
| CN107277450A (en) * | 2017-07-14 | 2017-10-20 | 福建铁工机智能机器人有限公司 | A kind of utilization AR realizes the method and apparatus of the live disaster relief of remote command |
| CN107329268A (en) * | 2017-08-03 | 2017-11-07 | 福建铁工机智能机器人有限公司 | A kind of utilization AR glasses realize the shared method in sight spot |
| CN107547554A (en) * | 2017-09-08 | 2018-01-05 | 北京枭龙科技有限公司 | A kind of smart machine remote assisting system based on augmented reality |
| CN107578110A (en) * | 2017-09-19 | 2018-01-12 | 北京枭龙科技有限公司 | A kind of repair message method for visualizing based on augmented reality |
| CN107872655A (en) * | 2017-11-21 | 2018-04-03 | 北京恒华伟业科技股份有限公司 | A kind of method and system for determining hidden danger point |
| CN107977716A (en) * | 2017-11-21 | 2018-05-01 | 北京恒华伟业科技股份有限公司 | The method and system that a kind of hidden danger point is shown |
| CN107948591A (en) * | 2017-11-21 | 2018-04-20 | 北京恒华伟业科技股份有限公司 | A kind of bridge information processing method and system |
| CN107957778A (en) * | 2017-11-21 | 2018-04-24 | 北京恒华伟业科技股份有限公司 | A kind of method and system of hidden danger point mark |
| CN108200331A (en) * | 2017-12-16 | 2018-06-22 | 苏州晨本智能科技有限公司 | It is a kind of based on the interactive robot of long-range explanation for following technology and VR technologies automatically |
| CN108170273A (en) * | 2017-12-28 | 2018-06-15 | 南京华讯方舟通信设备有限公司 | A kind of expert's remote guide system and implementation method based on hololens glasses |
| CN110097573A (en) * | 2018-01-31 | 2019-08-06 | 深圳市掌网科技股份有限公司 | A kind of combined tracking method and system based on augmented reality |
| WO2019164057A1 (en) * | 2018-02-23 | 2019-08-29 | (주)프론티스 | Server, method, and wearable device for supporting maintenance of military equipment in augmented reality technology using correlation rule mining |
| WO2019164056A1 (en) * | 2018-02-23 | 2019-08-29 | (주)프론티스 | Server, method and wearable device for supporting maintenance of military equipment on basis of binary search tree in augmented reality, virtual reality, or mixed reality based general object recognition |
| CN108965791B (en) * | 2018-04-04 | 2021-08-24 | 广州高新兴机器人有限公司 | Method and system for interacting with Internet of things equipment through robot AR camera |
| CN108632379A (en) * | 2018-05-11 | 2018-10-09 | 厦门柏讯信息科技有限公司 | A kind of automobile remote diagnostic and maintenance system based on panorama VR live streamings |
| CN108776444B (en) * | 2018-05-25 | 2021-09-03 | 江西博异自动化科技有限公司 | Augmented reality man-machine interaction system suitable for CPS automation control system |
| CN108769517B (en) * | 2018-05-29 | 2021-04-16 | 亮风台（上海）信息科技有限公司 | Method and equipment for remote assistance based on augmented reality |
| CN109034404B (en) * | 2018-06-27 | 2021-07-13 | 上海擎测机电工程技术有限公司 | AR (augmented reality) -glasses-based multi-user cooperative maintenance system and method for thermal power plant and AR glasses |
| CN108986556A (en) * | 2018-08-22 | 2018-12-11 | 高海龙 | A kind of aircraft remote technology clearance method, apparatus and smart machine |
| CN109300112A (en) * | 2018-08-27 | 2019-02-01 | 百度在线网络技术（北京）有限公司 | Instrument maintaining method and device based on AR |
| CN109635957A (en) * | 2018-11-13 | 2019-04-16 | 广州裕申电子科技有限公司 | A kind of equipment maintenance aid method and system based on AR technology |
| CN109725247B (en) * | 2018-12-18 | 2020-11-20 | 蒂姆维澳(上海)网络技术有限公司 | Circuit board remote maintenance guarantee method and system |
| CN109714575A (en) * | 2018-12-29 | 2019-05-03 | 广州德毅维才软件技术有限公司 | A kind of repair method based on internet remote command |
| CN116795464A (en) * | 2018-12-29 | 2023-09-22 | 中兴通讯股份有限公司 | Method for realizing remote assistance and related equipment |
| CN111614922A (en) * | 2019-02-22 | 2020-09-01 | 中国移动通信有限公司研究院 | Information interaction method, network terminal and terminal |
| CN110138831A (en) * | 2019-03-29 | 2019-08-16 | 亮风台（上海）信息科技有限公司 | A kind of method and apparatus carrying out remote assistance |
| CN110174423A (en) * | 2019-04-26 | 2019-08-27 | 云南电网有限责任公司电力科学研究院 | It is a kind of for X-ray detection it is long-range monitoring instruct system |
| CN110061755B (en) * | 2019-04-30 | 2021-04-16 | 徐州重型机械有限公司 | Operation assisting method and system, wearable device and engineering vehicle |
| CN112399125B (en) * | 2019-08-19 | 2022-06-10 | 中国移动通信集团广东有限公司 | Remote assistance method, device and system |
| CN111526118B (en) * | 2019-10-29 | 2023-06-30 | 南京翱翔信息物理融合创新研究院有限公司 | Remote operation guiding system and method based on mixed reality |
| CN111031284A (en) * | 2019-12-12 | 2020-04-17 | 国网浙江余姚市供电有限公司 | Infrastructure site management and control method, system and terminal |
| CN111260084B (en) * | 2020-01-09 | 2024-03-15 | 长安大学 | Remote system and method based on augmented reality cooperative assembly maintenance |
| CN113360057A (en) * | 2020-03-03 | 2021-09-07 | 中国移动通信集团贵州有限公司 | Remote interaction method and system based on augmented reality and electronic equipment |
| CN111294568B (en) * | 2020-03-13 | 2020-12-04 | 北京维卓致远医疗科技发展有限责任公司 | Method for realizing video fusion by medical remote control |
| CN113542332B (en) * | 2020-04-22 | 2023-04-07 | 中移智行网络科技有限公司 | Customer service video interaction method and equipment based on positioning labels |
| CN111784005A (en) * | 2020-06-17 | 2020-10-16 | 北京中铁电梯工程有限公司 | Elevator maintenance method, maintenance device, maintenance system and computer-readable storage medium based on Internet of things |
| CN111950506A (en) * | 2020-08-25 | 2020-11-17 | 中山大学 | MOCVD equipment maintenance assisting method and system based on AR technology |
| CN112130666A (en) * | 2020-09-16 | 2020-12-25 | 博彦科技（上海）有限公司 | Data processing method and system |
| WO2022061858A1 (en) * | 2020-09-28 | 2022-03-31 | Orange | Method for assisting a first user provided with a terminal at a scene |
| CN112509148A (en) * | 2020-12-04 | 2021-03-16 | 全球能源互联网研究院有限公司 | Interaction method and device based on multi-feature recognition and computer equipment |
| CN112565844B (en) * | 2020-12-04 | 2023-05-12 | 维沃移动通信有限公司 | Video communication method and device and electronic equipment |
| CN112492281A (en) * | 2020-12-10 | 2021-03-12 | 郑州捷安高科股份有限公司 | Remote auxiliary maintenance method, device, equipment and storage medium |
| CN113674426A (en) * | 2021-07-19 | 2021-11-19 | 中国电建集团西北勘测设计研究院有限公司 | Equipment maintenance method and device using AR technology |
| CN114063954A (en) * | 2021-11-11 | 2022-02-18 | 广西电网有限责任公司崇左供电局 | Interactive system for remote cooperation of communication operation and inspection operation |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015060393A1 (en) * | 2013-10-25 | 2015-04-30 | 独立行政法人産業技術総合研究所 | Remote action guidance system and processing method therefor |
| CN104834379A (en) * | 2015-05-05 | 2015-08-12 | 江苏卡罗卡国际动漫城有限公司 | Repair guide system based on AR (augmented reality) technology |
| CN105814500A (en) * | 2013-06-27 | 2016-07-27 | Abb技术有限公司 | Method and data presenting device for assisting a remote user to provide instructions |
-
2016
- 2016-09-05 CN CN201610801442.5A patent/CN106339094B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105814500A (en) * | 2013-06-27 | 2016-07-27 | Abb技术有限公司 | Method and data presenting device for assisting a remote user to provide instructions |
| WO2015060393A1 (en) * | 2013-10-25 | 2015-04-30 | 独立行政法人産業技術総合研究所 | Remote action guidance system and processing method therefor |
| CN104834379A (en) * | 2015-05-05 | 2015-08-12 | 江苏卡罗卡国际动漫城有限公司 | Repair guide system based on AR (augmented reality) technology |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106339094A (en) | 2017-01-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106339094B (en) | Interactive remote expert cooperation examination and repair system and method based on augmented reality | |
| CN108596974B (en) | Dynamic scene robot positioning and mapping system and method | |
| CN108171770B (en) | Facial expression editing method based on generative confrontation network | |
| CN107808004B (en) | Model training method and system, server and storage medium | |
| CN102831176B (en) | The method of commending friends and server | |
| CN109543549A (en) | Image processing method and device, mobile end equipment, server for more people's Attitude estimations | |
| CN104838420B (en) | Image content-based carries out image rotation to correct image orientation | |
| CN109284784A (en) | A kind of content auditing model training method and device for live scene video | |
| CN109903392A (en) | Augmented reality method and apparatus | |
| CN109299643A (en) | A kind of face identification method and system based on big attitude tracking | |
| CN112307886A (en) | Pedestrian re-identification method and device | |
| CN108491823A (en) | Method and apparatus for generating eye recognition model | |
| CN104777908B (en) | A kind of apparatus and method synchronously positioned for more people | |
| CN108073851A (en) | A kind of method, apparatus and electronic equipment for capturing gesture identification | |
| CN110009059A (en) | Method and apparatus for generating model | |
| CN112633409A (en) | Prediction method, longitudinal federal learning model training method and transverse federal learning model training method | |
| CN110827602A (en) | Cable joint manufacturing and operation and maintenance skill training device and method based on VR + AR technology | |
| CN109558605A (en) | Method and apparatus for translating sentence | |
| CN108521516A (en) | Control method and device for terminal device | |
| CN105353878A (en) | Augmented reality information processing method, device and system | |
| CN110516598A (en) | Method and apparatus for generating image | |
| Szajna et al. | The application of augmented reality technology in the production processes | |
| CN110162757A (en) | A kind of tableau format extracting method and system | |
| CN111665930A (en) | Multi-mode emotion recognition method and system integrating cloud and edge computing | |
| Shen et al. | Fast adaptive optimization of weighted vector median filters |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Shao Peng Inventor after: Zhang Zhen Inventor after: Zhu Chunjian Inventor after: Zhang Guodong Inventor after: Zhang Yan Inventor after: Liang Bo Inventor before: Zhang Zhen Inventor before: Zhu Chunjian Inventor before: Zhang Guodong Inventor before: Zhang Yan Inventor before: Liang Bo |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 250103 room 1-101, office building, 2269 development road, high tech Zone, Ji'nan, Shandong Patentee after: Shandong Wanteng Digital Technology Co.,Ltd. Address before: 250103 room 1-101, office building, 2269 development road, high tech Zone, Ji'nan, Shandong Patentee before: SHANDONG WANTENG ELECTRONIC TECHNOLOGY CO.,LTD. |