CN113419630B - An adaptive occlusion removal method based on projection AR - Google Patents

Abstract

The invention discloses an adaptive occlusion elimination method based on projection AR. The method senses the real-time state of the assembly scene through the cooperative work of a camera-projector, and performs real-time detection and tracking of occluded targets and occluded projection information in the projection light path. and positioning, through the collaborative work of multiple projectors to compensate the occluded information in real time. The invention can improve the versatility of the blocking target detection algorithm, judge separately whether the blocking target moves, and improve the efficiency and precision of blocking target detection.

Description

An adaptive occlusion removal method based on projection AR

technical field

The invention belongs to the technical field of augmented reality, and in particular relates to an adaptive occlusion elimination method.

Background technique

In the field of aerospace, aircraft products have the characteristics of complex structure, many types and quantities of parts, and high precision requirements. Traditional manual assembly is difficult and inefficient, and at the same time requires high comprehensive quality of workers.

As augmented reality technology (Augment Reality, AR) begins to be applied to various mechanical assembly scenarios. Using AR technology, virtual information can be superimposed on the real scene, and the established virtual environment can provide more concentrated and comprehensive assembly information. The auxiliary assembly system based on AR technology can conduct unified and centralized management of a large amount of information required for assembly, greatly shorten assembly lead time, reduce waste of resources and reduce costs. Matching and integrating the virtual environment with the real environment expands the channels and scope of information acquisition for the staff, enhances the perception of important information, and reduces the requirements for the staff's visual and spatial capabilities; In terms of work, it can greatly reduce assembly errors, shorten assembly time, and improve assembly efficiency and assembly quality.

Projection-based AR technology, as one of the AR visualization methods, does not require the operator to wear any equipment during its realization, so it will not have any constraints on the operator; in addition, projection AR integrates real objects, virtual objects, and interactive environments into one Therefore, the interactive form of projected AR is more natural and diverse; finally, with the change of the number and position of projectors, projected AR can expand its display range almost infinitely. Based on the above advantages, projection AR technology is widely used in the field of auxiliary assembly of complex products such as aerospace.

However, the spatial augmented reality technology (Projector-based AugmentedReality, PBAR) based on projection display is often accompanied by the disadvantage of insufficient mobility. Due to the relatively fixed position of the projector, entities such as the operator's hands and mechanical parts to be assembled often have the problem of blocking the guidance information during the auxiliary assembly guidance process. At present, there are few studies on the problem of projection occlusion and there are serious deficiencies.

According to the research status at home and abroad, at present, in view of the projection occlusion problem in projection AR, the method of multi-projection is used to eliminate shadows at home and abroad. That is, after the occluded area is identified and detected, another projector is used to project from another angle to perform supplementary projection on the occluded area. At present, the research focus of scholars at home and abroad is mainly on the detection, identification and positioning of occluders and shadow areas. The research on multi-projection collaborative work is still shallow, and there are still the following deficiencies:

1) The identification and detection methods for occluders and shadow areas are not accurate enough, most of which are based on the establishment of skeleton models for human body occlusion, which is not general and versatile; there is no direct connection between occluders and occluded shadow areas, and occlusion needs to be identified in the depth camera On the basis of objects, the shadow area on the projection surface is mapped through spatial information transformation.

2) The projection screen of the supplementary projection is not positioned and aligned with the projection screen of the main projector, so the spatial position of the supplementary projection screen may be disordered, and visual interference such as double images may appear in the projection effect.

3) The projection screen of the supplementary projection has not been processed, and the projection screen needs to be segmented before the supplementary projection, and only the occluded information should be compensated, otherwise the unoccluded projection screen will be visually disturbed by the reprojection of multiple projectors above discomfort.

Contents of the invention

In order to overcome the deficiencies of the prior art, the present invention provides an adaptive occlusion elimination method based on projection AR. The method realizes the real-time state perception of the assembly scene through the cooperative work of the camera and the projector, and detects the occluded target and the object in the projection light path. Real-time detection, tracking and positioning of occluded projection information, and real-time compensation for occluded information through the collaborative work of multiple projectors. The invention can improve the versatility of the blocking target detection algorithm, judge separately whether the blocking target moves, and improve the efficiency and precision of blocking target detection.

The technical solution adopted by the present invention to solve its technical problems is as follows:

Step 1: Build an AR guided assembly system for fasteners, including an image acquisition client for scene state perception, a projection client for AR visualization display, and a virtual-real registration client for multi-projection screen positioning;

Establish a three-dimensional model of fasteners, establish a visual scheme for guiding the installation of fasteners based on process information, and superimpose process information in the form of videos and patterns on the holes to be installed to give workers installation guidance;

Step 2: Establish the data transmission and communication architecture between the three clients of the fastener AR guided assembly system, and the data transmission between each client through the Socket communication of the TCP network protocol;

Step 3: Start the Basler industrial camera and projector, and jointly calibrate the Basler industrial camera and projector through the Zhang Zhengyou calibration method; solve the internal and external parameters of the camera, and calculate the internal and external parameters of the projector to realize the virtual coordinate system and the camera coordinate system Unification with the projected coordinate system;

Step 4: Place four artificial identification codes based on ellipse features in the area to be assembled, and use the camera-projector joint calibration technology to locate and align the multi-projection images, and realize the virtual and real registration function;

Step 5: Establish the coordinate relationship between the projection interaction menu of the fastener AR guided assembly system and the menu in the Unity scene. The main projector projects the projection interaction menu of different fastener models. The operator clicks on the interaction menu with a finger, and the system makes Respond and project the process information of the corresponding type of fastener, and the operator installs the fastener according to the guiding information;

Step 6: Basler industrial cameras monitor the assembly scene in real time; the user selects the scene range to be monitored by setting the ROI area of interest to improve system response speed and detection accuracy; Basler industrial cameras monitor whether the user clicks on the interactive menu on the one hand On the other hand, judge whether the projection information is blocked in real time, and track and locate the blocked target in real time;

Step 7: If an occluded target appears in the scene, the system will segment and extract the occluded guide information image through image processing technology according to the coordinate information of the occluded target and the shadow area on the projection surface; Select an unoccluded supplementary projector to perform supplementary projection from another angle, compensate for the guiding information, and realize adaptive elimination of occlusion;

Step 8: Due to the existence of shadows produced by the occluder on the projection surface, the brightness of the supplementary projected image is lower than that of other unoccluded areas, and the system performs photometric compensation processing on the projected image;

The system knows the pixel coordinates of the shadow area, and adjusts the contrast and brightness of the image in the shadow area through image processing, so that the projection screen remains consistent in brightness and color;

Step 9: According to the guidance information, the assembly operator clicks on different projected interactive menus in turn to complete the installation of fasteners step by step.

The beneficial effects of the present invention are as follows:

1. The present invention can adjust and limit the projection position of the projector by controlling the position of the artificial marker, and can be used for accurate projection of the projector under mobile projection and prevention of projection shake and other issues;

2. The present invention can effectively solve the problem of occlusion of projection information during the working process of the projector through the cooperative work of the camera and the projector, improve the visual effect of projection, and improve the efficiency of information guidance.

Description of drawings

Fig. 1 is a schematic diagram of the assembly system of the present invention.

Fig. 2 is a schematic diagram of adaptive elimination of projection occlusion according to the present invention.

FIG. 3 is a schematic diagram of multi-projection screen positioning in the present invention.

FIG. 4 is a flow chart of the algorithm for detecting and tracking occluded targets in the present invention.

Fig. 5 is the interactive interface and visualization form of the present invention.

Fig. 6 is a comparison diagram of effects before and after occlusion compensation according to the present invention.

Fig. 7 is a comparison diagram of the effect before and after photometric compensation according to the present invention.

Fig. 8 is the fastener installation of the present invention before shielding compensation.

Fig. 9 is the fastener installation of the present invention after occlusion compensation.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

Aiming at the problems existing in the prior art, the present invention provides an adaptive occlusion elimination method based on projection AR, which mainly realizes fastener auxiliary guidance installation, adaptive elimination of projection guidance information occlusion and human-computer interaction function of projection menu, specific steps as follows:

Step 1: Build an AR guided assembly system for fasteners, including an image acquisition client for scene state perception, a projection client for AR visualization display, and a virtual-real registration client for multi-projection screen positioning;

Establish a three-dimensional model of fasteners, establish a visual scheme for guiding the installation of fasteners based on process information, and superimpose process information in the form of videos and patterns on the holes to be installed to give workers installation guidance;

Step 2: Establish the data transmission and communication architecture between the three clients of the fastener AR guided assembly system, and the data transmission between each client through the Socket communication of the TCP network protocol;

Step 3: Start the Basler industrial camera and projector, and jointly calibrate the Basler industrial camera and projector through the Zhang Zhengyou calibration method; solve the internal and external parameters of the camera, and calculate the internal and external parameters of the projector to realize the virtual coordinate system and the camera coordinate system Unification with the projected coordinate system;

Step 4: Place four artificial identification codes based on ellipse features in the area to be assembled, and use the camera-projector joint calibration technology to locate and align the multi-projection images, and realize the virtual and real registration function;

Step 5: Establish the coordinate relationship between the projection interaction menu of the fastener AR guided assembly system and the menu in the Unity scene. The main projector projects the projection interaction menu of different fastener models. The operator clicks on the interaction menu with a finger, and the system makes Respond and project the process information of the corresponding type of fastener, and the operator installs the fastener according to the guiding information;

Step 6: Basler industrial cameras monitor the assembly scene in real time; the user selects the scene range to be monitored by setting the ROI area of interest to improve system response speed and detection accuracy; Basler industrial cameras monitor whether the user clicks on the interactive menu on the one hand On the other hand, judge whether the projection information is blocked in real time, and track and locate the blocked target in real time;

Step 7: If an occluded target appears in the scene, the system will segment and extract the occluded guide information image through image processing technology according to the coordinate information of the occluded target and the shadow area on the projection surface; Select an unoccluded supplementary projector to perform supplementary projection from another angle, compensate for the guiding information, and realize adaptive elimination of occlusion;

Step 8: Due to the existence of shadows produced by the occluder on the projection surface, the brightness of the supplementary projected image is lower than that of other unoccluded areas, and the system performs photometric compensation processing on the projected image;

The system knows the pixel coordinates of the shadow area, and adjusts the contrast and brightness of the image in the shadow area through image processing, so that the projection screen remains consistent in brightness and color;

Step 9: According to the guidance information, the assembly operator clicks on different projected interactive menus in turn to complete the installation of fasteners step by step.

The present invention can effectively solve the following problems in the existing projection occlusion solution technology:

1. Improve the versatility of the occlusion target detection algorithm, judge whether the occlusion target moves separately, and improve the efficiency and accuracy of occlusion target detection;

2. Directly obtain the coordinate position information of the shadow area on the projection surface, no longer need to be calculated through spatial coordinate transformation;

3. Perform positioning and alignment processing on multi-projection images, pre-process supplementary projection images and source projection images to reduce visual interference such as ghosting; perform photometric compensation on images to solve the problem of inconsistency in brightness of projection images caused by shadows.

Claims (1)

1. A projection AR-based adaptive occlusion elimination method is characterized by comprising the following steps:

step 1: constructing a fastener AR guide assembly system, which comprises an image acquisition client for scene state perception, a projection client for AR visual display and a virtual-real registration client for multi-projection picture positioning;

establishing a three-dimensional model of the fastener, establishing a fastener guide installation visualization scheme according to process information, and overlaying the process information to a hole site to be installed in the form of videos and patterns to provide installation guidance for workers;

and 2, step: establishing a data transmission communication framework among three clients of the fastener AR guide assembly system, and performing data transmission among the clients through Socket communication of a TCP network protocol;

and 3, step 3: starting the Basler industrial camera and the projector, and carrying out combined calibration on the Basler industrial camera and the projector by a Zhang-Zhengyou calibration method; solving the internal and external parameters of the camera, and calculating the internal and external parameters of the projector to realize the unification of a virtual coordinate system, a camera coordinate system and a projection coordinate system;

and 4, step 4: placing four artificial identification codes based on elliptical characteristics in an area to be assembled, positioning and aligning a multi-projection picture through a camera-projector combined calibration technology, and realizing a virtual and real registration function;

and 5: establishing coordinate relation between a projection interactive menu of the AR guide assembly system and menus in the Unity scene, projecting projection interactive menus of different fastener types by the main projector, clicking the interactive menus by an operator through fingers, responding by the system and projecting process information of fasteners of corresponding types, and installing the fasteners by the operator according to the guide information;

step 6: carrying out real-time scene monitoring on an assembly scene by a Basler industrial camera; a user selects a scene range to be monitored by setting an ROI (region of interest) so as to improve the response speed and the detection precision of the system; the Basler industrial camera monitors whether a user clicks an interactive menu on one hand, judges whether projection information is shielded or not in real time on the other hand, and tracks and positions a shielded target in real time;

and 7: if an occlusion target appears in the scene, the system segments and extracts the occluded guide information image through an image processing technology according to the occlusion target and the coordinate information of the shadow area on the projection surface; the system autonomously selects an unblocked supplementary projector for supplementary projection from another angle by detecting and tracking the blocked target and the shadow area, compensates the guide information and realizes the self-adaptive elimination of the blocking;

and 8: because the shadow generated by the shielding object on the projection surface exists, the brightness of the image after supplementary projection is lower than that of other non-shielded areas, and the system performs luminosity compensation processing on the projected image;

the system knows the pixel coordinates of the shadow area, and adjusts the contrast and the brightness of the shadow area image through image processing so that the projection picture is kept consistent in brightness and color;

and step 9: and the assembly operator sequentially clicks different projection interactive menus according to the guide information to gradually complete the installation of the fastening piece.

Images