CN116627284A

CN116627284A - Controlling an optical indicator system by manipulating a physical object

Info

Publication number: CN116627284A
Application number: CN202310129126.8A
Authority: CN
Inventors: 库尔特·D·鲁布
Original assignee: Veltec Vision International Ltd
Current assignee: Veltec Vision International Ltd
Priority date: 2022-02-18
Filing date: 2023-02-17
Publication date: 2023-08-22
Also published as: DE102023201313A1

Abstract

A method and system for controlling projection of an optical indicator on a work surface is disclosed. According to the application, the projection means project the first optical indicator and the second optical indicator onto the work surface. The measurement system monitors a work area in which a work surface is disposed. The measurement system locates the control object within the workspace and identifies the markers disposed on the control object. The measurement system monitors the first arrangement and the second arrangement of the control object. The projection device projects a first optical indicator onto the work surface and the projection device projects a second optical indicator onto the work surface in response to the measurement system detecting manipulation of the control object between the first arrangement and the second arrangement.

Description

Controlling an optical indicator system by manipulating a physical object

Priority file

The priority of U.S. provisional patent application Ser. No. 63/311,690, having application date 2022, month 02, 18, is hereby incorporated by reference in its entirety.

Technical Field

The present application relates generally to a method of displaying an optical indicator using an optical projector. In particular, the present application aims to provide an enhanced user interface allowing control of optical indicators by manipulating physical objects.

Background

Optical indicators are projected onto the work surface to guide precise placement of parts, to locate features of interest in many manual assembly or manufacturing process tasks, such as placement of pre-cut carbon fiber sheets to assemble aircraft components, or inspection to correct defects in the surface or finish of parts. The optical indicators are typically displayed by transmitting a laser beam from a pair of rotating galvanometer mirrors to track the desired 3D pattern profile for use as assembly aids or templates on the part surface. If tracking is performed at a sufficiently fast laser speed, the optical indicator will appear as a stable image without flicker to guide the assembly or manufacturing process task.

Conventionally, as shown in fig. 1, an optical indicator such as a laser scanning template pattern is generated by a laser projector 10. The laser projector 10 is registered to the work surface 12 of the workpiece 14 by placing the retro-reflective targets 16 at precisely measured locations on the work surface 12. Retroreflective target 16 is scanned by the mirrors of the scanning system contained in laser projector 10 and the 3D position and orientation of projector 10 is calculated using a mathematical process commonly referred to as back-convergence. Once aligned, as shown in fig. 2, a template pattern 17 defining various process steps or corresponding workpiece 14 part features is accurately projected directly onto the work surface 12.

While this technique is suitable for certain applications, such as manufacturing composite aircraft components, it may be difficult in other applications to reach the resources involved in installing parts or tools and accurately measuring their target locations. This requires rapid alignment, laser projection and operator feedback.

To overcome the limitations of conventional laser projection systems, camera-based photogrammetry is currently integrated, allowing for direct measurement of the position on a part or tool surface relative to the projector. Such a system utilizing camera-based photogrammetry may be found in U.S. patent No. 9,200,899 (the contents of which are incorporated herein by reference). Referring to fig. 2, photogrammetry system 18 includes one or more cameras 20 that are electronically integrated with laser projector 10, typically through a controller or processor. The photogrammetry system 18 measures the position of the workpiece 14 by detecting retro-reflective targets, measuring features or work surface 12 of the workpiece 14, using a hand held probe 22, positioning patterns projected by the laser projector 10, or combinations thereof. The alignment of the laser projector 11 relative to the photogrammetry system 18 in the common three-dimensional coordinate system is then determined, allowing for accurate projection of the laser template or other assembly aid.

This method is also referred to in US patent 9,881,383 (the contents of which are incorporated herein by reference) in which the object position is monitored in real time by a photogrammetry system as the optical template is scanned by a laser to match the measured object motion path. This allows the optical template to be registered to the object surface so that when an operator picks up and manipulates the object to perform an assembly or inspection task, the optical template appears to be part of the object surface, thereby providing a high degree of usability for the assembly or inspection task.

Navigation through the user interface to provide instructions to the host computer to perform inspection or assembly tasks. Given the availability of using an optical projection system when an operator is typically near the projected image, returning to the host may be a major factor in completing assembly and inspection tasks in time. Thus, it is beneficial to signal the computer from a remote location. A hand-held remote control device may be used, but may be confusing if complex options are involved, such as providing multiple buttons on the remote control device or requiring repeated use of a few buttons. In addition, the remote control device may be lost, damaged or stolen.

To address such difficulties, a laser projector 10 such as that described in U.S. patent No. 5,957,559 (the contents of which are incorporated herein by reference) projects control functionality in which the projection process is controlled by detecting retroreflective patches for various selections. As shown in fig. 4, laser projector 10 projects icon patterns 26-36 that represent different modes of display assembly in the assembly of the prefabricated roof truss. By placing the retroreflective patch 24 in the path of the laser beam 38 from the laser projector 10, the operator 40 can select at a remote location a few feet from the host computer. The reflective patch is positioned over one of the icon patterns 26-36 within the laser beam and signals the host by reflecting the laser beam 42.

However, this type of projected menu has some known drawbacks. First, the projection menu increases the requirements on the projection system, reducing efficiency. The projected icon patterns 26-36 and other display patterns are displayed sequentially by the scan path of the laser beam 38, which may produce a flicker if the laser path projecting multiple icon patterns 26-36 for selection by the operator is too long. Second, it is necessary to activate the menu specification, typically to create special start-up features in the projected pattern at convenient intervals. In addition, it is necessary to project the menu onto a suitable surface (i.e., a flat surface at a suitable angle to the laser projector 10) for clear visualization. Interaction will typically fail if an operator's action moves the retroreflective patch 24 out of the laser beam 38 but not sensed (e.g., drags the projection feature to a desired location).

None of these systems can be used to provide a comprehensive interface between an operator and a controller to signal work task progress in real time. In view of this, it is desirable to develop a projection system that allows an operator to more fully control the progress of a work task while away from the controller.

Disclosure of Invention

The method and system of the present application provides operators with the ability to interact with the controller, thereby providing real-time feedback to the controller to perform and complete work tasks. The measurement system tracks movement of the control object to determine deployment of the work function on the work surface. In one embodiment, the measurement system monitors the type of movement of the control object, indicating that the work function is to be performed and subsequently completed. In an alternative embodiment, the operator is instructed to make a predetermined movement to signal to the measurement system the various stages of the work function, i.e. start, go, complete, etc. In each case, the measurement system signals the controller to modify the projection of the laser image onto the work surface to guide the operator to perform the next work function. Intended work functions include, but are not limited to, paint make-up, layering, machining, assembly, decal, and the like.

Drawings

Advantages of the present disclosure will become apparent and more readily appreciated by reference to the following detailed description when taken in conjunction with the accompanying drawings, wherein:

FIG. 1 illustrates a laser projector scanning a reflective target attached to a workpiece;

FIG. 2 illustrates scanning a template on a work surface of a work piece after the laser projector registers the work piece with respect to the laser projector;

FIG. 3 illustrates a laser projection system in combination with a photogrammetry system for registering a laser projector with respect to a workpiece;

FIG. 4 illustrates a prior art icon system;

FIG. 5 shows a first embodiment of the laser projection system of the present application;

FIG. 6 illustrates an alternative embodiment of a control object;

fig. 7 shows another alternative embodiment of the control object in the form of a glove.

Detailed Description

Referring to fig. 5, a first embodiment of the projection system of the present application is generally indicated by the reference numeral 60. The projection system 60 includes a projection device 62 for projecting a first optical image 64 onto a work surface 66 of a work piece 68, as described in more detail below. In one embodiment, projection device 62 takes the form of a laser projector, which will be implemented throughout for simplicity. However, other light projection devices are within the scope of the application. The laser projector 62 scans the first optical image 64 to direct the operator to a work function or an assembly function. The first optical image 64 may take the form of a laser template to identify the location where the assembly task is to be performed, such as where the lamina matches the work surface 66. Alternatively, the first optical indicator 64 identifies the defect, such as a surface defect identified on the work surface, by scanning an arrow or other icon configured to identify the location of the defect. Still further, the first optical image 64 may provide assembly instructions to an operator, for example, identifying components to match the work surface 64 based on predetermined assembly content.

The position of the laser projector 62 relative to the work surface 66 is registered in a common coordinate system by scanning the work surface markers 70 in a conventional manner. In one embodiment, the work surface indicia 70 is a retro-reflective target located on a fiducial or other predetermined location of the work piece 68 or work surface 66 to enable accurate projection of the first optical indicator 64. Alternative markers are also within the scope of the application, including but not limited to coded targets, spherical retro-reflective targets, and combinatorial markers.

The measurement device 72 takes the form of a photogrammetry system comprising a first camera 74 and a second camera 76. It will be appreciated by those of ordinary skill in the art that photogrammetry systems that include a single camera are also within the scope of the present application. Each camera 74, 76 includes an illumination source 78, 80 for illuminating a work area 82 in which the work piece 68 is disposed. In one embodiment, the illumination sources 78, 80 take the form of an array of light emitting diodes that flash at predetermined or periodic intervals based on the desired illumination rate within the work area 82. Each camera 74, 76 includes a light sensor 84, such as a CCD or CMOS sensor, including pixels for identifying reflected light from the illumination sources 78, 80, the purpose of which is described in more detail below.

The projection device 62 and the measurement device 72 are electrically connected via a controller 86. In one embodiment, the projection device 62 and the measurement device 72 are separate elements registered in a common coordinate system using techniques disclosed in U.S. patent No. 9,200,899 (the contents of which are incorporated herein by reference). Alternatively, the projection device 62 and the measurement device 72 are physically contained within a single housing, such as a VPS1 type laser projector sold by Virtek Vision International, inc, see U.S. patent No. US 9,881,383 (the contents of which are incorporated herein by reference). Alternative measurement systems, such as the iGPS system available from Nikon technology, in combination with conventional laser projection systems or standard video projection systems, may also be used, see U.S. patent No. 7,826,069 (the contents of which are incorporated herein by reference). In either embodiment, it is desirable to determine the position of the projection device 62 relative to the measurement device 72 to provide accurate image projection.

A control object 88 is also disposed within the workspace 82. Control indicia 90 are attached to control object 88. The control indicia 90 takes the form of retroreflective targets and functions in a known manner. In an alternative embodiment, a plurality of control objects 88 each include control indicia 90 and are disposed within workspace 82. In some cases, it may be beneficial to distinguish one control mark from another control mark by way of encoding or the like.

In one embodiment, more than one projection device 62 may be implemented to provide adequate coverage of the large workspace 82. It should be appreciated that the projection system 60 is capable of projecting optical indicators 64 onto one or more work surfaces 66 via one or more laser projectors 62. As described above, the optical indicator 64 may be an optical template pattern that directs placement of the assembly on the work surface 66, or take the form of a dot or similar indicator projected onto the magazine that directs the operator to select the location of the next part in coordination with the assembly task. As described above, the optical indicator 88 may also be used to identify defects on the protruding work surface 66, such as is described in co-pending U.S. patent application Ser. No. 17/497,211 (the contents of which are incorporated herein by reference). It should also be appreciated that the optical indicator 64 may perform any of a number of alternative projection-related functions without changing the scope of the present application.

One or more control objects 88 may be used to manipulate the optical indicator 66 and direct the controller 86 to switch between the first optical indicator and the second optical indicator to provide operator input between the first work function and the second work function, respectively. The measurement system 72 tracks the relative positions of the work surface 82 and the control object 88 by tracking the work surface markers 70 and the control markers 90. The indicia 70, 90 are selected based on the measurement system 72, and the measurement system 72 may include a reflective disc, electronic sensor, or other indicia (such as a printed QR code) to distinguish the indicia for use with the system. The markers 70, 90 are disposed at predetermined locations, such as fiducials, of the work service 66 and the control object 88, respectively.

Projection device 62 identifies indicia 70, 90 by laser beam 92 or other illumination, in cooperation with measurement system 72 to determine the position of work surface 66 and control object 88 within a common three-dimensional coordinate system for accurately projecting optical indicator 64. This is accomplished by reflecting the laser beam 92 to the laser sensor 94 in a known manner.

By the measuring device 72, the photogrammetry system works in conjunction with the markers 70, 90, allowing the optical indicator 64 to remain substantially stationary relative to the work surface 66 even if the work surface 66 is moved or manipulated. In this manner, the measurement device 72 provides motion compensation and adjusts for drift in real time even when the work surface 66 is dynamically or accidentally moved.

As described above, during operation, the control object 88 is disposed within the workspace 82. The measurement device 72 passively tracks the control object 88 without the projection device 62 scanning the laser beam 92 over the control mark 90. The operator feedback display 94 facilitates the use of the control object 88, and the projection device 62 dynamically projects the operator feedback display 94 onto the surface of the control object 88 as it is maneuvered. As the measurement device 72 monitors the operator's movement of the control object 88, the projection device 62, alone or in combination with the optical indicator 64 projected onto the work surface 66, can dynamically project a feedback display 94 onto the control object 88 even if the control device 62 is moving.

In one embodiment, rotation of control object 88 is used to select a mode of operation, such as file loading or progression through a series of optical indicators 64. The measurement device 72 tracks the rotation or other movement of the control object 88 and signals the controller 72 for interpretation. Once interpreted, the current mode of operation is indicated by a feedback display 94 projected onto the control object 88, the control object 88 being a file icon projected by the projection device 62. Once the current mode of operation is displayed on control object 88 to verify to the operator which mode has been implemented, the scale of feedback display 94 is temporarily modified to provide feedback to the operator that the selection has been made in a manner similar to the exaggerated representation of the button being pressed. When the appropriate mode of operation is displayed on the control object 88, the control object 88 may be clicked left or right, or some other movement of the progress signal of the optical indicator 64 (e.g., the first optical indicator and the second optical indicator) may be signaled, or the controlled click object 88 may be lifted to indicate that a process step has been completed. Such movement of control object 88 is by way of example only. Alternative movements may be utilized to signal the controller 86 via the measurement device 72 to initiate or complete an operational function. When the control object 88 is released and again stationary, the feedback display 94 is no longer projected and the measurement system 72 passively returns to monitoring the movement and position of the control object 88.

While the previous embodiments illustrate one use of control object 88, alternative implementations of control object 88 and methods of manipulation or interaction between control object 56 and projection system 60 are also possible. For example, reference numeral 96 generally indicates an alternative embodiment of an optical indicator system, i.e., projection system 60 includes an alternative control object. In this embodiment, an alternative marking 98 is attached to the back of glove 100. Glove 100 is worn by an operator, essentially converting the operator's hand into an alternative control object 96. As the back of glove 100 is exposed to measuring device 72, measuring device 72 begins to track glove 100, enabling projection device 62 to project feedback display 94 directly onto glove 100. The operator may also swipe the glove to the right or left to indicate the inspection status of the defect location or completion of the assembly task currently identified by the optical indicator 64. The operator may also reduce progress of glove 100 to project a subsequent defect location to be inspected or assembly step, initiating a subsequent step identified by projecting a next or subsequent optical indicator 64. Still further, the sleeve may partially cover the alternative marker 98, for example, or the movement of the measurement device 62 following the alternative marker 98 may be too abrupt, resulting in termination of the projection interaction between the projection system 60 and the operator.

Reference numeral 102 in fig. 7 shows another alternative embodiment of the control object. In this embodiment, the control object 102 is attached to a stationary arrangement within the workspace 82. The alternative control object 102 includes alternative markers 104 defining a plurality of reflection points 106 monitored by the measurement device 72. In this embodiment, when the operator covers one or more reflection points 106, the control object 102 signals the measurement device 72 of the work function deployment. As described above, the projection device 62 scans the feedback display 94 onto the control object 102. Thus, the control object 102 interacts with the operator in a manner similar to the touch screen interaction with the controller 86. In this manner, the control object 102 is attached to a fixed location within the workspace 82, preventing the control object 102 from being lost or misplaced.

In yet another embodiment, the projection system 60 of the present application utilizes a workpiece as the control object 56. The workpiece includes a work surface 66, such as the surface of a seat cushion or other object, that requires templated instructions to instruct an operator to complete a series of assembly tasks. Thus, the work piece onto which the optical indicator 64 is projected serves as a control object. In a similar manner as described above, the measurement device 72 passively monitors and tracks the work surface 66. After the operator has completed the assembly task, the work piece 68 itself is rotated, tapped or lifted to select an operational mode, processed through an optical indicator, or to indicate completion of various processing steps.

It is also possible for the projection system 60 to be trained to monitor the assembly movement of the workpiece 68 as a queue during assembly, or to indicate completion of various processes or assembly steps. In this way, the operator may signal the status of the assembly process to the controller 86 via the measurement device 72 without revealing additional movement of the workpiece 68. Thus, the optical indicator system identifies certain workpiece movements that indicate that a task is complete. Upon completion, the projection device 62 projects a subsequent or second optical indicator corresponding to a subsequent or second assembly task. For example, rotating or lifting the seat cushion after the first snap fastener assembly is mounted to the seat cushion will signal the measuring device 72 to project a subsequent template pattern corresponding to the second snap fastener assembly and seat cushion to be mounted. Thus, utilizing a part or component as the control object 88 further improves the efficiency associated with controlling the projection of the optical indicator 64.

Sometimes, when the optical indicator 64 is projected onto the work surface 66, it may be necessary to adjust or align its projected position. Thus, the face markings 70 may be individually manipulated on the face 66 to adjust and fix the optical indicators 70 for accurate alignment within the face 66. If the measurement device 72 detects the control surface markings 70, the projection device 62 automatically projects an alignment template pattern configured to help identify important features of the work surface 66 to assist an operator in manually aligning the work surface markings 70. It should be appreciated that the alignment template pattern 64, in turn, moves simultaneously with the work surface markings 70, allowing the operator to properly place the work surface markings 70 on the work surface 66 to efficiently calibrate the projection device 62 in a common coordinate system with the workpiece 68.

Other variations for properly aligning or adjusting the optical indicator 64 on the work surface 66 are possible without changing the scope of the application. For example, the projection device 62 may first project an alignment template pattern onto a predetermined location in the work area 82 and the work surface 66. From these projections, manual alignment of the workpiece 68 is achieved, followed by manual alignment of the face marks 70, which are then placed in alignment with the matching template pattern 64. Alternatively, the face marker 70 may first be placed on the face 66 and moved by manual manipulation to match the alignment template pattern projected by the projection device 62 based on the original placement of the one or more markers 70.

The present application provides an operator with the unique ability to provide real-time feedback to the controller at a remote location to accurately track a series of manufacturing events. In the example of glove 100 serving as control object 96, measurement device 72 tracks glove 100, such as with an index finger in proximity to identify paint defects. In this embodiment, the laser projector 62 identifies target objects along the path the operator is pointing to but that the operator is not currently able to reach, such as paint defects or other objects that were originally automatically inspected for omission. Once glove 100, or more specifically alternative control indicia 98, is detected, feedback is provided to the operator by projecting feedback display 94 onto the back of glove 100. In one embodiment, the defect ID or the assembled status is projected as a feedback display 94. Once the task is completed, the operator need only swipe the hand, detect hand movement by the measurement device 72, and alter the state at the controller 86. A feedback display 94 may also be projected onto the work surface 66 to indicate the status of the task. The status of all required and completed tasks is tracked by the controller or by cloud storage and the file is updated to automatically provide electronic monitoring of a series of required tasks. When the complete sequence is complete, a feedback display 94 indicating completion is projected onto the control object 88, 90, 104 or the work surface 66 as desired.

The laser projector 62 is capable of simultaneously emitting a plurality of optical indicators 64 and a feedback display 94. For example, a crosshair may be projected onto the work surface 66 to identify the location where a weld stud is desired. The first operator marks the crosshair position with a center punch. A second operator installs the weld stud at the marked location. Any multiple operators coordinate work simultaneously, and the operators manipulate control objects 88, 96, 104 to trigger a sequence of projected feedback displays 94.

The application has been described in an illustrative manner, and it is to be understood that the terminology which has been used is intended to be in the nature of words of description rather than of limitation. Many modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the application may be practiced otherwise than as specifically described.

Claims

1. A method of controlling projection of an optical indicator on a work surface, the method comprising:

providing a projection device for projecting a first optical indicator and a second optical indicator onto a work surface, and providing a measurement system for monitoring a work area in which the work surface is arranged;

the measurement system locates a control object within the workspace and identifies a marker disposed on the control object;

the measurement system monitors a first arrangement and a second arrangement of the control object;

the projection device projects the first optical indicator onto the work surface; and

the projection device projects the second optical indicator onto the work surface in response to the measurement system detecting manipulation of the control object between the first arrangement and the second arrangement.

2. The method of claim 1, wherein the step of providing a projection device is further defined as: a laser projector is provided.

3. The method of claim 1, wherein the step of providing a measurement system is further defined as: a photogrammetry apparatus is provided that includes a camera.

4. The method of claim 1, wherein the step of projecting the second optical indicator in response to manipulating the control object is further defined as: the control object is physically moved.

5. The method of claim 1, wherein the step of projecting the second optical indicator in response to manipulating the control object is further defined as: and partially shielding the control object.

6. The method of claim 1, further comprising the step of: altering the first and second optical indicators by altering the arrangement of the marks.

7. The method of claim 1, further comprising the step of: the projection means projects a feedback display onto the control object.

8. The method of claim 1, wherein the step of locating a control object within a workspace is further defined as: and identifying the object containing the working surface as the control object.

9. The method of claim 7, wherein the step of projecting the feedback display onto the control object surface is further defined as: the projection means follows the movement of the control object such that the feedback display projects substantially stationary relative to the control object.

10. The method of claim 7, further comprising the step of: the feedback display is modified in response to manipulating the control object to indicate a change in the first and second optical indicators.

11. The method of claim 1, wherein the control object comprises a reflector attached to a work glove.

12. A projection system for projecting and controlling an optical indicator on a work surface, the projection system comprising:

projection means for projecting the first optical indicator and the second optical indicator onto a work surface;

a control object identified by a mark arranged on the control object;

a measurement system for monitoring a first arrangement and a second arrangement of the control object;

a controller for storing the first and second optical indicators and for communicating with the projection device and the measurement system; and

wherein the measurement system detects manipulation of the control object between the first arrangement and the second arrangement and communicates with the controller such that the projection device cycles between projecting the first optical indicator and the second optical indicator to the work surface.

13. The system of claim 12, wherein the projection device comprises a laser projector.

14. The system of claim 12, wherein the measurement system comprises a photogrammetry device.

15. The system of claim 14, wherein the photogrammetry device comprises a camera including a pixelated light sensor.

16. The system of claim 12, wherein the control object comprises at least one of a glove, a tool, or a hand-held object.

17. The system of claim 16, wherein the control object comprises a control mark comprising a retro-reflective band.

18. The system of claim 12, wherein the measurement device comprises an illumination source for illuminating the work area.

19. The system of claim 18, wherein the measurement system detects that the illumination source produces light that is in turn reflected from the control mark.

20. The system of claim 19, wherein the measurement device detects movement of the control mark from light reflected from the control mark to a camera defining the measurement device, and the measurement device signals such movement to the controller so that the controller monitors the start and completion of a manufacturing task.

21. The system of claim 12, wherein the projection device is configured to project a feedback display onto the control object in response to movement of the control device.

22. The system of claim 21, wherein the measurement device tracks movement of the control object and signals the position of the control object to the projection device via the controller, thereby enabling the projection device to continuously project a feedback display onto the control object.