WO2019119022A1 - Augmented visual assistance system for assisting a person working at a remote workplace, method and headwear for use therewith - Google Patents

Augmented visual assistance system for assisting a person working at a remote workplace, method and headwear for use therewith Download PDF

Info

Publication number
WO2019119022A1
WO2019119022A1 PCT/AU2018/051230 AU2018051230W WO2019119022A1 WO 2019119022 A1 WO2019119022 A1 WO 2019119022A1 AU 2018051230 W AU2018051230 W AU 2018051230W WO 2019119022 A1 WO2019119022 A1 WO 2019119022A1
Authority
WO
WIPO (PCT)
Prior art keywords
electric signal
knowledge
communications module
workplace
projector
Prior art date
Application number
PCT/AU2018/051230
Other languages
French (fr)
Inventor
Terrence John Lancaster
Original Assignee
Ehatsystems Pty Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from AU2017905104A external-priority patent/AU2017905104A0/en
Application filed by Ehatsystems Pty Ltd filed Critical Ehatsystems Pty Ltd
Publication of WO2019119022A1 publication Critical patent/WO2019119022A1/en

Links

Classifications

    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01Head-up displays
    • G02B27/017Head mounted
    • GPHYSICS
    • G03PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03BAPPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B17/00Details of cameras or camera bodies; Accessories therefor
    • G03B17/48Details of cameras or camera bodies; Accessories therefor adapted for combination with other photographic or optical apparatus
    • G03B17/54Details of cameras or camera bodies; Accessories therefor adapted for combination with other photographic or optical apparatus with projector
    • AHUMAN NECESSITIES
    • A42HEADWEAR
    • A42BHATS; HEAD COVERINGS
    • A42B3/00Helmets; Helmet covers ; Other protective head coverings
    • A42B3/04Parts, details or accessories of helmets
    • A42B3/0406Accessories for helmets
    • A42B3/042Optical devices
    • AHUMAN NECESSITIES
    • A42HEADWEAR
    • A42BHATS; HEAD COVERINGS
    • A42B3/00Helmets; Helmet covers ; Other protective head coverings
    • A42B3/04Parts, details or accessories of helmets
    • A42B3/30Mounting radio sets or communication systems
    • FMECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F41WEAPONS
    • F41HARMOUR; ARMOURED TURRETS; ARMOURED OR ARMED VEHICLES; MEANS OF ATTACK OR DEFENCE, e.g. CAMOUFLAGE, IN GENERAL
    • F41H1/00Personal protection gear
    • F41H1/04Protection helmets
    • FMECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F41WEAPONS
    • F41HARMOUR; ARMOURED TURRETS; ARMOURED OR ARMED VEHICLES; MEANS OF ATTACK OR DEFENCE, e.g. CAMOUFLAGE, IN GENERAL
    • F41H11/00Defence installations; Defence devices
    • F41H11/12Means for clearing land minefields; Systems specially adapted for detection of landmines
    • GPHYSICS
    • G03PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03BAPPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B29/00Combinations of cameras, projectors or photographic printing apparatus with non-photographic non-optical apparatus, e.g. clocks or weapons; Cameras having the shape of other objects
    • GPHYSICS
    • G03PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03BAPPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B31/00Associated working of cameras or projectors with sound-recording or sound-reproducing means
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F1/00Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
    • G06F1/16Constructional details or arrangements
    • G06F1/1613Constructional details or arrangements for portable computers
    • G06F1/163Wearable computers, e.g. on a belt
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00Teaching not covered by other main groups of this subclass
    • G09B19/003Repetitive work cycles; Sequence of movements
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00Teaching not covered by other main groups of this subclass
    • G09B19/24Use of tools
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00Electrically-operated educational appliances
    • G09B5/06Electrically-operated educational appliances with both visual and audible presentation of the material to be studied
    • G09B5/067Combinations of audio and projected visual presentation, e.g. film, slides
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01Head-up displays
    • G02B27/0101Head-up displays characterised by optical features
    • G02B2027/0138Head-up displays characterised by optical features comprising image capture systems, e.g. camera
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00Electrically-operated educational appliances
    • G09B5/06Electrically-operated educational appliances with both visual and audible presentation of the material to be studied

Definitions

  • Augmented visual assistance system for assisting a person working at a remote workplace, method and headwear for use therewith
  • the present invention relates to the remote instruction or training of people in practical skills on the job rather than in the classroom, and headwear for facilitating such training.
  • Preferred embodiments of the invention can provide a way to efficiently provide assistance and/or training from more experienced people to people such as workers, who may be semi-skilled, in an efficient way. People receiving assistance and/or training are referred to herein as "knowledge acquirers”. Preferred embodiments of the invention can also provide a new way of imparting skills or training remotely.
  • a system for providing visual assistance to a knowledge acquirer working at a workplace remote from a knowledge provider comprising: headwear including :
  • a forward-facing camera for obtaining a moving image of a view of the workplace as seen by the knowledge acquirer and converting it to a first video electric signal;
  • a microphone positioned for detecting the voice of the knowledge acquirer and converting it to an audio electric signal;
  • a projector for projecting an overlay image onto the workplace so as to at least partially overlap the moving image of the workplace captured by the camera; a controller for:
  • the communications module being adapted to wirelessly communicate the processed signals to said knowledge provider;
  • a source of electric power connected to said camera, said speakers, said controller, said communications module and said projector;
  • a remote computing unit configured for viewing by the knowledge provider the first video signal and for receiving input instructions from the knowledge provider, the input instructions being converted to the second electric signal and communicated to the projector via the communications module for display at the worksite by the projector.
  • the communications module is a mobile phone running a mobile application (App) and is in communication with a mobile data network, the App being configured for transmitting data between the phone and the remote computing unit via the internet.
  • the communications module may be also part of the headwear where a separate mobile phone is not permitted.
  • the headwear can include a central control unit, batteries, headphones, microphone, camera, projector and a light source positioned on the headwear such that weight thereof is evenly balanced on a user's head when a headband is adjusted to fit the user's head.
  • the headwear includes a hard shell suitable for protection against impacts.
  • the input device can be a mouse, the mouse being used to add predetermined symbols or graphical representations to a display on the computing unit so as to overlay the image.
  • the input device can also be used to draw freehand lines on the image.
  • a forward-facing camera for obtaining a moving image of a view of the workplace as seen by the knowledge acquirer and converting it to a first video electric signal;
  • a microphone positioned for detecting the voice of the knowledge acquirer and converting it to an audio electric signal;
  • speakers for producing sounds adjacent at least one ear of the knowledge acquirer; a projector for projecting an overlay image onto the workplace so as to at least partially overlap the moving image of the workplace captured by the camera; a controller for:
  • the communications module being adapted to wirelessly communicate the first processed video electric signal and said processed audio electric signal to said knowledge provider, and to wirelessly receive said incoming voice signal and the second processed video electric signal from the knowledge provider;
  • the headwear has a hard shell suitable for protection against impacts.
  • Figure 1 is a front view of headwear according to a first embodiment of the invention
  • Figure 2 is a side view of the headwear in Figure 1;
  • Figure 3 is a sectional view the status indicators lights for viewing by the knowledge acquirer in an upper corner of their eyes.
  • Figure 4 is a schematic drawing showing the interaction between a wearer of the headwear and a knowledge provider.
  • a headwear 10 according to a preferred embodiment of the invention is shown in
  • the headwear 10 is in the form of a safety hat and configured for use with a system for training or instructing a knowledge acquirer working at a workplace remote from a knowledge provider person.
  • the hat 10 has as its basis a standard hard-shell industrial safety hat 11 as worn on construction sites, though it will be appreciated that it may take other forms such as semirigid headwear or a soft cap, for use in workplaces not requiring head protection.
  • a camera 14 Upon the hat 10 is mounted a camera 14, LED lights 16, a speaker 18 mounted near each ear, a microphone 20, a projector 21 and a controller and processor unit 22.
  • a battery pack 24 is provided within an upper internal portion of the hat and contains a rechargeable battery.
  • the components incorporated into the hat 10, i.e. the camera 14, LED lights 16, speakers 18, microphone 20, projector 21, controller and processor unit 22, and battery pack 24 are positioned on the headwear such that weight thereof is evenly balanced on a user's head when a headband is adjusted to fit the user's head.
  • the hat 10 may be configured to be of light weight construction. This may be achieved through selection of the components mentioned above, or through the construction of the hat 10.
  • the camera 14 is preferably a CCD colour camera which is forward -facing and configured for obtaining a moving image of a view of the workplace as seen by the knowledge acquirer.
  • the knowledge acquirer is a worker who is a semi-skilled technician wearing hat 10.
  • Hat 10 obtains a moving image of a view of the workplace and converts it to a first video electric signal.
  • the camera 14 is centrally mounted in a brim of the hat 10 so as to be directly above a point which is between the eyes of the semi-skilled technician.
  • the camera 14 may be independently adjustable, i.e. zoom and tilt, and is preferably mounted within the confines of the hat 10 so as to be protected from damage and may be configured for taking still photographs.
  • a led light 16 is provided to illuminate the workplace, to improve the vision of the semi-skilled technician and enable the camera 14 to capture a clear video signal.
  • the microphone 20 is configured for detecting the voice of the semi-skilled technician and converting it to an audio electric signal to enable the semi-skilled technician to communicate with a knowledge provider.
  • the microphone 20 is preferably mounted to a forward portion of the hat 10.
  • the microphone may also be used in a voice command system for controlling the hat 10 as illustrated in figure 3 item 20.
  • the speakers 18 are configured for producing sounds audible to the wearer so that verbal instructions may be relayed to the semi-skilled technician.
  • the speakers are preferably fitted to a side of the headwear, close to a wearers ear. A single speaker or multiple speakers may be used.
  • the speakers 18 allow the knowledge provider to communicate with the semi-skilled technician.
  • earphones or headphones can be provided.
  • the earphones may be earbuds mounted on a retractable cable to keep the earbuds safely kept inside the hat 10 when not in use.
  • a 3.5mm headphone jack may be provided instead to enable standard headphones or ear buds to be inserted directly into the technician's ear canal.
  • the projector 21 is configured for projecting an image onto the workplace to partially or fully overlap the moving image of the workplace captured by the camera. This allows further communication with the semi-skilled technician from the knowledge provider and, in use, a knowledge provider at a remote location to draw the attention of the semiskilled technician to a particular point on seeing the video electric signal from the camera 14. As will be described in further detail below, the projector allows display of an image onto the workplace so as to either communicate information to the semi-skilled technician or to draw their attention to an object or position.
  • the controller (A/V processor 22) is configured for receiving and processing the first video electric signal obtained from the camera 14 and the audio electric signal obtained from the microphone 20, before passing those processed signals to a communications module (not shown).
  • the communications module is mounted within the shell of the hat 11 and is adapted to wirelessly communicate the first processed video electric signal and said processed audio electric signal to the knowledge provider, and to wirelessly receive said incoming voice signal and the second processed electric signal from the knowledge provider.
  • the controller is also configured for receiving and processing an incoming voice signal from the communications module and directing the processed incoming voice signal to the speakers 18.
  • the controller is also configured for receiving from the communications module a second electric signal containing graphical information and for passing the processed signals to the projector 21.
  • Figure 4 illustrates the hat 10 within the described system 70.
  • the communications module 42 of the hat 10 is configured for communication with a mobile device 44 such as a mobile phone or tablet, preferably using standard protocols such as Bluetooth for example.
  • the mobile device 44 should have the ability to operate in standard mobile or cell communication protocols and frequency bands, such as the 3G, 4G or 5G ranges for example. It will be appreciated that communication via other radio signals are also possible, particularly in worksites with infrastructure to support such communication.
  • the mobile device 44 is used to transfer the first and second electric signal and voice signals over the Telecommunications networks (internet) between a knowledge acquirer 46 and a knowledge provider 48.
  • Dedicated software in the form of an App installed on the mobile device 44 may be used to affect this transfer.
  • the App may be used to allow direct connection between remote knowledge acquirers and may include a directory for identifying and contacting suitable knowledge providers (closed group) as required.
  • the App may also be used to schedule appointments with knowledge providers.
  • the App or the hat 10 may also capture other parameters for recording as collected data. Some of this data may be available to the knowledge provider during the session while other parameters are merely recorded for later analysis.
  • the parameters may be environmental parameters, such as temperature, noise, vibrations or light conditions at the workplace, or biometric parameters of the knowledge acquirer, such as body temperature, heart rate, perspiration etc. Also, location systems such Global Positioning System may be incorporated as a parameter.
  • the App may include a video/audio processing software package which can operate on standard operating systems (such as Windows, Android or Apple OS).
  • the App and software at the knowledge provider's terminal or computing unit should have communications software that has high bandwidth capacity for conferencing between the knowledge acquirer with the mobile communications device and the knowledge provider's computer and their OS.
  • a communications system should be provided at the knowledge provider's location to enable the encoding and decoding of the audio/video information.
  • a central switching computer 40 is used to identify on a computer network the location of the knowledge provider and the knowledge acquirer and provide appropriate switching instructions when instructed to do so.
  • An artificial intelligence unit 50 may be used to capture two-way audio/visual information to record the interaction between the technician 46 and the knowledge provider 48 for future reference, audit and training purposes.
  • an instruction module may also be provided.
  • the instruction module may also be configured to dynamically study the workplace, ascertain the problem and give instructions. This may involve identifying a training module relevant to the situation and providing instructions from a pre-recorded collection of instructions, which may be based on previous interactions between knowledge provider and knowledge acquirer.
  • the instruction module may be self-learning from previous interactions between knowledge provider and knowledge acquirer.
  • the hat 10 also includes a power source which is connected to the camera 14, the speakers 18, the controller 22, the communications module and the projector 21.
  • the power source is preferably a rechargeable battery.
  • the hat 10 can include a display 26 disposed under the brim 29 of the hat or at the edge of a user's eyesight for displaying various information, such as for example connectivity status, whether a particular component is switched on and battery level. LED's may be provided for displaying this information.
  • Hat 10 relies on near synchronous transmission of video and audio between the two ends of the call, i.e. between the knowledge provider and the knowledge acquirer, to allow the passing of critical information and giving of directives in real time. Because of this requirement, it is important that the knowledge acquirer is aware that the link is in synchronisation or not. Due to the nature of the Telecommunications networks there maybe occasions when the video and audio arriving at the knowledge provider would be time lagging that of knowledge acquirer. This means the audio and visual instructions or directives will be late when relayed via the speakers and projector. Under these conditions the knowledge acquirer must be alerted to condition and stop any manual operation or actions until such time as the time lag has decreased to a predetermined time, which is preferably 0.5 Sec or less.
  • a synchronisation indicator or network status indicator 28 is provided to measure the delay in the round trip from the Hat 10 and back again which is preferably equal to or less than 0.5 Sec.
  • a warning indicator shall be used by the knowledge acquirer to ascertain if the delay is satisfactory or not. It shall indicate the following states:
  • the controller 22 is configured for measuring packets of information from both directions and gain information on the round-trip delay.
  • a special ID marker is inserted into the data stream, preferably in the direction from the hat 10 to the knowledge provider.
  • the ID marker includes a unique code and sequence stamp and does not interfere with the decoding of audio packets.
  • the handling of missing or out of sequence packets shall be handled in the same way as the TCP/IP packet handler.
  • the lag performance monitor shall use a smoothing algorithm to enable an average of the individual lags to be calculated. Considering that the lag values are expected to follow a standard statistic natural log distribution curve the following equation shall be used :
  • the algorithm should take no more than lOOmSec to produce an initial value of L
  • the indicator 28 is incorporated into the underside of the left-hand side of the brim 29 of the hat 10, the knowledge acquirer can be warned without impeding or distracting them.
  • the indicator 28 is preferably positioned to remain only just visible in upper corner of the wearers left eye.
  • the link shall be monitored for delay performance. If the monitoring algorithm indicates that the link has a delay of 0.5 Sec or less, then the indicator light should be set to give a "green" status on the Network status indicator. If at any time the performance unit starts to detect a delay of 0.5 to 1 sec the Network status indicator shall change to an "amber" colour. Further, if the delay that is detected is over 1 sec for a period 0.25secs or longer the Network status indicator shall change its colour to "red”.
  • a system status indicator 30 is also provided. Once the controller 22 is satisfied with the internal check and the battery capacity level is more than 50% the status indicator 30 shall be turn on in a "green” state. If the battery capacity level is between 10 and 50% then the status indicator 30 shall be in an "amber” state. Once the battery capacity level goes below 10% the status indicator 30 shall be in a "red” state.
  • the status indicator light 30 is preferably incorporated into the underside of the right-hand side of the brim 29 of the hat 10. It shall be positioned to remain only just visible in upper corner of the knowledge acquirer's eye. Another form indicator provides a light source to shine into the eye of the knowledge acquirer to get their attention may also be provided
  • the hat 10 is preferably configured to be as unobtrusive as possible so as to provide minimal interference to the work of the knowledge acquirer. This is important as in some situations, for example when the knowledge acquirer simultaneously needs to operate another device, such as a laptop, tablet or specialised equipment and displays, or electric control panels, for example.
  • the hat 10 is an example of headwear which is part of a system for instructing or training a knowledge acquirer working at a workplace remote from a knowledge provider person.
  • a remote terminal 52 or computing unit which may be a personal computer configured for viewing the first video signal and for receiving input instructions from the knowledge provider.
  • the input instructions are preferably made via a mouse or stylus type device and are preferably in the form of markings made on a screen displaying the first video signal.
  • the markings may be in the form of predetermined icons, text or freehand lines.
  • X-Y coordinates may be provided on the screen to assist with locating a particular point or tracking a length.
  • a live stream of pixel by pixel information may be provided to create a live drawn picture.
  • the input instructions are converted to a second electric signal and communicated to the headwear via the communications module for display at the worksite by the projector 21.
  • the software running on the computing unit 52 may include a button for launching a visual assistance program, which displays a series of predetermined icons on the screen and which can be dragged onto the display to enable the knowledge provider to communicate information with the knowledge acquirer.
  • a visual assistance program which displays a series of predetermined icons on the screen and which can be dragged onto the display to enable the knowledge provider to communicate information with the knowledge acquirer.
  • Freestyle lines and/or predetermined shapes may also be used and projected onto the workplace via the projector.
  • an image of the workplace may also be used and once projected onto the workplace, the knowledge acquirer will be easily able to identify inconsistencies and readily modify the workplace accordingly.
  • different option screens can be either minimised or excluded from being displayed at the knowledge provider's computer.
  • the knowledge acquirer is notified that the projector is about to start, which preferably occurs after a predetermined delay, such as two seconds, to avoid the projector being pointed into a person's eyes.
  • Either the knowledge acquirer via the mobile device or the knowledge provider via the computing unit 52 may have access to controls of the projector, such as colour, contrast and brightness.
  • the computing unit 52 software shall obtain positional co-ordinates from computing unit's standard operating system mouse pointer position and send this data to the hat 10. Also, in this data is information as to the selection of a symbol over freehand and the type of symbol chosen (i.e. an arrow).
  • the controller 22 will receive the data from the knowledge provider's computing unit 52 and interpret the positional information into the correct position on the projected surface. In addition, the information data received will give a code for the symbol chosen by the knowledge provider and displayed on the surface of the object viewed.
  • the computing unit 52 software shall interpret the code and construct a new scene for the projector which is a picture of the same symbol, with a black background projected in the same co-ordinates as the pointer on the computing unit 52. This symbol is now projected onto the surface that the knowledge acquirer is viewing.
  • the position of the symbol is added to the composite picture that is seen by the camera 14.
  • the knowledge provider can now see their local mouse pointer projected at the knowledge acquirer's location. At this point there can be an option to remove the local pointer as the knowledge provider can control the remote symbol from their mouse and display in the knowledge provider's computing unit 52.
  • the knowledge provider can activate a button on the browser program to turn off the symbol projection.
  • the knowledge provider will click the left-hand mouse button once to restore the local mouse pointer on the screen.
  • the knowledge provider can then click on the deactivate button on the browser page. This will cause the program to send data to hat 10 to cease the projection of the symbol and turn off the projector.
  • Hat 10 may send data back to computing unit 52 to confirm the visual assistance program has been turned off.
  • this confirmation is received at the computing unit 52, the program will shut down the options window which is running in the background and reset the button on the display to the original activate button.
  • a free hand image maybe projected, operating in much the same way as described above, though with the options window open this time the knowledge provider would select a "freehand" mode of operation for the visual assistance program. On selection of this option the option window shall be minimised so as not to block the video picture from hat 10.
  • An example use of the system is in the remote training of a technician in the telecommunications industry performing fibre splicing in an underground pit.
  • the technician may have completed a theory of fibre splicing through an "on-line" registered training organisation, as well as all his basic Fibre cable and OH&S training.
  • the technician has at hand all the necessary equipment to carry out fibre splicing (that is the heat fusing of two individual fibres together to make one continuous length of fibre) and has scheduled an appointment to complete his practical test in fibre splicing using a fibre splicing machine AST123.
  • the technician Once the technician has arrived at the location for the test and set up the area around the underground pit for safe working, he will remove the pit lid and remove the fibre cable as identified. He will then prepare the two ends of the fibre cables for the splicing operation and set up a splicing table and the splicing machine on the same.
  • the technician places the hat 10 on their head and powers it up by pushing a power button. During this time, a technician would typically check that the batteries are charged.
  • the microphone, headphones, camera and projector may be powered through the controller 22.
  • a communication link is then activated to pair the hat 10 with a mobile device 44.
  • the App is then launched on the mobile device 44 for controlling the communication between the technician and a knowledge provider.
  • the App may perform a number of checks of hat 10 to ensure it is operational.
  • the technician Once the technician is satisfied that the hat 10 is setup and operating correctly, they move to a position in front of the splicing machine where they can operate the machine with their hands.
  • the technician may be seated in position with the mobile device 44 on their hip and can then check the video from the camera 14 and adjust the focus so the splicing machine is in clear view.
  • the technician contacts the Trainer. This may be via a calendar entry with both parties contact details, or alternatively it can include selecting from a menu a conference link item and scrolling down a list of connection addresses until he sees the required address.
  • the App will automatically establish a connection via the Telecommunications or data communications network using a standard internet protocol for video/audio links to the knowledge providers remote computing unit 52.
  • the Trainer will receive an internet call through his client application which is part of the initialisation of the system. Once connected, the Trainer and technician will be able to talk to and listen to each other. At the same time the Trainer will be able to see the view from the technician's camera. Provided that sufficient bandwidth and capacity of the link are available, the Trainer should have a picture-perfect view of what the technician is looking at in real time. Once the task is complete, the call can be terminated through the App on the mobile device. If desirable, the hat 10 may be removed and replaced with a standard hard hat, while the hat 10 can be powered down and placed on charge for later use.
  • the system has a number of uses, in particular in the industrial sectors and telecommunications, and will have particular application in the area of training and assistance with troubleshooting.
  • the following is a list of applications in no particular order:

Abstract

A system for providing visual assistance to a knowledge acquirer at a workplace remote from a knowledge provider, said system comprising: headwear including: a forward-facing camera for obtaining a moving image of the workplace and converting it to a first video electric signal; a microphone positioned for detecting voice and converting it to an audio electric signal; speakers for producing audible instructions; a projector for projecting an overlay image onto the workplace; a controller for: receiving and processing said first video electric signal and said audio electric signal before passing said processed signals to a communications module, the communications module being adapted to wirelessly communicate the processed signals to said knowledge provider; receiving and processing an incoming voice signal from said communications module and directing said processed incoming voice signal to said speakers; and receiving a second electric signal containing graphic information and passing said processed signal to the projector.

Description

Figure imgf000003_0001
Augmented visual assistance system for assisting a person working at a remote workplace, method and headwear for use therewith
Field of the invention
The present invention relates to the remote instruction or training of people in practical skills on the job rather than in the classroom, and headwear for facilitating such training.
Background
The demographics of the population is aging and in some industries the number of long-experienced workers is declining. Accordingly, it is not always possible or convenient to have an experienced service worker or skilled technician available to perform a task, particularly at a remote location. These experienced workers or technicians are referred to herein as "knowledge providers".
Also, at times a person, which may be an unskilled or semi-skilled worker, is faced with tasks that they are not trained to deal with or are unsure about and there is a need for that person to receive guidance to complete the required task. It is not always possible or practical to have an experienced service person available on site to guide the worker.
Preferred embodiments of the invention can provide a way to efficiently provide assistance and/or training from more experienced people to people such as workers, who may be semi-skilled, in an efficient way. People receiving assistance and/or training are referred to herein as "knowledge acquirers". Preferred embodiments of the invention can also provide a new way of imparting skills or training remotely.
Summary
There is provided a system for providing visual assistance to a knowledge acquirer working at a workplace remote from a knowledge provider, said system comprising:
Figure imgf000004_0001
headwear including :
a forward-facing camera for obtaining a moving image of a view of the workplace as seen by the knowledge acquirer and converting it to a first video electric signal; a microphone positioned for detecting the voice of the knowledge acquirer and converting it to an audio electric signal;
speakers for producing audible instructions for the knowledge acquirer;
a projector for projecting an overlay image onto the workplace so as to at least partially overlap the moving image of the workplace captured by the camera; a controller for:
receiving and processing said first video electric signal and said audio electric signal before passing said processed signals to a communications module, the communications module being adapted to wirelessly communicate the processed signals to said knowledge provider;
receiving and processing an incoming voice signal from said communications module and directing said processed incoming voice signal to said speakers; and receiving from said communications module a second electric signal containing graphic information, processing said second electric signal and passing said processed signal to the projector, wherein the communications module is also adapted to wirelessly receive said incoming voice signal and said second processed electric signal from the knowledge provider; and
a source of electric power connected to said camera, said speakers, said controller, said communications module and said projector; and
a remote computing unit configured for viewing by the knowledge provider the first video signal and for receiving input instructions from the knowledge provider, the input instructions being converted to the second electric signal and communicated to the projector via the communications module for display at the worksite by the projector.
Preferably, the communications module is a mobile phone running a mobile application (App) and is in communication with a mobile data network, the App being configured for transmitting data between the phone and the remote computing unit via the internet. The
Figure imgf000005_0001
communications module may be also part of the headwear where a separate mobile phone is not permitted.
The headwear can include a central control unit, batteries, headphones, microphone, camera, projector and a light source positioned on the headwear such that weight thereof is evenly balanced on a user's head when a headband is adjusted to fit the user's head. Preferably, the headwear includes a hard shell suitable for protection against impacts.
There is also provided a headwear for use in a system of the above described type.
There is also provided a method of assisting a knowledge acquirer using a system of the above described type at a workplace remote from a knowledge provider operating a computing unit, said method comprising:
obtaining from said camera a moving image of a view as seen by the knowledge acquirer and converting said moving image to a first video electric signal,
obtaining from said microphone an audio electric signal,
by said controller:
receiving and processing said first video electric signal and said audio electric signal before passing said processed signals wirelessly to a communications module, and
wirelessly receiving and processing an incoming voice signal and a second electric signal containing graphic information from said communications module and directing said processed incoming voice signal to said speakers and said second electric signal to the projector, and
by said knowledge provider:
viewing the video electric signal on the computing unit and using an input device to input instructions to the computing unit, the instructions being communicated from the computing unit to the communications module for display at the worksite via the projector.
Figure imgf000006_0001
The input device can be a mouse, the mouse being used to add predetermined symbols or graphical representations to a display on the computing unit so as to overlay the image. The input device can also be used to draw freehand lines on the image. There is also provided a headwear for instructing or training a person working at a workplace remote from a knowledge provider person, said headwear comprising :
a forward-facing camera for obtaining a moving image of a view of the workplace as seen by the knowledge acquirer and converting it to a first video electric signal; a microphone positioned for detecting the voice of the knowledge acquirer and converting it to an audio electric signal;
speakers for producing sounds adjacent at least one ear of the knowledge acquirer; a projector for projecting an overlay image onto the workplace so as to at least partially overlap the moving image of the workplace captured by the camera; a controller for:
receiving and processing said first video electric signal and said audio electric signal before passing said processed signals to a communications module, the communications module being adapted to wirelessly communicate the first processed video electric signal and said processed audio electric signal to said knowledge provider, and to wirelessly receive said incoming voice signal and the second processed video electric signal from the knowledge provider;
receiving and processing an incoming voice signal from said communications module and directing said processed incoming voice signal to said speakers;
receiving from said communications module a second electric signal containing graphic information and passing said processed signals to the projector, and a source of electric power connected to said camera, said speakers, said controller, said communications module and said projector.
Preferably, the headwear has a hard shell suitable for protection against impacts.
Figure imgf000007_0001
Brief description of the drawings
In order that the invention may be more easily understood, an embodiment will now be described, by way of example only, with reference to the accompanying drawings, in which :
Figure 1: is a front view of headwear according to a first embodiment of the invention;
Figure 2: is a side view of the headwear in Figure 1;
Figure 3: is a sectional view the status indicators lights for viewing by the knowledge acquirer in an upper corner of their eyes.
Figure 4: is a schematic drawing showing the interaction between a wearer of the headwear and a knowledge provider.
Detailed description
A headwear 10 according to a preferred embodiment of the invention is shown in
Figure 1. The headwear 10 is in the form of a safety hat and configured for use with a system for training or instructing a knowledge acquirer working at a workplace remote from a knowledge provider person. The hat 10 has as its basis a standard hard-shell industrial safety hat 11 as worn on construction sites, though it will be appreciated that it may take other forms such as semirigid headwear or a soft cap, for use in workplaces not requiring head protection.
Upon the hat 10 is mounted a camera 14, LED lights 16, a speaker 18 mounted near each ear, a microphone 20, a projector 21 and a controller and processor unit 22. A battery pack 24 is provided within an upper internal portion of the hat and contains a rechargeable battery.
The components incorporated into the hat 10, i.e. the camera 14, LED lights 16, speakers 18, microphone 20, projector 21, controller and processor unit 22, and battery pack 24 are positioned on the headwear such that weight thereof is evenly balanced on a
Figure imgf000008_0001
user's head when a headband is adjusted to fit the user's head. The hat 10 may be configured to be of light weight construction. This may be achieved through selection of the components mentioned above, or through the construction of the hat 10. The camera 14 is preferably a CCD colour camera which is forward -facing and configured for obtaining a moving image of a view of the workplace as seen by the knowledge acquirer. In the examples provided, the knowledge acquirer is a worker who is a semi-skilled technician wearing hat 10. Hat 10 obtains a moving image of a view of the workplace and converts it to a first video electric signal. The camera 14 is centrally mounted in a brim of the hat 10 so as to be directly above a point which is between the eyes of the semi-skilled technician. The camera 14 may be independently adjustable, i.e. zoom and tilt, and is preferably mounted within the confines of the hat 10 so as to be protected from damage and may be configured for taking still photographs. A led light 16 is provided to illuminate the workplace, to improve the vision of the semi-skilled technician and enable the camera 14 to capture a clear video signal.
The microphone 20 is configured for detecting the voice of the semi-skilled technician and converting it to an audio electric signal to enable the semi-skilled technician to communicate with a knowledge provider. The microphone 20 is preferably mounted to a forward portion of the hat 10. The microphone may also be used in a voice command system for controlling the hat 10 as illustrated in figure 3 item 20.
The speakers 18 are configured for producing sounds audible to the wearer so that verbal instructions may be relayed to the semi-skilled technician. The speakers are preferably fitted to a side of the headwear, close to a wearers ear. A single speaker or multiple speakers may be used. The speakers 18 allow the knowledge provider to communicate with the semi-skilled technician. Also, for noisy environments, earphones or headphones can be provided. The earphones may be earbuds mounted on a retractable cable to keep the earbuds safely kept inside the hat 10 when not in use. Also, a 3.5mm headphone jack may be provided instead to enable standard headphones or ear buds to be inserted directly into the technician's ear canal.
Figure imgf000009_0001
The projector 21 is configured for projecting an image onto the workplace to partially or fully overlap the moving image of the workplace captured by the camera. This allows further communication with the semi-skilled technician from the knowledge provider and, in use, a knowledge provider at a remote location to draw the attention of the semiskilled technician to a particular point on seeing the video electric signal from the camera 14. As will be described in further detail below, the projector allows display of an image onto the workplace so as to either communicate information to the semi-skilled technician or to draw their attention to an object or position.
The controller (A/V processor 22) is configured for receiving and processing the first video electric signal obtained from the camera 14 and the audio electric signal obtained from the microphone 20, before passing those processed signals to a communications module (not shown). The communications module is mounted within the shell of the hat 11 and is adapted to wirelessly communicate the first processed video electric signal and said processed audio electric signal to the knowledge provider, and to wirelessly receive said incoming voice signal and the second processed electric signal from the knowledge provider. The controller is also configured for receiving and processing an incoming voice signal from the communications module and directing the processed incoming voice signal to the speakers 18. The controller is also configured for receiving from the communications module a second electric signal containing graphical information and for passing the processed signals to the projector 21.
Figure 4 illustrates the hat 10 within the described system 70. The communications module 42 of the hat 10 is configured for communication with a mobile device 44 such as a mobile phone or tablet, preferably using standard protocols such as Bluetooth for example. The mobile device 44 should have the ability to operate in standard mobile or cell communication protocols and frequency bands, such as the 3G, 4G or 5G ranges for
Figure imgf000010_0001
example. It will be appreciated that communication via other radio signals are also possible, particularly in worksites with infrastructure to support such communication.
The mobile device 44 is used to transfer the first and second electric signal and voice signals over the Telecommunications networks (internet) between a knowledge acquirer 46 and a knowledge provider 48. Dedicated software in the form of an App installed on the mobile device 44 may be used to affect this transfer. The App may be used to allow direct connection between remote knowledge acquirers and may include a directory for identifying and contacting suitable knowledge providers (closed group) as required. The App may also be used to schedule appointments with knowledge providers.
The App or the hat 10 may also capture other parameters for recording as collected data. Some of this data may be available to the knowledge provider during the session while other parameters are merely recorded for later analysis. The parameters may be environmental parameters, such as temperature, noise, vibrations or light conditions at the workplace, or biometric parameters of the knowledge acquirer, such as body temperature, heart rate, perspiration etc. Also, location systems such Global Positioning System may be incorporated as a parameter. The App may include a video/audio processing software package which can operate on standard operating systems (such as Windows, Android or Apple OS). The App and software at the knowledge provider's terminal or computing unit should have communications software that has high bandwidth capacity for conferencing between the knowledge acquirer with the mobile communications device and the knowledge provider's computer and their OS. A communications system should be provided at the knowledge provider's location to enable the encoding and decoding of the audio/video information.
As illustrated in Figure 3, a central switching computer 40 is used to identify on a computer network the location of the knowledge provider and the knowledge acquirer and provide appropriate switching instructions when instructed to do so. An artificial intelligence unit 50 may be used to capture two-way audio/visual information to record the
Figure imgf000011_0001
interaction between the technician 46 and the knowledge provider 48 for future reference, audit and training purposes.
In addition to the switching computer 40, the artificial intelligence unit 50, or incorporated thereinto, an instruction module may also be provided. The instruction module may also be configured to dynamically study the workplace, ascertain the problem and give instructions. This may involve identifying a training module relevant to the situation and providing instructions from a pre-recorded collection of instructions, which may be based on previous interactions between knowledge provider and knowledge acquirer. The instruction module may be self-learning from previous interactions between knowledge provider and knowledge acquirer.
The hat 10 also includes a power source which is connected to the camera 14, the speakers 18, the controller 22, the communications module and the projector 21. The power source is preferably a rechargeable battery.
As illustrated in Figure 3, the hat 10 can include a display 26 disposed under the brim 29 of the hat or at the edge of a user's eyesight for displaying various information, such as for example connectivity status, whether a particular component is switched on and battery level. LED's may be provided for displaying this information.
Hat 10 relies on near synchronous transmission of video and audio between the two ends of the call, i.e. between the knowledge provider and the knowledge acquirer, to allow the passing of critical information and giving of directives in real time. Because of this requirement, it is important that the knowledge acquirer is aware that the link is in synchronisation or not. Due to the nature of the Telecommunications networks there maybe occasions when the video and audio arriving at the knowledge provider would be time lagging that of knowledge acquirer. This means the audio and visual instructions or directives will be late when relayed via the speakers and projector. Under these conditions the knowledge acquirer must be alerted to condition and stop any manual operation or
Figure imgf000012_0001
actions until such time as the time lag has decreased to a predetermined time, which is preferably 0.5 Sec or less.
A synchronisation indicator or network status indicator 28 is provided to measure the delay in the round trip from the Hat 10 and back again which is preferably equal to or less than 0.5 Sec. A warning indicator shall be used by the knowledge acquirer to ascertain if the delay is satisfactory or not. It shall indicate the following states:
a) Green - Less than or equal to 0.5 Sec
b) Amber - Less than 1 Sec but greater than 0.5 Sec.
c) Red - Greater than 1 Sec.
To determine network status, the controller 22 is configured for measuring packets of information from both directions and gain information on the round-trip delay. Using the protocol for the streaming video and associated audio, a special ID marker is inserted into the data stream, preferably in the direction from the hat 10 to the knowledge provider. Upon receipt of the ID marker at the terminal of the knowledge provider, it is immediately sent back as an extra data packet. The ID marker includes a unique code and sequence stamp and does not interfere with the decoding of audio packets. The handling of missing or out of sequence packets shall be handled in the same way as the TCP/IP packet handler. The lag performance monitor shall use a smoothing algorithm to enable an average of the individual lags to be calculated. Considering that the lag values are expected to follow a standard statistic natural log distribution curve the following equation shall be used :
L = e a2
Where L = mode value of lag
m = mean of L
s = standard deviation (0.25)
The algorithm should take no more than lOOmSec to produce an initial value of L As the indicator 28 is incorporated into the underside of the left-hand side of the brim 29 of the hat 10, the knowledge acquirer can be warned without impeding or
Figure imgf000013_0001
distracting them. The indicator 28 is preferably positioned to remain only just visible in upper corner of the wearers left eye.
During a call, the link shall be monitored for delay performance. If the monitoring algorithm indicates that the link has a delay of 0.5 Sec or less, then the indicator light should be set to give a "green" status on the Network status indicator. If at any time the performance unit starts to detect a delay of 0.5 to 1 sec the Network status indicator shall change to an "amber" colour. Further, if the delay that is detected is over 1 sec for a period 0.25secs or longer the Network status indicator shall change its colour to "red".
A system status indicator 30 is also provided. Once the controller 22 is satisfied with the internal check and the battery capacity level is more than 50% the status indicator 30 shall be turn on in a "green" state. If the battery capacity level is between 10 and 50% then the status indicator 30 shall be in an "amber" state. Once the battery capacity level goes below 10% the status indicator 30 shall be in a "red" state.
To enable a warning to the knowledge acquirer without impeding them or distracting them, the status indicator light 30 is preferably incorporated into the underside of the right-hand side of the brim 29 of the hat 10. It shall be positioned to remain only just visible in upper corner of the knowledge acquirer's eye. Another form indicator provides a light source to shine into the eye of the knowledge acquirer to get their attention may also be provided
The hat 10 is preferably configured to be as unobtrusive as possible so as to provide minimal interference to the work of the knowledge acquirer. This is important as in some situations, for example when the knowledge acquirer simultaneously needs to operate another device, such as a laptop, tablet or specialised equipment and displays, or electric control panels, for example. The hat 10 is an example of headwear which is part of a system for instructing or training a knowledge acquirer working at a workplace remote from a knowledge provider
Figure imgf000014_0001
person. Such a system includes a remote terminal 52 or computing unit, which may be a personal computer configured for viewing the first video signal and for receiving input instructions from the knowledge provider. The input instructions are preferably made via a mouse or stylus type device and are preferably in the form of markings made on a screen displaying the first video signal. The markings may be in the form of predetermined icons, text or freehand lines. X-Y coordinates may be provided on the screen to assist with locating a particular point or tracking a length. A live stream of pixel by pixel information may be provided to create a live drawn picture. The input instructions are converted to a second electric signal and communicated to the headwear via the communications module for display at the worksite by the projector 21.
In use, the software running on the computing unit 52 may include a button for launching a visual assistance program, which displays a series of predetermined icons on the screen and which can be dragged onto the display to enable the knowledge provider to communicate information with the knowledge acquirer. Freestyle lines and/or predetermined shapes may also be used and projected onto the workplace via the projector. Furthermore, an image of the workplace may also be used and once projected onto the workplace, the knowledge acquirer will be easily able to identify inconsistencies and readily modify the workplace accordingly. When the visual assistance program is launched, different option screens can be either minimised or excluded from being displayed at the knowledge provider's computer. Once the visual assistance program is launched, the knowledge acquirer is notified that the projector is about to start, which preferably occurs after a predetermined delay, such as two seconds, to avoid the projector being pointed into a person's eyes.
Either the knowledge acquirer via the mobile device or the knowledge provider via the computing unit 52, may have access to controls of the projector, such as colour, contrast and brightness.
Figure imgf000015_0001
The computing unit 52 software shall obtain positional co-ordinates from computing unit's standard operating system mouse pointer position and send this data to the hat 10. Also, in this data is information as to the selection of a symbol over freehand and the type of symbol chosen (i.e. an arrow).
The controller 22 will receive the data from the knowledge provider's computing unit 52 and interpret the positional information into the correct position on the projected surface. In addition, the information data received will give a code for the symbol chosen by the knowledge provider and displayed on the surface of the object viewed.
Based on the knowledge provider's input, the computing unit 52 software shall interpret the code and construct a new scene for the projector which is a picture of the same symbol, with a black background projected in the same co-ordinates as the pointer on the computing unit 52. This symbol is now projected onto the surface that the knowledge acquirer is viewing.
The position of the symbol is added to the composite picture that is seen by the camera 14. The knowledge provider can now see their local mouse pointer projected at the knowledge acquirer's location. At this point there can be an option to remove the local pointer as the knowledge provider can control the remote symbol from their mouse and display in the knowledge provider's computing unit 52.
When the visual assistance program is no longer required, the knowledge provider can activate a button on the browser program to turn off the symbol projection. To carry this operation out, the knowledge provider will click the left-hand mouse button once to restore the local mouse pointer on the screen. The knowledge provider can then click on the deactivate button on the browser page. This will cause the program to send data to hat 10 to cease the projection of the symbol and turn off the projector.
Figure imgf000016_0001
Hat 10 may send data back to computing unit 52 to confirm the visual assistance program has been turned off. When this confirmation is received at the computing unit 52, the program will shut down the options window which is running in the background and reset the button on the display to the original activate button.
Instead of using symbols, or in addition to, a free hand image maybe projected, operating in much the same way as described above, though with the options window open this time the knowledge provider would select a "freehand" mode of operation for the visual assistance program. On selection of this option the option window shall be minimised so as not to block the video picture from hat 10.
An example use of the system is in the remote training of a technician in the telecommunications industry performing fibre splicing in an underground pit. The technician may have completed a theory of fibre splicing through an "on-line" registered training organisation, as well as all his basic Fibre cable and OH&S training. The technician has at hand all the necessary equipment to carry out fibre splicing (that is the heat fusing of two individual fibres together to make one continuous length of fibre) and has scheduled an appointment to complete his practical test in fibre splicing using a fibre splicing machine AST123.
Once the technician has arrived at the location for the test and set up the area around the underground pit for safe working, he will remove the pit lid and remove the fibre cable as identified. He will then prepare the two ends of the fibre cables for the splicing operation and set up a splicing table and the splicing machine on the same.
At the time of the appointment, the technician places the hat 10 on their head and powers it up by pushing a power button. During this time, a technician would typically check that the batteries are charged. The microphone, headphones, camera and projector may be powered through the controller 22. A communication link is then activated to pair the hat 10 with a mobile device 44. The App is then launched on the mobile device 44 for controlling the communication between the technician and a knowledge provider.
Figure imgf000017_0001
The App may perform a number of checks of hat 10 to ensure it is operational.
Once the technician is satisfied that the hat 10 is setup and operating correctly, they move to a position in front of the splicing machine where they can operate the machine with their hands. The technician may be seated in position with the mobile device 44 on their hip and can then check the video from the camera 14 and adjust the focus so the splicing machine is in clear view. At the prearranged time, the technician contacts the Trainer. This may be via a calendar entry with both parties contact details, or alternatively it can include selecting from a menu a conference link item and scrolling down a list of connection addresses until he sees the required address. Once selected, the App will automatically establish a connection via the Telecommunications or data communications network using a standard internet protocol for video/audio links to the knowledge providers remote computing unit 52.
The Trainer will receive an internet call through his client application which is part of the initialisation of the system. Once connected, the Trainer and technician will be able to talk to and listen to each other. At the same time the Trainer will be able to see the view from the technician's camera. Provided that sufficient bandwidth and capacity of the link are available, the Trainer should have a picture-perfect view of what the technician is looking at in real time. Once the task is complete, the call can be terminated through the App on the mobile device. If desirable, the hat 10 may be removed and replaced with a standard hard hat, while the hat 10 can be powered down and placed on charge for later use.
The system has a number of uses, in particular in the industrial sectors and telecommunications, and will have particular application in the area of training and assistance with troubleshooting. The following is a list of applications in no particular order:
Figure imgf000018_0001
• Teaching training practical skills
• Assessment of practical skills on-the-job
• Providing expert assistance to staff who are unclear what is required
• Auditing of work completed to the correct standard
· Verification of work completed in a safe and healthy manner.
The system could be extended to other industries that have staff or independent workers who require expert assistance in the field or remote location. Whilst the above description includes the preferred embodiments of the invention, it is to be understood that many variations, alterations, modifications and/or additions may be introduced into the constructions and arrangements of parts previously described without departing from the essential features or the spirit or ambit of the invention. Although the above embodiments have been described for use by a technician, it will be appreciated that the system may be used by many different people in many different situations.
Throughout this specification and the claims which follow, unless the context requires otherwise, the word "comprise", and variations such as "comprises" and "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.
The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgment or admission or any form of suggestion that that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.

Claims

CLAIMS:
1. A system for providing visual assistance to a knowledge acquirer working at a workplace remote from a knowledge provider, said system comprising:
headwear including :
a forward-facing camera for obtaining a moving image of a view of the workplace as seen by the knowledge acquirer and converting it to a first video electric signal; a microphone positioned for detecting the voice of the knowledge acquirer and converting it to an audio electric signal;
speakers for producing audible instructions for the knowledge acquirer;
a projector for projecting an overlay image onto the workplace so as to at least partially overlap the moving image of the workplace captured by the camera; a controller for:
receiving and processing said first video electric signal and said audio electric signal before passing said processed signals to a communications module, the communications module being adapted to wirelessly communicate the processed signals to said knowledge provider;
receiving and processing an incoming voice signal from said communications module and directing said processed incoming voice signal to said speakers; and receiving from said communications module a second electric signal containing graphic information, processing said second electric signal and passing said processed signal to the projector, wherein the communications module is also adapted to wirelessly receive said incoming voice signal and said second processed electric signal from the knowledge provider; and
a source of electric power for powering said camera, said speakers, said controller, said communications module and said projector; and
a remote computing unit configured for viewing by the knowledge provider the first video signal and for receiving input instructions from the knowledge provider, the input instructions being converted to the second electric signal and communicated to the projector via the communications module for display at the worksite by the projector.
Figure imgf000020_0001
2. A system according to claim 1, wherein the communications module is a mobile phone running a mobile application (App) and is in communication with a mobile data network, the App being configured for transmitting data between the phone and the remote computing unit via data communications systems.
3. A system according to claim 1 or claim 2, wherein the headwear includes a central control unit, batteries, headphones, microphone, camera and a light source positioned on the headwear such that weight thereof is evenly balanced on the knowledge acquirer's head when a headband is adjusted to fit the knowledge acquirer's head.
4. A system according to any preceding claim, wherein the headwear includes a hard shell suitable for protection against impacts.
5. A headwear for use in a system according to any preceding claim.
6. A method of instructing or training a knowledge acquirer using a system according to any one of claims 1 to 4 at a workplace remote from a knowledge provider operating a computing unit, said method comprising:
obtaining from said camera a moving image of a view as seen by the knowledge acquirer and converting said moving image to a first video electric signal,
obtaining from said microphone an audio electric signal,
by said controller:
receiving and processing said first video electric signal and said audio electric signal before passing said processed signals wirelessly to a communications module, and
wirelessly receiving and processing an incoming voice signal and a second electric signal containing graphic information from said communications module and directing said processed incoming voice signal to said speakers and said second electric signal to the projector, and
by said knowledge provider:
viewing the video electric signal on the computing unit and using an input device to input instructions to the computing unit, the instructions being communicated from the computing unit to the communications module for display at the worksite via the projector.
Figure imgf000021_0001
7. A method according to claim 6, wherein the input device is a mouse, the mouse being used to add predetermined symbols or graphical representations to a display on the computing unit so as to overlay the image.
8. A method according to claim 6, wherein the input device is used to draw freehand lines on the image.
9. A headwear for instructing or training a person working at a workplace remote from a knowledge provider person, said headwear comprising:
a forward-facing camera for obtaining a moving image of a view of the workplace as seen by the knowledge acquirer and converting it to a first video electric signal; a microphone positioned for detecting the voice of the knowledge acquirer and converting it to an audio electric signal;
speakers for producing sounds adjacent at least one ear of the knowledge acquirer; a projector for projecting an overlay image onto the workplace so as to at least partially overlap the moving image of the workplace captured by the camera; a controller for:
receiving and processing said first video electric signal and said audio electric signal before passing said processed signals to a communications module, the communications module being adapted to wirelessly communicate the first processed video electric signal and said processed audio electric signal to said knowledge provider, and to wirelessly receive said incoming voice signal and the second processed electric signal from the knowledge provider;
receiving and processing an incoming voice signal from said communications module and directing said processed incoming voice signal to said speakers;
receiving from said communications module a second electric signal containing graphic information and passing said processed signals to the projector, and a source of electric power connected to said camera, said speakers, said controller, said communications module and said projector.
10. A headwear according to claim 6, having a hard shell suitable for protection against impacts.
PCT/AU2018/051230 2017-12-21 2018-11-16 Augmented visual assistance system for assisting a person working at a remote workplace, method and headwear for use therewith WO2019119022A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
AU2017905104 2017-12-21
AU2017905104A AU2017905104A0 (en) 2017-12-21 Augmented visual assistance system for assisting a person working at a remote workplace, method and headwear for use therewith

Publications (1)

Publication Number Publication Date
WO2019119022A1 true WO2019119022A1 (en) 2019-06-27

Family

ID=66992361

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/AU2018/051230 WO2019119022A1 (en) 2017-12-21 2018-11-16 Augmented visual assistance system for assisting a person working at a remote workplace, method and headwear for use therewith

Country Status (1)

Country Link
WO (1) WO2019119022A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023188951A1 (en) * 2022-03-28 2023-10-05 株式会社サンタ・プレゼンツ Remote instruction system
JP7406038B1 (en) * 2023-09-19 2023-12-26 株式会社日立パワーソリューションズ Work support system and work support method

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4439157A (en) * 1982-05-03 1984-03-27 The United States Of America As Represented By The Secretary Of The Navy Helmet mounted display projector
US6046712A (en) * 1996-07-23 2000-04-04 Telxon Corporation Head mounted communication system for providing interactive visual communications with a remote system
US6535182B2 (en) * 1998-12-07 2003-03-18 Koninklijke Philips Electronics N.V. Head-mounted projection display system
WO2009112063A2 (en) * 2007-09-18 2009-09-17 Vrmedia S.R.L. Information processing apparatus and method for remote technical assistance
US20160353055A1 (en) * 2015-05-29 2016-12-01 Purdue Research Foundation Simulated transparent display with augmented reality for remote collaboration
US9538962B1 (en) * 2014-12-31 2017-01-10 Verily Life Sciences Llc Heads-up displays for augmented reality network in a medical environment
US20170290707A1 (en) * 2015-03-18 2017-10-12 Tecmen Electronics Co., Ltd. Wireless projector-type welding helmet and welding machine equipped with the same

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4439157A (en) * 1982-05-03 1984-03-27 The United States Of America As Represented By The Secretary Of The Navy Helmet mounted display projector
US6046712A (en) * 1996-07-23 2000-04-04 Telxon Corporation Head mounted communication system for providing interactive visual communications with a remote system
US6535182B2 (en) * 1998-12-07 2003-03-18 Koninklijke Philips Electronics N.V. Head-mounted projection display system
WO2009112063A2 (en) * 2007-09-18 2009-09-17 Vrmedia S.R.L. Information processing apparatus and method for remote technical assistance
US9538962B1 (en) * 2014-12-31 2017-01-10 Verily Life Sciences Llc Heads-up displays for augmented reality network in a medical environment
US20170290707A1 (en) * 2015-03-18 2017-10-12 Tecmen Electronics Co., Ltd. Wireless projector-type welding helmet and welding machine equipped with the same
US20160353055A1 (en) * 2015-05-29 2016-12-01 Purdue Research Foundation Simulated transparent display with augmented reality for remote collaboration

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023188951A1 (en) * 2022-03-28 2023-10-05 株式会社サンタ・プレゼンツ Remote instruction system
JP7406038B1 (en) * 2023-09-19 2023-12-26 株式会社日立パワーソリューションズ Work support system and work support method

Similar Documents

Publication Publication Date Title
EP1552697B1 (en) Method and apparatus for telepresence
US9819843B2 (en) Head-mounted systems and methods for providing inspection, evaluation or assessment of an event or location
US20190050191A1 (en) Display system, display device, information display method, and program
WO2016086440A1 (en) Wearable guiding device for the blind
KR101431424B1 (en) Plant system for supporting operation/maintenance using smart helmet capable of bi-directional communication and method thereof
JP2016034490A (en) Seeing-eye mobile device positioning system and method of operating same
JP2016505889A (en) Communication glasses with safety function zone
KR20160104537A (en) A system for real time relaying and transmitting
KR102047988B1 (en) Vision aids apparatus for the vulnerable group of sight, remote managing apparatus and method for vision aids
JP7270154B2 (en) Remote work support system
US20180345501A1 (en) Systems and methods for establishing telepresence of a remote user
WO2019119022A1 (en) Augmented visual assistance system for assisting a person working at a remote workplace, method and headwear for use therewith
CN111127822B (en) Augmented reality-based fire-fighting auxiliary method and intelligent wearable equipment
KR20130131181A (en) Goggles having human interface function
CN112396718A (en) On-site construction safety and quality supervision research system based on AR technology
CN209911926U (en) Cable accessory manufacturing and guiding system
KR20160042717A (en) Real time monitoring device using smartphone
JP6733401B2 (en) Display system, display device, information display method, and program
JP2003308117A (en) System for maintenance
JP2014027459A (en) Communication apparatus, communication method, and communication program
JP2017062650A (en) Display system, display unit, information display method, and program
JP2020154569A (en) Display device, display control method, and display system
KR20200109201A (en) Remote work support system using head mounted display apparatus and augmented reality
CN211608330U (en) Wearable device
KR102134419B1 (en) Thermographic image sensing device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18893153

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WPC Withdrawal of priority claims after completion of the technical preparations for international publication

Ref document number: 2017905104

Country of ref document: AU

Date of ref document: 20200620

Free format text: WITHDRAWN AFTER TECHNICAL PREPARATION FINISHED

122 Ep: pct application non-entry in european phase

Ref document number: 18893153

Country of ref document: EP

Kind code of ref document: A1