WO2016004303A1 - Method and apparatus for construction machine visualization - Google Patents

Method and apparatus for construction machine visualization Download PDF

Info

Publication number
WO2016004303A1
WO2016004303A1 PCT/US2015/038981 US2015038981W WO2016004303A1 WO 2016004303 A1 WO2016004303 A1 WO 2016004303A1 US 2015038981 W US2015038981 W US 2015038981W WO 2016004303 A1 WO2016004303 A1 WO 2016004303A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
information
determining
gaze
displayed
Prior art date
Application number
PCT/US2015/038981
Other languages
French (fr)
Inventor
Daniele PAGLIANI
Nicola FINARDI
Richard Jackson
Ivan Di Federico
Original Assignee
Topcon Positioning Systems, Inc.
Tierra S.p.A.
InfoMobility S.r.L.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Topcon Positioning Systems, Inc., Tierra S.p.A., InfoMobility S.r.L. filed Critical Topcon Positioning Systems, Inc.
Priority to EP23153423.1A priority Critical patent/EP4191386A1/en
Priority to EP15739428.9A priority patent/EP3164787B1/en
Priority to AU2015283939A priority patent/AU2015283939A1/en
Publication of WO2016004303A1 publication Critical patent/WO2016004303A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • G06Q10/06311—Scheduling, planning or task assignment for a person or group
    • G06Q10/063114—Status monitoring or status determination for a person or group
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
    • G06Q50/08—Construction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/12—Picture reproducers
    • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B2027/0178—Eyeglass type

Definitions

  • the present disclosure relates generally to the display of information, and more particularly to construction machine visualization.
  • Operators of construction machines are required to operate the machines according to a desired site plan in order to modify the site to match the desired site plan.
  • the operator is required to continuously monitor the current state of the site and operate the machine in a manner to change the current site to the desired site plan.
  • an operator of a construction machine must constantly reference a desired site plan, typically on paper or on a display in the cab of the machine, to ensure that the modification of the current site is performed to change the site to the desired state.
  • a method includes determining a direction of a user's gaze.
  • the direction of the user's gaze can be determined using a head tracking system.
  • the direction of the user's gaze can also be determined using a gaze tracking system.
  • Information to be displayed to a user is then determined and the information is displayed to the user.
  • the information can be displayed by transmitting the information to a projection system and/or a pair of smart glasses.
  • changes in the user's gaze and changes in the current state of a construction site are determined and new information is generated and displayed to the user.
  • FIG. 1 depicts a construction machine, specifically, an excavator
  • FIG. 2 depicts an operator's view from inside a cab of a construction machine according to an embodiment
  • Fig. 3 depicts an operator's view from inside a cab of a construction machine according to an alternative embodiment
  • Fig. 4 depicts a pair of smart glasses according to an embodiment
  • FIG. 5 depicts a system for displaying information to a user via a two- dimensional projection system according to an embodiment
  • FIG. 6 depicts a system for displaying information to a user via a three- dimensional projection system according to an embodiment
  • Fig. 7 depicts a system for displaying information to a user via a pair of smart glasses according to an embodiment
  • FIG. 8 depicts an example of a display according to an embodiment
  • FIG. 9 depicts a flowchart of a method for displaying information according to an embodiment
  • FIG. 10 depicts a flowchart of a method for displaying information according to an alternative embodiment
  • Fig. 1 1 depicts a high-level schematic of a computer according to an embodiment.
  • FIG. 1 depicts excavator 100 comprising cab 102, boom 104, stick 106, and bucket 108.
  • Cab 102, boom 104, stick 106, and bucket 108 are rotatably mounted on an undercarriage 1 10.
  • An operator sits in cab 102 and controls excavator 100 to move it to a desired location and manipulate various material, such as dirt.
  • the accuracy and efficiency of operations facilitated by an operator vary depending on the experience of the operator.
  • Accurate and efficient operation of excavator 100 typically requires an operator with extensive experience including interpreting site plans and modifying a site using a construction machine to achieve the desired site plan.
  • Figure 2 depicts an operator's view from inside cab 102 of excavator 100 according to one embodiment of the invention. From inside cab 102, an operator can see stick 106 and bucket 108.
  • the view from inside cab 102 includes information overlay 200 (also referred to as a heads up display or HUD) which, in this example, depicts an image of excavator 100 and a desired plan to modify the earth around excavator 100.
  • information overlay 200 also referred to as a heads up display or HUD
  • an operator can see the view of a site outside cab 102 as well as information overlay 200.
  • An operator can select and/or change information displayed by information overlay 200 via intelligent display 202.
  • Figure 3 depicts an operator's view from inside cab 102 of excavator 100 according to one embodiment. Similar to Figure 2, from inside cab 102, an operator can see boom 104, stick 106 and bucket 108. As operator is operating excavator 100, additional information is displayed including excavation area overlay 302 which depicts an area which is to be modified according to a predetermined desired site plan. Volume overlay 304 provides an operator with information concerning a volume of earth to be removed from excavation area 302. Completion overlay 306 provides an operator with an indication of the level of completion of an operation. In one embodiment, additional overlays are presented to the operator. For example, an overlay pertaining to a current position of bucket 108 compared to an expected position of bucket 108 can be displayed.
  • protocols can be control area network (CAN) bus protocols such as CAN J1939, CAN open, or CAN custom.
  • Information may be displayed to an operator using various hardware and methods.
  • information is displayed to an operator via a pair of smart glasses which incorporate display devices.
  • Figure 4 depicts smart glasses 400 including display devices which are used to provide a wearer with information by displaying information over the wearer's view through the smart glasses.
  • Glasses 400 are used in one embodiment to provide information overlay 200 of Figure 2 and excavation area overlay 302, volume overlay 304, and completion overlay 306 of Figure 3.
  • a projection system is used to display information to an operator by projecting an image onto a transparent surface such as a windshield of a machine.
  • Figure 5 depicts system 500 for advanced visualization of site information using overlays.
  • Controller 502 located in a cab of a machine in one embodiment, manages the system and receives head tracking information via head tracking driver 504 from head tracking system 506 (also located in the cab of the machine in one embodiment). Controller 502 is also in communication with projection system 508 via projection driver 510. Projection system 508 comprises see-through display 522 and projection source 524. Controller 502 is also in communication with intelligent display 512. Specifically, software 514 of controller 502 is configured to allow controller 502 to communicate with screen 516 of intelligent display 512. Touch driver 518 of controller 502 is in communication with touch sensitive component 520 of intelligent display 512 and also with software 514.
  • Intelligent display 512 can be used by an operator to select and/or change information displayed to a user via see-through display 522 and projection source 524. Each of these components are typically located in a cab of a machine but can be located or attached to other areas of the machine.
  • controller 502 receives information from head tracking system 506 via head tracking driver 504 in order to determine where an operator is looking based on the operator's head position. Based on where the operator is looking, controller 502 determines what and where information should be presented to a user via an overlay (or HUD) using projection system 508. Information for display via projection system 508 can be selected and/or changed via intelligent display 512 using touch sensitive component 520.
  • projection system 508 is a monoscopic system comprising a transparent surface and a light source.
  • the transparent surface is used to visualize an image generated by the light source.
  • the surface can cover or overlap a portion of a windshield or the entire windshield of a vehicle such as excavator 100 (shown in Figure 1 ).
  • the light source in one embodiment, is capable of creating an appropriate light wave for the corresponding transparent surface in order to display information to the operator of excavator 100 (shown in Figure 1 ).
  • An image generated by the light source and transparent surface can be focused at a desired distance or at an infinite distance.
  • head tracking system 506 is capable of evaluating a position of an operator within cab 102 (shown in Figure 1 ).
  • Head tracking system 506 in one embodiment, provides information about the current head position of the operator to controller 502 via head tracking driver 504. Images displayed to the operator are then varied based on information from head tracking system 506. For example, the content visualized on the display or location of the information on the display may change based on the operator's head movement and current position.
  • Controller 502 in one embodiment, stores and manages digital design information such as surfaces, line work, and other site data. Controller 502 manages machine geometry calculations such as excavation and navigation calculations. In one embodiment, controller 502 manages communications with a site communications system and other local machines (not shown). Controller 502, in one embodiment generates images for display via one or more display systems such as projection or smart glasses. Controller 502 can determine a current state of a site using information from various sensors attached to the machine such as image sensors, object sensors, etc. Information from these sensors is received by controller 502 which, in one embodiment, uses the information to generate a virtual representation of the current state of the site. This virtual representation can be used to determine differences between the current state of the site and a desired site plan.
  • Intelligent display 512 in one embodiment, is a high-brightness, multi- touch display device and is the primary device for user input and selection. In one embodiment, intelligent display 512 displays a two dimensional image of a machine in which intelligent display 512 is located and provides machine and site data to an operator. Intelligent display 512 can display a machine dashboard and provide information concerning the machine's current condition as well as provide machine diagnostics. Machine parameters can be received by controller 502 and transmitted to a device, such as intelligent display 512, for display to the user.
  • system 500 can be used to provide an operator with various information including site and machine operation via a heads-up display as well as machine conditions and diagnostics via intelligent display 512.
  • System 500 displays two-dimensional information using a HUD via projection system 508.
  • the display of information to a user can be a three-dimensional or a display generated using smart glasses.
  • Figure 6 depicts system 600 for displaying information via three- dimensional images.
  • Controller 602 functions similar to controller 502 shown in Figure 5 and includes head/eye/gaze tracking driver 604 for receiving information from head/eye/gaze tracking system 606.
  • system 600 determines an operator's gaze by tracking the movement of the operator's eyes. The gaze and eye information, along with head position information, is used by system 600 to dynamically manage the parallax for the operator's eyes.
  • Projection system 608 comprises stereo ready see-through display 622 onto which three dimensional images are projected by stereo projection source 624. Display information for projection system 608 is transmitted, in one embodiment, stereo projection driver 610 of controller 602 to stereo projection source 624 of projection system 608.
  • Intelligent display 612 comprising screen 616 and touch sensitive component 620 operate in a manner similar to intelligent display 512 of Figure 5.
  • information is displayed to an operator via a pair of smart glasses such as smart glasses 400 shown in Figure 4.
  • Figure 7 depicts a system in which information is displayed to an operator via a pair of smart glasses.
  • Controller 702 functions similar to controller 502 shown in Figure 5 and includes a head/eye/gaze tracking driver for receiving information from head/eye/gaze tracking system 706.
  • system 700 determines an operator's gaze by tracking the movement of the operator's eyes. This gaze and eye information, along with head position information, is used by system 600 to dynamically manage the parallax for the operator's eyes.
  • head, eye, and gaze tracking can be used to generate a 360 degree visualization of virtual information around the operator.
  • system 700 allows information to be displayed to a user in any direction the user looks since the display of information is presented to the user via display devices incorporated in the associated pair of smart glasses 708.
  • Smart glasses 708 in one embodiment, comprises stereo ready see-through display 722 onto which three dimensional images are projected by stereo projection source 724.
  • Display information for projection system 708 is transmitted, in one embodiment, from wireless communication port 726 of controller 702 to stereo projection wireless communication port 728 of smart glasses 708.
  • Information received via wireless communication port 728 is sent to stereo projection source 724 of projection system 708. Images are then displayed to the operator via smart glasses 708.
  • Intelligent display 712 comprising screen 716 and touch sensitive component 720 operates in a manner similar to intelligent display 512 of Figure 5.
  • Figure 8 depicts an example of a display according to one embodiment using projector 802 and display surface 804.
  • Projector 802 projects an image onto display surface 804.
  • projector 806 projects an image onto display surface 808.
  • different projectors and display surfaces are used.
  • a laser scan projector (scan line) can be used with a reflective film to scatter light emitted from the projector.
  • a laser matrix projector can be used with a reflective film to scatter light emitted from the projector.
  • a laser scan projector full scan
  • a projector is used with a reflective film sized to be used as a display in conjunction with a window and/or view port of a cab of a machine.
  • a projector is used with a reflective film sized to be used with a pair of smart glasses.
  • Smart glasses can be monocular (i.e., displaying information via one lens of the smart glasses), or binocular (i.e., displaying information via both lenses of the smart glasses).
  • a three dimensional image can be displayed to the user. In other embodiments, a mix of two dimensional and three dimensional images are displayed to the user.
  • Figure 9 depicts a flow chart of a method 900 for displaying information to a user according to one embodiment.
  • Various configurations as described above can operate using the steps identified in method 900.
  • sensor input from various sensors are received and analyzed.
  • a current head position of a user and/or operator is determined.
  • the current head position and a previous head position are compared.
  • relative head movement is determined in order to evaluate whether displayed information should be changed based on head movement.
  • information is transmitted to the display system based on possible change in head position as well as changes in conditions monitored (e.g., movement of the machine or parts of the machine, etc.)
  • Figure 10 depicts a flow chart of a method 1000 for displaying information to a user of a machine according to an embodiment.
  • a direction of a user's gaze is determined.
  • a user's gaze is determined based on a head position of the user.
  • controller 502 (shown in Figure 5) receives information from head tracking system 506 via head tracking driver 504 in order to determine where a user is looking based on the user's head position.
  • a user's gaze is determined based on a position of the user's eyes and a direction the user's eyes are looking.
  • controller 602 (shown in Figure 6) receives information concerning a user's gaze from head/eye/gaze tracking system 606 via head/eye/gaze tracking driver 604.
  • information to be displayed to the user is determined based on the direction of the user's gaze and a current state of a site.
  • the information to be displayed to the user can include how a portion of a site the user is looking at needs to be modified.
  • the information to be displayed to the user is determined by determining a difference between the current state of the site and a desired site plan. An operation to be performed based on the different between the current state of the site and the desired site plan can also be determined.
  • the information displayed to the user can include the operation to be performed.
  • the difference between the current state of the site and the desired site plan is used to generate a value pertaining to a percent of completion of the site. The value pertaining to the percent of completion of the site can be displayed to the user.
  • the information is displayed to the user.
  • the information is displayed by transmitting the information from controller 502 (or 602) to a projection system, such as projection system 508.
  • the information is displayed by transmitting the information from controller 502 (or 602) to a pair of smart glasses, such as smart glasses 400 (shown in Figure 4).
  • step 1008 it is determined that a user's gaze has changed.
  • a change in the user's gaze in one embodiment, is determined using information from head/eye/gaze tracking system 606. In another embodiment, a change in the user's gaze is determined using information from head tracking system 506.
  • controller 502 determines that the current state of the site has changed using input received from various sensors associated with and/or attached to a machine the user is operating.
  • new information to be displayed to the user is determined. This new information can be based on a change in the user's gaze and/or a change in the current state of the site.
  • the new information is displayed to the user.
  • FIG. 1 1 A high-level block diagram of such a computer is illustrated in FIG. 1 1 .
  • Computer 1 102 contains a processor 1 104 which controls the overall operation of the computer 1 102 by executing computer program instructions which define such operation.
  • the computer program instructions may be stored in a storage device 1 1 12, or other computer readable medium (e.g., magnetic disk, CD ROM, etc.), and loaded into memory 1 1 10 when execution of the computer program instructions is desired.
  • the method steps of Figures 9 and 10 can be defined by the computer program instructions stored in the memory 1 1 10 and/or storage 1 1 12 and controlled by the processor 1 104 executing the computer program instructions.
  • the computer program instructions can be implemented as computer executable code programmed by one skilled in the art to perform an algorithm defined by the method steps of Figures 9 and 10. Accordingly, by executing the computer program instructions, the processor 1 104 executes an algorithm defined by the method steps of Figures 9 and 10.
  • the computer 1 102 also includes one or more network interfaces 1 106 for communicating with other devices via a network.
  • the computer 1 102 also includes input/output devices 1 108 that enable user interaction with the computer 1 102 (e.g., display, keyboard, mouse, speakers, buttons, etc.)
  • Figure 1 1 is a high level representation of some of the components of such a computer for illustrative purposes.

Abstract

A system for displaying information to an operator of a machine comprises a head tracking system and a projection system. One system for displaying information uses a projection system and a see-through display to present two-dimensional images to an operator. One system uses a projection system and a see-through display to present three-dimensional images to an operator. One system uses a pair of smart glasses to display information to a user based a direction a user is looking.

Description

TITLE
METHOD AND APPARATUS FOR CONSTRUCTION MACHINE VISUALIZATION
BACKGROUND
[0001] The present disclosure relates generally to the display of information, and more particularly to construction machine visualization.
[0002] Operators of construction machines are required to operate the machines according to a desired site plan in order to modify the site to match the desired site plan. The operator is required to continuously monitor the current state of the site and operate the machine in a manner to change the current site to the desired site plan. Typically, an operator of a construction machine must constantly reference a desired site plan, typically on paper or on a display in the cab of the machine, to ensure that the modification of the current site is performed to change the site to the desired state.
SUMMARY
[0003] In one embodiment, a method includes determining a direction of a user's gaze. The direction of the user's gaze can be determined using a head tracking system. The direction of the user's gaze can also be determined using a gaze tracking system. Information to be displayed to a user is then determined and the information is displayed to the user. The information can be displayed by transmitting the information to a projection system and/or a pair of smart glasses. In one embodiment, changes in the user's gaze and changes in the current state of a construction site are determined and new information is generated and displayed to the user.
BRIEF DESCRIPTION OF THE DRAWINGS
[0004] Fig. 1 depicts a construction machine, specifically, an excavator;
[0005] Fig. 2 depicts an operator's view from inside a cab of a construction machine according to an embodiment; [0006] Fig. 3 depicts an operator's view from inside a cab of a construction machine according to an alternative embodiment;
[0007] Fig. 4 depicts a pair of smart glasses according to an embodiment;
[0008] Fig. 5 depicts a system for displaying information to a user via a two- dimensional projection system according to an embodiment;
[0009] Fig. 6 depicts a system for displaying information to a user via a three- dimensional projection system according to an embodiment;
[0010] Fig. 7 depicts a system for displaying information to a user via a pair of smart glasses according to an embodiment;
[0011] Fig. 8 depicts an example of a display according to an embodiment;
[0012] Fig. 9 depicts a flowchart of a method for displaying information according to an embodiment;
[0013] Fig. 10 depicts a flowchart of a method for displaying information according to an alternative embodiment; and
[0014] Fig. 1 1 depicts a high-level schematic of a computer according to an embodiment.
DETAILED DESCRIPTION
[0015] FIG. 1 depicts excavator 100 comprising cab 102, boom 104, stick 106, and bucket 108. Cab 102, boom 104, stick 106, and bucket 108 are rotatably mounted on an undercarriage 1 10. An operator sits in cab 102 and controls excavator 100 to move it to a desired location and manipulate various material, such as dirt. The accuracy and efficiency of operations facilitated by an operator vary depending on the experience of the operator. Accurate and efficient operation of excavator 100 typically requires an operator with extensive experience including interpreting site plans and modifying a site using a construction machine to achieve the desired site plan.
[0016] Figure 2 depicts an operator's view from inside cab 102 of excavator 100 according to one embodiment of the invention. From inside cab 102, an operator can see stick 106 and bucket 108. In addition, according to one embodiment, the view from inside cab 102 includes information overlay 200 (also referred to as a heads up display or HUD) which, in this example, depicts an image of excavator 100 and a desired plan to modify the earth around excavator 100. As shown in Figure 2, an operator can see the view of a site outside cab 102 as well as information overlay 200. An operator can select and/or change information displayed by information overlay 200 via intelligent display 202.
[0017] Figure 3 depicts an operator's view from inside cab 102 of excavator 100 according to one embodiment. Similar to Figure 2, from inside cab 102, an operator can see boom 104, stick 106 and bucket 108. As operator is operating excavator 100, additional information is displayed including excavation area overlay 302 which depicts an area which is to be modified according to a predetermined desired site plan. Volume overlay 304 provides an operator with information concerning a volume of earth to be removed from excavation area 302. Completion overlay 306 provides an operator with an indication of the level of completion of an operation. In one embodiment, additional overlays are presented to the operator. For example, an overlay pertaining to a current position of bucket 108 compared to an expected position of bucket 108 can be displayed. In addition, information about excavator 100 engine rotations per minute (RPM), instantaneous fuel rate, fuel consumption, and other information received from excavator's electronic control unit (ECU) via various protocols can be displayed. In one embodiment, protocols can be control area network (CAN) bus protocols such as CAN J1939, CAN open, or CAN custom.
[0018] Information may be displayed to an operator using various hardware and methods. In one embodiment, information is displayed to an operator via a pair of smart glasses which incorporate display devices. Figure 4 depicts smart glasses 400 including display devices which are used to provide a wearer with information by displaying information over the wearer's view through the smart glasses. Glasses 400 are used in one embodiment to provide information overlay 200 of Figure 2 and excavation area overlay 302, volume overlay 304, and completion overlay 306 of Figure 3. In another embodiment, a projection system is used to display information to an operator by projecting an image onto a transparent surface such as a windshield of a machine. [0019] Figure 5 depicts system 500 for advanced visualization of site information using overlays. Controller 502, located in a cab of a machine in one embodiment, manages the system and receives head tracking information via head tracking driver 504 from head tracking system 506 (also located in the cab of the machine in one embodiment). Controller 502 is also in communication with projection system 508 via projection driver 510. Projection system 508 comprises see-through display 522 and projection source 524. Controller 502 is also in communication with intelligent display 512. Specifically, software 514 of controller 502 is configured to allow controller 502 to communicate with screen 516 of intelligent display 512. Touch driver 518 of controller 502 is in communication with touch sensitive component 520 of intelligent display 512 and also with software 514. Intelligent display 512 can be used by an operator to select and/or change information displayed to a user via see-through display 522 and projection source 524. Each of these components are typically located in a cab of a machine but can be located or attached to other areas of the machine.
[0020] In one embodiment, controller 502 receives information from head tracking system 506 via head tracking driver 504 in order to determine where an operator is looking based on the operator's head position. Based on where the operator is looking, controller 502 determines what and where information should be presented to a user via an overlay (or HUD) using projection system 508. Information for display via projection system 508 can be selected and/or changed via intelligent display 512 using touch sensitive component 520.
[0021] In one embodiment, projection system 508 is a monoscopic system comprising a transparent surface and a light source. The transparent surface is used to visualize an image generated by the light source. The surface can cover or overlap a portion of a windshield or the entire windshield of a vehicle such as excavator 100 (shown in Figure 1 ). The light source, in one embodiment, is capable of creating an appropriate light wave for the corresponding transparent surface in order to display information to the operator of excavator 100 (shown in Figure 1 ). An image generated by the light source and transparent surface can be focused at a desired distance or at an infinite distance. [0022] In one embodiment, head tracking system 506 is capable of evaluating a position of an operator within cab 102 (shown in Figure 1 ). Head tracking system 506, in one embodiment, provides information about the current head position of the operator to controller 502 via head tracking driver 504. Images displayed to the operator are then varied based on information from head tracking system 506. For example, the content visualized on the display or location of the information on the display may change based on the operator's head movement and current position.
[0023] Controller 502, in one embodiment, stores and manages digital design information such as surfaces, line work, and other site data. Controller 502 manages machine geometry calculations such as excavation and navigation calculations. In one embodiment, controller 502 manages communications with a site communications system and other local machines (not shown). Controller 502, in one embodiment generates images for display via one or more display systems such as projection or smart glasses. Controller 502 can determine a current state of a site using information from various sensors attached to the machine such as image sensors, object sensors, etc. Information from these sensors is received by controller 502 which, in one embodiment, uses the information to generate a virtual representation of the current state of the site. This virtual representation can be used to determine differences between the current state of the site and a desired site plan.
[0024] Intelligent display 512, in one embodiment, is a high-brightness, multi- touch display device and is the primary device for user input and selection. In one embodiment, intelligent display 512 displays a two dimensional image of a machine in which intelligent display 512 is located and provides machine and site data to an operator. Intelligent display 512 can display a machine dashboard and provide information concerning the machine's current condition as well as provide machine diagnostics. Machine parameters can be received by controller 502 and transmitted to a device, such as intelligent display 512, for display to the user.
[0025] As described above, system 500 can be used to provide an operator with various information including site and machine operation via a heads-up display as well as machine conditions and diagnostics via intelligent display 512. System 500 displays two-dimensional information using a HUD via projection system 508. In other embodiments, the display of information to a user can be a three-dimensional or a display generated using smart glasses.
[0026] Figure 6 depicts system 600 for displaying information via three- dimensional images. Controller 602 functions similar to controller 502 shown in Figure 5 and includes head/eye/gaze tracking driver 604 for receiving information from head/eye/gaze tracking system 606. In order to generate three-dimensional images that are viewable by an operator, system 600 determines an operator's gaze by tracking the movement of the operator's eyes. The gaze and eye information, along with head position information, is used by system 600 to dynamically manage the parallax for the operator's eyes. Projection system 608 comprises stereo ready see-through display 622 onto which three dimensional images are projected by stereo projection source 624. Display information for projection system 608 is transmitted, in one embodiment, stereo projection driver 610 of controller 602 to stereo projection source 624 of projection system 608. Intelligent display 612 comprising screen 616 and touch sensitive component 620 operate in a manner similar to intelligent display 512 of Figure 5.
[0027] In one embodiment, information is displayed to an operator via a pair of smart glasses such as smart glasses 400 shown in Figure 4. Figure 7 depicts a system in which information is displayed to an operator via a pair of smart glasses. Controller 702 functions similar to controller 502 shown in Figure 5 and includes a head/eye/gaze tracking driver for receiving information from head/eye/gaze tracking system 706. In order to generate three-dimensional images that are viewable by an operator, system 700 determines an operator's gaze by tracking the movement of the operator's eyes. This gaze and eye information, along with head position information, is used by system 600 to dynamically manage the parallax for the operator's eyes. In addition, head, eye, and gaze tracking can be used to generate a 360 degree visualization of virtual information around the operator. In contrast with projection systems which project information onto a fixed transparent display (e.g. a windshield of a vehicle), system 700 allows information to be displayed to a user in any direction the user looks since the display of information is presented to the user via display devices incorporated in the associated pair of smart glasses 708. Smart glasses 708, in one embodiment, comprises stereo ready see-through display 722 onto which three dimensional images are projected by stereo projection source 724. Display information for projection system 708 is transmitted, in one embodiment, from wireless communication port 726 of controller 702 to stereo projection wireless communication port 728 of smart glasses 708. Information received via wireless communication port 728 is sent to stereo projection source 724 of projection system 708. Images are then displayed to the operator via smart glasses 708. Intelligent display 712 comprising screen 716 and touch sensitive component 720 operates in a manner similar to intelligent display 512 of Figure 5.
[0028] Figure 8 depicts an example of a display according to one embodiment using projector 802 and display surface 804. Projector 802 projects an image onto display surface 804. Similarly, projector 806 projects an image onto display surface 808. In various embodiments, different projectors and display surfaces are used. For example, a laser scan projector (scan line) can be used with a reflective film to scatter light emitted from the projector. A laser matrix projector can be used with a reflective film to scatter light emitted from the projector. A laser scan projector (full scan) can be used with reflective film to scatter light emitted from the projector. In one embodiment, a projector is used with a reflective film sized to be used as a display in conjunction with a window and/or view port of a cab of a machine. In one embodiment, a projector is used with a reflective film sized to be used with a pair of smart glasses. Smart glasses can be monocular (i.e., displaying information via one lens of the smart glasses), or binocular (i.e., displaying information via both lenses of the smart glasses). In one embodiment, a three dimensional image can be displayed to the user. In other embodiments, a mix of two dimensional and three dimensional images are displayed to the user.
[0029] Figure 9 depicts a flow chart of a method 900 for displaying information to a user according to one embodiment. Various configurations as described above can operate using the steps identified in method 900. At step 902, sensor input from various sensors (previously described) are received and analyzed. At step 904, a current head position of a user and/or operator is determined. At step 906 the current head position and a previous head position are compared. At step 908, relative head movement is determined in order to evaluate whether displayed information should be changed based on head movement. At step 910, information is transmitted to the display system based on possible change in head position as well as changes in conditions monitored (e.g., movement of the machine or parts of the machine, etc.)
[0030] Figure 10 depicts a flow chart of a method 1000 for displaying information to a user of a machine according to an embodiment. At step 1002 a direction of a user's gaze is determined. In one embodiment, a user's gaze is determined based on a head position of the user. For example, controller 502 (shown in Figure 5) receives information from head tracking system 506 via head tracking driver 504 in order to determine where a user is looking based on the user's head position. In one embodiment, a user's gaze is determined based on a position of the user's eyes and a direction the user's eyes are looking. For example, controller 602 (shown in Figure 6) receives information concerning a user's gaze from head/eye/gaze tracking system 606 via head/eye/gaze tracking driver 604.
[0031] At step 1004, information to be displayed to the user is determined based on the direction of the user's gaze and a current state of a site. For example, the information to be displayed to the user can include how a portion of a site the user is looking at needs to be modified. In one embodiment, the information to be displayed to the user is determined by determining a difference between the current state of the site and a desired site plan. An operation to be performed based on the different between the current state of the site and the desired site plan can also be determined. The information displayed to the user can include the operation to be performed. In one embodiment, the difference between the current state of the site and the desired site plan is used to generate a value pertaining to a percent of completion of the site. The value pertaining to the percent of completion of the site can be displayed to the user.
[0032] At step 1006, the information is displayed to the user. In one embodiment, the information is displayed by transmitting the information from controller 502 (or 602) to a projection system, such as projection system 508. In one embodiment, the information is displayed by transmitting the information from controller 502 (or 602) to a pair of smart glasses, such as smart glasses 400 (shown in Figure 4).
[0033] At step 1008, it is determined that a user's gaze has changed. A change in the user's gaze, in one embodiment, is determined using information from head/eye/gaze tracking system 606. In another embodiment, a change in the user's gaze is determined using information from head tracking system 506.
[0034] At step 1010, it is determined that the current state of the site has changed. In one embodiment, controller 502 (or 602) determines that the current state of the site has changed using input received from various sensors associated with and/or attached to a machine the user is operating.
[0035] At step 1012, new information to be displayed to the user is determined. This new information can be based on a change in the user's gaze and/or a change in the current state of the site. At step 1014, the new information is displayed to the user.
[0036] Smart glasses 400, 708, intelligent display 202 and the various components of systems 500, 600, and 700 can each be implemented using a computer. A high-level block diagram of such a computer is illustrated in FIG. 1 1 . Computer 1 102 contains a processor 1 104 which controls the overall operation of the computer 1 102 by executing computer program instructions which define such operation. The computer program instructions may be stored in a storage device 1 1 12, or other computer readable medium (e.g., magnetic disk, CD ROM, etc.), and loaded into memory 1 1 10 when execution of the computer program instructions is desired. Thus, the method steps of Figures 9 and 10 can be defined by the computer program instructions stored in the memory 1 1 10 and/or storage 1 1 12 and controlled by the processor 1 104 executing the computer program instructions. For example, the computer program instructions can be implemented as computer executable code programmed by one skilled in the art to perform an algorithm defined by the method steps of Figures 9 and 10. Accordingly, by executing the computer program instructions, the processor 1 104 executes an algorithm defined by the method steps of Figures 9 and 10. The computer 1 102 also includes one or more network interfaces 1 106 for communicating with other devices via a network. The computer 1 102 also includes input/output devices 1 108 that enable user interaction with the computer 1 102 (e.g., display, keyboard, mouse, speakers, buttons, etc.) One skilled in the art will recognize that an implementation of an actual computer could contain other components as well, and that Figure 1 1 is a high level representation of some of the components of such a computer for illustrative purposes.
[0037] The foregoing Detailed Description is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the inventive concept disclosed herein is not to be determined from the Detailed Description, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are only illustrative of the principles of the inventive concept and that various modifications may be implemented by those skilled in the art without departing from the scope and spirit of the inventive concept. Those skilled in the art could implement various other feature combinations without departing from the scope and spirit of the inventive concept.

Claims

CLAIMS:
1 . A method comprising:
determining a direction of a user's gaze;
determining information to be displayed to the user based on the direction of the user's gaze, a current state of a construction site, and a current state of a machine; and displaying the information to the user.
2. The method of claim 1 , wherein the direction of the user's gaze is determined based on a head position of the user.
3. The method of claim 1 , wherein the direction of the user's gaze is determined based on a position of the user's eyes and a direction the user's eyes are looking.
4. The method of claim 1 , the determining information to be displayed to a user comprising:
determining a difference between the current state of the construction site and a desired site plan;
determining an operation to be performed based on the difference,
wherein the information comprises the operation to be performed.
5. The method of claim 1 , the determining information to be displayed to a user comprising:
determining a difference between the current state of the construction site and a desired site plan;
generating a value pertaining to a percent of completion of the construction site based on the determining the difference between the current state of the construction site and the desired site plan,
wherein information comprises the percent of completion of the construction site.
6. The method of claim 1 , further comprising:
determining that a user's gaze has changed;
determining new information to be displayed to the user based on the determining that the user's gaze has changed;
displaying the new information to the user.
7. The method of claim 6, further comprising:
determining that the current state of the construction site has changed,
wherein the determining new information to be displayed to the user is further based on a new state of the site.
8. The method of claim 1 , wherein the displaying the information to the user comprises:
transmitting the information to be displayed to the user to a projection system.
9. The method of claim 1 , wherein the displaying the information to the user comprises:
transmitting the information to be displayed to the user to a pair of smart glasses.
10. An apparatus comprising:
a processor; and
a memory to store computer program instructions, the computer program instructions when executed on the processor, cause the processor to perform operations comprising:
determining a direction of a user's gaze;
determining information to be displayed to the user based on the direction of the user's gaze, a current state of a construction site, and a current state of a machine; and displaying the information to the user.
1 1 . The apparatus of claim 10, wherein the direction of the user's gaze is determined based on a head position of the user.
12. The apparatus of claim 10, wherein the direction of the user's gaze is
determined based on a position of the user's eyes and a direction the user's eyes are looking.
13. The apparatus of claim 10, the determining information to be displayed to a user comprising:
determining a difference between the current state of the construction site and a desired site plan;
determining an operation to be performed based on the difference,
wherein the information comprises the operation to be performed.
14. The apparatus of claim 10, the determining information to be displayed to a user comprising:
determining a difference between the current state of the construction site and a desired site plan;
generating a value pertaining to a percent of completion of the construction site based on the determining the difference between the current state of the construction site and the desired site plan,
wherein information comprises the percent of completion of the construction site.
15. The apparatus of claim 10, the operations further comprising:
determining that a user's gaze has changed;
determining new information to be displayed to the user based on the determining that the user's gaze has changed;
displaying the new information to the user.
16. A computer readable medium storing computer program instructions, which, when executed on a processor, cause the processor to perform operations comprising: determining a direction of a user's gaze;
determining information to be displayed to the user based on the direction of the user's gaze, a current state of a construction site, and a current state of a machine; and displaying the information to the user.
17. The computer readable medium of claim 16, wherein the direction of the user's gaze is determined based on a head position of the user.
18. The computer readable medium of claim 16, wherein the direction of the user's gaze is determined based on a position of the user's eyes and a direction the user's eyes are looking.
19. The computer readable medium of claim 16, the determining information to be displayed to a user comprising:
determining a difference between the current state of the construction site and a desired site plan;
determining an operation to be performed based on the difference,
wherein the information comprises the operation to be performed.
20. The computer readable medium of claim 16, the determining information to be displayed to a user comprising:
determining a difference between the current state of the construction site and a desired site plan;
generating a value pertaining to a percent of completion of the construction site based on the determining the difference between the current state of the construction site and the desired site plan,
wherein information comprises the percent of completion of the construction site.
PCT/US2015/038981 2014-07-03 2015-07-02 Method and apparatus for construction machine visualization WO2016004303A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP23153423.1A EP4191386A1 (en) 2014-07-03 2015-07-02 Method and apparatus for construction machine visualization
EP15739428.9A EP3164787B1 (en) 2014-07-03 2015-07-02 Method and apparatus for construction machine visualization
AU2015283939A AU2015283939A1 (en) 2014-07-03 2015-07-02 Method and apparatus for construction machine visualization

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201462020521P 2014-07-03 2014-07-03
US62/020,521 2014-07-03
US14/789,493 US10503249B2 (en) 2014-07-03 2015-07-01 Method and apparatus for construction machine visualization
US14/789,493 2015-07-01

Publications (1)

Publication Number Publication Date
WO2016004303A1 true WO2016004303A1 (en) 2016-01-07

Family

ID=55016985

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2015/038981 WO2016004303A1 (en) 2014-07-03 2015-07-02 Method and apparatus for construction machine visualization

Country Status (4)

Country Link
US (2) US10503249B2 (en)
EP (2) EP4191386A1 (en)
AU (1) AU2015283939A1 (en)
WO (1) WO2016004303A1 (en)

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2016336319A1 (en) * 2015-10-05 2018-02-22 Komatsu Ltd. Construction management system and construction management method
AU2016402225B2 (en) 2016-04-04 2022-02-10 Topcon Positioning Systems, Inc. Method and apparatus for augmented reality display on vehicle windscreen
FI20165452A (en) * 2016-05-31 2017-12-01 Novatron Oy User interface and excavator
JP2018005091A (en) * 2016-07-06 2018-01-11 富士通株式会社 Display control program, display control method and display controller
DE102016011354A1 (en) * 2016-09-20 2018-03-22 Liebherr-Werk Biberach Gmbh Control station for a crane, excavator and the like
US10242457B1 (en) * 2017-03-20 2019-03-26 Zoox, Inc. Augmented reality passenger experience
EP3687936B1 (en) * 2017-09-26 2023-09-13 Palfinger AG Operating device and loading crane having an operating device
NL2019827B1 (en) 2017-10-30 2019-05-08 Jumbo Maritime B V Method for visualizing a 2D technical drawing on a physical surface
EP3499405A1 (en) * 2017-12-13 2019-06-19 My Virtual Reality Software AS Method and device for augmenting a person's view of a mining vehicle on a mining worksite in real-time
JP7159903B2 (en) * 2019-02-19 2022-10-25 コベルコ建機株式会社 Target position estimation device
JP7201486B2 (en) * 2019-03-13 2023-01-10 日立建機株式会社 working machine
JP7318258B2 (en) * 2019-03-26 2023-08-01 コベルコ建機株式会社 Remote control system and remote control server
JP7287047B2 (en) * 2019-03-27 2023-06-06 コベルコ建機株式会社 Remote control system and remote control server
US20210004591A1 (en) * 2019-09-14 2021-01-07 Ron Zass Sequence of events monitoring in construction sites
JP7013509B2 (en) * 2020-03-25 2022-01-31 株式会社インフォマティクス Drawing projection system, drawing projection method and program in construction machinery
US11568620B2 (en) * 2020-10-28 2023-01-31 Shopify Inc. Augmented reality-assisted methods and apparatus for assessing fit of physical objects in three-dimensional bounded spaces
WO2023205502A1 (en) * 2022-04-22 2023-10-26 Meta Platforms Technologies, Llc Task optimization in an extended reality environment

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130307771A1 (en) * 2012-05-18 2013-11-21 Microsoft Corporation Interaction and management of devices using gaze detection

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06156119A (en) 1992-11-18 1994-06-03 Kansei Corp Head-up display device for construction machine
ZA948824B (en) 1993-12-08 1995-07-11 Caterpillar Inc Method and apparatus for operating geography altering machinery relative to a work site
US6094625A (en) * 1997-07-03 2000-07-25 Trimble Navigation Limited Augmented vision for survey work and machine control
GB0410415D0 (en) * 2004-05-11 2004-06-16 Bamford Excavators Ltd Operator display system
US7599762B2 (en) * 2005-08-24 2009-10-06 Rockwell Automatino Technologies, Inc. Model-based control for crane control and underway replenishment
US8145390B2 (en) * 2007-04-26 2012-03-27 Cnh America Llc Swath finder feature integrated with multipurpose display
JP2010018141A (en) 2008-07-10 2010-01-28 Caterpillar Japan Ltd Display device in construction machine
US20100241471A1 (en) * 2009-03-19 2010-09-23 Scenario Design, Llc Integration system supporting dimensioned modeling system
WO2011155749A2 (en) * 2010-06-07 2011-12-15 연세대학교 산학협력단 Tower crane navigation system
US9324197B2 (en) * 2011-03-11 2016-04-26 Intelligent Agricultural Soultions Method and system for managing the hand-off between control terminals
US8947219B2 (en) * 2011-04-22 2015-02-03 Honda Motors Co., Ltd. Warning system with heads up display
US9222771B2 (en) * 2011-10-17 2015-12-29 Kla-Tencor Corp. Acquisition of information for a construction site
US9070216B2 (en) * 2011-12-14 2015-06-30 The Board Of Trustees Of The University Of Illinois Four-dimensional augmented reality models for interactive visualization and automated construction progress monitoring
DE102012001951A1 (en) * 2012-02-02 2013-08-08 Daimler Ag A display device for a vehicle and method for operating a display device
AU2014202349A1 (en) * 2012-08-02 2014-05-22 Harnischfeger Technologies, Inc. Depth-related help functions for a wheel loader training simulator
US20140184643A1 (en) * 2012-12-27 2014-07-03 Caterpillar Inc. Augmented Reality Worksite
KR20140090552A (en) * 2013-01-09 2014-07-17 엘지전자 주식회사 Head Mounted Display and controlling method for eye-gaze calibration
US9096129B2 (en) * 2013-07-29 2015-08-04 Freescale Semiconductor, Inc. Method and system for facilitating viewing of information in a machine
US9335545B2 (en) * 2014-01-14 2016-05-10 Caterpillar Inc. Head mountable display system

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130307771A1 (en) * 2012-05-18 2013-11-21 Microsoft Corporation Interaction and management of devices using gaze detection

Also Published As

Publication number Publication date
EP3164787A1 (en) 2017-05-10
US20200073474A1 (en) 2020-03-05
AU2015283939A1 (en) 2017-02-02
US10503249B2 (en) 2019-12-10
US20160004305A1 (en) 2016-01-07
EP4191386A1 (en) 2023-06-07
US11086397B2 (en) 2021-08-10
EP3164787B1 (en) 2023-03-01

Similar Documents

Publication Publication Date Title
US11086397B2 (en) Method and apparatus for construction machine visualization
JP4468370B2 (en) Three-dimensional display method, apparatus and program
US10902239B2 (en) Methods and systems for training an object detection algorithm using synthetic images
JP4883774B2 (en) Information processing apparatus, control method therefor, and program
US10789744B2 (en) Method and apparatus for augmented reality display on vehicle windscreen
CA2730379C (en) Vehicle user interface unit for a vehicle electronic device
US10655305B2 (en) Work vehicle and method for controlling work vehicle
CN105204619A (en) Holographic projection device, terminal and intelligent watch
US20180373032A1 (en) Work vehicle and display control method
EP3748477A1 (en) Method and system for spawning attention pointers (atp) for drawing attention of an user in a virtual screen display with augmented and virtual reality
RU2604430C2 (en) Interaction with three-dimensional virtual scenario
CN102510503B (en) Stereoscopic display method and stereoscopic display equipment
JP2016147532A (en) Image generation device, and head-up display
JP2022058753A (en) Information processing apparatus, information processing method, and program
JP5866498B2 (en) Display control device, projection device, display control program, and recording medium
WO2014061310A1 (en) Display object control system, display object control method, and program
JP2012252386A (en) Display device
US10845888B2 (en) Techniques for integrating different forms of input with different forms of output when interacting with an application
JP2001209826A (en) Method and device for editing virtual space, computer program storage medium and virtual reality presenting device
JP2011180690A (en) Display control apparatus, display control method, and display control program
JP2020201448A (en) Display system, display method, controller and computer program
JP2007280014A (en) Display parameter input device
WO2020250270A1 (en) Operational assistance system and operational assistance method
CN117707332A (en) Information processing apparatus, control method, and non-transitory computer storage medium
CN117289789A (en) Controlling device settings using head gestures

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15739428

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

REEP Request for entry into the european phase

Ref document number: 2015739428

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2015739428

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2015283939

Country of ref document: AU

Date of ref document: 20150702

Kind code of ref document: A