EP3687937A1 - Operator assistance system and a method in relation to the system - Google Patents

Operator assistance system and a method in relation to the system

Info

Publication number
EP3687937A1
EP3687937A1 EP18762407.7A EP18762407A EP3687937A1 EP 3687937 A1 EP3687937 A1 EP 3687937A1 EP 18762407 A EP18762407 A EP 18762407A EP 3687937 A1 EP3687937 A1 EP 3687937A1
Authority
EP
European Patent Office
Prior art keywords
image
working equipment
view
occluded
processing unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP18762407.7A
Other languages
German (de)
French (fr)
Other versions
EP3687937B1 (en
Inventor
Per Gustafsson
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hiab AB
Original Assignee
Cargotec Patenter AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Cargotec Patenter AB filed Critical Cargotec Patenter AB
Publication of EP3687937A1 publication Critical patent/EP3687937A1/en
Application granted granted Critical
Publication of EP3687937B1 publication Critical patent/EP3687937B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B66—HOISTING; LIFTING; HAULING
    • B66C—CRANES; LOAD-ENGAGING ELEMENTS OR DEVICES FOR CRANES, CAPSTANS, WINCHES, OR TACKLES
    • B66C13/00—Other constructional features or details
    • B66C13/18—Control systems or devices
    • B66C13/46—Position indicators for suspended loads or for crane elements

Definitions

  • the present disclosure relates to an operator assistance system and a method in connection with the system.
  • the operator assistance system is in particular used on a vehicle provided with a working equipment, e.g. a crane, in assisting the operator during loading and unloading procedures.
  • a working equipment e.g. a crane
  • Working vehicles are often provided with various working equipment, e.g.
  • movable cranes which are attached to the vehicle via a joint.
  • These cranes comprise movable crane parts, e.g. booms, that may be extended, and that are joined together by joints such that the crane parts may be folded together at the vehicle and extended to reach a load.
  • Various tools e.g. buckets or forks, may be attached to the crane tip, often via a rotator.
  • the object of the present invention is to achieve an operator assistance system provided with a digital see-through capability for vehicles with a working equipment that solves the aforementioned visibility issues.
  • the invention relates to an operator assistance system for a vehicle provided with a working equipment.
  • the assistance system comprises an image capturing system arranged at the vehicle and/or at the working equipment and capable of capturing parameters related to images, of the working equipment and of the environment outside the vehicle in a predetermined field of view.
  • the operator assistance system further comprises a processing unit configured to receive image related parameter signals from the image capturing system and to process the image related parameter signals, and a display unit configured to present images to an operator.
  • the image capturing system comprises at least two sensor unit assemblies capable of capturing parameters related to essentially overlapping images of the predetermined field of view, and that the processing unit is configured to generate a merged image based upon said overlapping images.
  • the processing unit is configured to determine a shape and position of an image representation of the working equipment occluding a part of an image in the predetermined field of view, obtained by one of the sensor unit assemblies, by processing image related signals from at least one of the other sensor unit assemblies.
  • the processing unit is further configured to determine if the part occluded by the working equipment is visible in any image in the predetermined field of view obtained by any of the other sensor unit assemblies. If the occluded part is visible by any of the other sensor unit assemblies, an occluded part image, being an image representation of the occluded part, is determined in the field of view obtained by said other sensor unit assembly.
  • the processing unit is then configured to merge the occluded part image into the merged image in the determined position of the working equipment, and to display the merged image at the display unit.
  • At least one of the sensor unit assemblies comprises at least one angle sensor and/or at least one length sensor structured to be arranged at the working equipment, and adapted to measure angles and lengths related to movements of the working equipment, and at least one camera unit mounted at the vehicle and/or at the working equipment (6).
  • the sensor unit assemblies comprises at least two camera units that are mounted at separate mounting positions at the vehicle in relation to the working equipment, or at the working equipment, such that different sides of the working equipment are visible at images obtained by the at least two camera units.
  • the processing unit is provided with a set of image representations of the working equipment, and is further configured to apply a pattern recognition algorithm to identify any image representation of the working equipment in an image in the predetermined field of view by comparison to the set of image representations. Thereby it is assured that a safe and fast recognition of an occluding object is achieved.
  • the processing unit is further configured to identify a predetermined part, e.g. a hook, a load, a fork, of the occluding working equipment to be visible in the merged image, and to display said predetermined part at said display unit. This is advantageous as the operator then easily can manoeuvre the hook or fork as it is clearly visible and highlighted at the display unit.
  • the processing unit is configured to determine a presentation mode of the occluded part image, from a set of presentation modes including a transparent mode, wherein the working equipment defining the occluded part image is fully transparent, a semi-transparent mode, wherein the working equipment defining the occluded part image has a variable opacity. This is an important feature as an operator may choose an optimal presentation in dependence of a specific situation.
  • the working equipment is a crane, a demountable arm, a boom, or a bucket, or any other tool arranged at a vehicle.
  • a method is provided that is applied by an operator assistance system for a vehicle provided with a working equipment.
  • the assistance system comprises an image capturing system arranged at the vehicle, and/or at the working equipment, and capable of capturing parameters related to images of the working equipment and of the environment outside the vehicle in a predetermined field of view.
  • the assistance system further comprises a processing unit configured to receive image related parameter signals from the image capturing system and to process the image related parameter signals, and a display unit configured to present images to an operator.
  • the image capturing system comprises at least two sensor unit assemblies capable of capturing parameters related to essentially overlapping images of the predetermined field of view, and that the processing unit is configured to generate a merged image based upon the overlapping images.
  • the method comprises:
  • the method further comprises:
  • identifying a predetermined part e.g. a hook, a load, a fork, of the occluding working equipment to be visible in said merged image
  • the method comprises:
  • Figure 1 is a schematic illustration of a vehicle provided with an operator assistance system according to the present invention.
  • Figure 2 is a schematic illustration of images obtained by sensor unit assemblies, and of a merged image.
  • Figures 3a-3c are schematic views from above illustrating various aspects of embodiments of the present invention.
  • Figure 4 is a schematic view from above illustrating another embodiment of the present invention.
  • FIG. 5 is a flow diagram illustrating the method according to the present invention. Detailed description
  • a block diagram is disclosed schematically illustrating an operator assistance system 2 for a vehicle 4 provided with a working equipment 6.
  • the vehicle may be a cargo vehicle, a truck, a forklift or a working vehicle provided with a working equipment 6 that is e.g. a crane, a demountable arm, a mast boom, a hook-lift equipment provided with a predetermined part 22, e.g. a hook, a fork, or a bucket.
  • the assistance system comprises an image capturing system 8 arranged at the vehicle and/or at the working equipment 6, and capable of capturing parameters related to images of the working equipment 6 and of the environment outside the vehicle in a predetermined field of view 10, a processing unit 12 configured to receive image related parameter signals 14 from the image capturing system 8, to process the image related parameter signals, and a display unit 16 configured to present captured images to an operator.
  • the image capturing system 8 comprises at least two sensor unit assemblies 18, 20 capable of capturing parameters related to essentially overlapping images of said predetermined field of view 10.
  • the processing unit 12 has a processing capability and is configured to generate a merged image based upon the overlapping images.
  • the sensor unit assembly is a camera unit, an angle sensor, a length sensor, or any other sensing unit capable of capturing parameters related to images.
  • the parameters related to images may be parameters directly related to images, e.g. optical parameters detected by e.g. a camera unit, or indirectly related to images, e.g. various parameters representing movement and position of the working equipment, e.g. angles, lengths, width related to parts of the working equipment.
  • the processing unit is configured to determine a shape and position of an image representation of the working equipment 6 occluding a part of an image in the predetermined field of view 10, obtained by one of said sensor unit assemblies, by processing image related signals from at least one of the other sensor unit assemblies.
  • the processing unit 12 is further configured to determine if the part occluded by the working equipment 6 is visible in any image in the predetermined field of view 10 obtained by any of the other sensor unit assemblies. If the occluded part is visible by any of the other sensor unit assemblies, an occluded part image, being an image representation of the occluded part, is determined in the field of view 10 obtained by said other sensor unit assembly.
  • the processing unit is then configured to merge the determined occluded part image into the merged image in the determined position of the working equipment 6, and to display the merged image at the display unit 16.
  • At least one of the sensor unit assemblies comprises at least one angle sensor and at least one length sensor structured to be arranged at the working equipment, and adapted to measure angles and lengths related to movements of said working equipment.
  • at least one camera unit 18, 20 is provided which is mounted at the vehicle 4 and/or at the working equipment 6.
  • the angle sensor(s) and/or the length sensor(s) are arranged at the working equipment and structured to generate sensor signals including angle values and length values which are applied to the processing unit. Based upon those sensor signals and information indicating the type of working equipment the processing unit may determine the shape and position of the working equipment.
  • the sensor unit assemblies comprise at least two camera units 18, 20 that are mounted at separate mounting positions at the vehicle 4 in relation to the working equipment 6, or at the working equipment 6, such that different sides of the working equipment are visible at images obtained by the at least two camera units 18, 20.
  • the camera units 18, 20 are mounted at separate mounting positions at the working equipment, e.g. a boom of a crane, and move thereby together with the crane.
  • the definition that the camera units are mounted at different sides of the working equipment should be interpreted broadly, and not being limited to different sides in a horizontal plane.
  • the important aspect is that the view of sights obtained by the camera units mounted at the different sides cover parts that potentially may be occluded by the working equipment during normal use.
  • the camera units may be mounted at different heights or at other positions where a full overview may be obtained.
  • two camera units are mounted at different sides of the working equipment, e.g.
  • FIG. 2 is shown above two images obtained by two camera units arranged at opposite sides of the working equipment, or by a sensor unit assembly at the working equipment, and at least one camera unit.
  • the working equipment is visible at the right image where it occludes parts of the environment and thereby prevents the operator from having complete visual overview.
  • the position and shape of the occluding working equipment is determined, and the positions are then applied in the image obtained by another camera unit to identify the corresponding positions therein. This is schematically illustrated on the image to the left where the image representation of the occluded part is indicated by dashed lines.
  • the processing unit is provided with a set of image
  • representations of the working equipment and more particularly a set of searchable data representations of images of the working equipment in various views and sizes.
  • the processing unit then is configured to apply a search procedure to identify any image representation of the working equipment in an image in the predetermined field of view obtained by any of the sensor unit assemblies by comparison to the set of image representations.
  • This is preferably performed by applying a dedicated pattern recognition algorithm capable of comparing captured images with the stored set of data representations of images of the working equipment.
  • the working equipment may occlude a part, e.g. a larger part or a smaller part, of the field of view obtained by one camera unit. Even if only a smaller part of the field of view is occluded the search procedure may be performed.
  • the search procedure preferably is continuously performed for all images obtained by all sensor unit assemblies, e.g. the camera units, and that occluded parts in images obtained from different sensor unit assemblies and in different positions then may be identified, and eventually applied in a merged image.
  • the processing unit is configured to identify a predetermined part 22 of the occluding working equipment (see figure 1 and figures 3a-3c), e.g. a hook, a load, a bucket, or a fork, to be visually presented in the merged image, and to display the predetermined part at the display unit. This is preferably performed by applying another pattern recognition algorithm to be set up in advance dependent on the presently used predetermined part of the working equipment.
  • the merged image is presented to the operator on the display unit.
  • the display unit may be any type of presentation unit configured and arranged such that the operator is provided with the necessary support in order to operate the working equipment safely and efficiently. Various presentation modes are available, where the occluding working equipment
  • the processing unit is configured to determine a presentation mode of the occluded part image, from a set of presentation modes including a transparent mode, wherein the working equipment defining the occluded part image is fully transparent and a semi-transparent mode, wherein the working equipment defining the occluded part image has a variable opacity.
  • the processing unit is configured to determine a presentation mode where an outer boundary of the occluded part image is indicated in the merged image.
  • the boundary of the occluded part image may e.g. be indicated by a dashed line.
  • Figures 3a-3c illustrate various presentation modes.
  • stereo cameras may be used for making 3D pictures, or for range imaging. Unlike most other approaches to depth sensing, such as structured light or time-of-flight measurements, stereo vision is a purely passive technology which also works in bright daylight.
  • the image capturing system may comprise at least one camera unit, but may in addition also include one or many sensor units, e.g. angle sensor units, length sensor units, capable of capturing various supporting image data to be supplied to the processing unit.
  • sensor units e.g. angle sensor units, length sensor units, capable of capturing various supporting image data to be supplied to the processing unit.
  • the image capturing system applies the Lidar-technology.
  • Lidar is sometimes considered an acronym of Light Detection and Ranging (sometimes Light Imaging, Detection, and Ranging), and is a surveying method that measures distance to a target by illuminating that target with a laser light.
  • Lidar is popularly used to make high-resolution maps, with applications in geodesy, forestry, laser guidance, airborne laser swath mapping (ALSM), and laser altimetry.
  • Lidar sometimes is called laser scanning and 3D scanning, with terrestrial, airborne, and mobile applications.
  • the image capturing system also includes a 3D scanning device.
  • a 3D scanner is a device that analyses a real-world object or environment to collect data on its shape and possibly its appearance (e.g. colour). The collected data can then be used to construct digital three-dimensional models.
  • Many different technologies can be used to build these 3D-scanning devices; each technology comes with its own limitations, advantages and costs. Many limitations in the kind of objects that can be digitised are still present, for example, optical technologies encounter many difficulties with shiny, mirroring or
  • industrial computed tomography scanning can be used to construct digital 3D models, applying non-destructive testing.
  • 3D scanner The purpose of a 3D scanner is usually to create a point cloud of geometric samples on the surface of the subject. These points can then be used to extrapolate the shape of the subject (a process called reconstruction). If colour information is collected at each point, then the colours on the surface of the subject can also be determined.
  • 3D scanners share several traits with cameras. Like most cameras, they have a cone-like field of view, and like cameras, they can only collect information about surfaces that are not obscured. While a camera collects colour information about surfaces within its field of view, a 3D scanner collects distance information about surfaces within its field of view. The "picture" produced by a 3D scanner describes the distance to a surface at each point in the picture. This allows the three dimensional position of each point in the picture to be identified.
  • a so-called time-of-flight Lidar scanner may be used, together with the camera units, to produce a 3D model.
  • the Lidar can aim its laser beam in a wide range: its head rotates horizontally, a mirror flips vertically.
  • the laser beam is used to measure the distance to the first object on its path.
  • the time-of-flight 3D laser scanner is an active scanner that uses laser light to probe the subject.
  • a time-of-flight laser range finder finds the distance of a surface by timing the round- trip time of a pulse of light.
  • a laser is used to emit a pulse of light and the amount of time before the reflected light is seen by a detector is measured. Since the speed of light c is known, the round-trip time determines the travel distance of the light, which is twice the distance between the scanner and the surface.
  • the accuracy of a time-of-flight 3D laser scanner depends on how precisely we can measure the t; 3.3 picoseconds (approx.) is the time taken for light to travel 1 millimetre.
  • the laser range finder only detects the distance of one point in its direction of view.
  • the scanner scans its entire field of view one point at a time by changing the range finder's direction of view to scan different points.
  • the view direction of the laser range finder can be changed either by rotating the range finder itself, or by using a system of rotating mirrors. The latter method is commonly used because mirrors are much lighter and can thus be rotated much faster and with greater accuracy.
  • Typical time-of-flight 3D laser scanners can measure the distance of 10,000-100,000 points every second.
  • the image capturing system uses a structured-light 3D scanner that projects a pattern of light on the subject and look at the deformation of the pattern on the subject.
  • the pattern is projected onto the subject using either an LCD projector or other stable light source.
  • a camera offset slightly from the pattern projector, looks at the shape of the pattern and calculates the distance of every point in the field of view.
  • structured-light 3D scanners is speed and precision. Instead of scanning one point at a time, structured light scanners scan multiple points or the entire field of view at once. Scanning an entire field of view in a fraction of a second reduces or eliminates the problem of distortion from motion.
  • the display unit may be a display arranged e.g. at a control unit or in the vehicle.
  • the display unit 14 is a pair of glasses, for example of the type sold under the trademark Hololens.
  • the pair of glasses is structured to present the 3D representation such that the 3D representation is overlaid on the
  • the display unit 14 is a pair of virtual reality goggles. These types of goggles comprise two displays to be arranged in front of the operator's eyes. This variation is particularly advantageous when the operator has no direct line of sight to an object to be handled. Often VR goggles are provided with an orientation sensor that senses the orientation of the VR goggles. It may then be possible for a user to change the field of view to locate potential obstacles close to the load, provided that the object detecting device has a larger field of vision than the image presented at the displays of the VR goggles.
  • the present invention also relates to a method applied by an operator assistance system for a vehicle provided with a working equipment. The method will now be described in detail with reference to the flow diagram shown in figure 5.
  • the assistance system is described above and comprises thus an image capturing system arranged at the vehicle and/or at the working equipment, and capable of capturing parameters related to images of the working equipment and of the environment outside said vehicle in a predetermined field of view.
  • the assistance system further comprises a processing unit configured to receive image related parameter signals from the image capturing system and to process the image related parameter signals, and a display unit configured to present images to an operator.
  • the image capturing system comprises at least two sensor unit assemblies capable of capturing parameters related to essentially overlapping images of said predetermined field of view.
  • the sensor unit assembly may be a camera unit, an angle sensor, a length sensor, or any other sensor unit capable of capturing parameters related to images.
  • the processing unit is configured to generate a merged image based upon said overlapping images.
  • the method then comprises determining a shape and position of an image representation of the working equipment occluding a part of an image in the predetermined field of view, by processing image related signals, and determining if the part occluded by the working equipment is visible in any image in the predetermined field of view obtained by any of the other sensor unit assemblies. If the occluded part is visible by any of the other sensor unit assemblies, the method further comprises determining an occluded part image, being an image
  • the method preferably comprises identifying a predetermined part, e.g. a hook, a load, a fork, of the occluding working equipment to be visible in the merged image, and then displaying the predetermined part at the display unit.
  • a predetermined part e.g. a hook, a load, a fork
  • the method comprises determining a presentation mode of the occluded part image.
  • the presentation mode is determined from a set of presentation modes including a transparent mode, wherein the working equipment defining the occluded part image is fully transparent, and a semi-transparent mode, wherein the working equipment defining the occluded part image has a variable opacity.
  • the method further comprises determining a presentation mode where an outer boundary of the occluded part image is indicated in the merged image, e.g. by a dashed line.

Landscapes

  • Engineering & Computer Science (AREA)
  • Automation & Control Theory (AREA)
  • Mechanical Engineering (AREA)
  • Component Parts Of Construction Machinery (AREA)

Abstract

An operator assistance system (2) for a vehicle (4) provided with a working equipment (6), the assistance system comprises an image capturing system (8) arranged at said vehicle and/or at said working equipment (6), and capable of capturing parameters related to images, of said working equipment (6) and of the environment outside said vehicle in a predetermined field of view (10). A processing unit (12) is provided configured to receive image related parameter signals (14) from said image capturing system (8) and to process said image related parameter signals (14), and a display unit (16) configured to present images to an operator. The processing unit (12) is configured to determine a shape and position of an image representation of said working equipment (6) occluding a part of an image in the predetermined field of view (10), obtained by one of at least two sensor unit assemblies (16, 18), by processing image related signals from at least one of the other sensor unit assemblies, the processing unit (12) is further configured to determine if the part occluded by said working equipment (6) is visible in any image in the predetermined field of view (10) obtained by any of the other sensor unit assemblies. If the occluded part is visible by any of the other sensor unit assemblies, an occluded part image, being an image representation of said occluded part, is determined in the field of view (10) obtained by said other sensor unit assembly. The processing unit is configured to merge said occluded part image into said merged image in said determined position of said working equipment (6), and to display said merged image at said display unit (16).

Description

Operator assistance system and a method in relation to the system Technical field
The present disclosure relates to an operator assistance system and a method in connection with the system. The operator assistance system is in particular used on a vehicle provided with a working equipment, e.g. a crane, in assisting the operator during loading and unloading procedures.
Background
Working vehicles are often provided with various working equipment, e.g.
movable cranes, which are attached to the vehicle via a joint. These cranes comprise movable crane parts, e.g. booms, that may be extended, and that are joined together by joints such that the crane parts may be folded together at the vehicle and extended to reach a load. Various tools, e.g. buckets or forks, may be attached to the crane tip, often via a rotator.
Today, many working vehicles are provided with various camera systems applied to display images of the load and the environment around the vehicle in order to assist the operator during the working procedure. Full visibility of terrain and targets is an important requirement in the design and operation of mobile work machines. The machine operator must see the work targets all the time while the machine and its parts, such as a boom, are moving. Furthermore, the operator must be aware of any bystanders being around or passing by the machine.
However, providing full visibility is not always possible due to occlusions caused by the working equipment, e.g. by boom motion or obstructions caused by the machine's profile. The lack of visibility slows down the use of the machine, endangers others working and moving in the area, and physically stresses the operator due to constantly avoiding and trying to look around any obstacles occluding the sight for the operator. The object of the present invention is to achieve an operator assistance system provided with a digital see-through capability for vehicles with a working equipment that solves the aforementioned visibility issues. Summary
The above-mentioned object is achieved by the present invention according to the independent claims.
Preferred embodiments are set forth in the dependent claims.
According to a first aspect, the invention relates to an operator assistance system for a vehicle provided with a working equipment. The assistance system comprises an image capturing system arranged at the vehicle and/or at the working equipment and capable of capturing parameters related to images, of the working equipment and of the environment outside the vehicle in a predetermined field of view. The operator assistance system further comprises a processing unit configured to receive image related parameter signals from the image capturing system and to process the image related parameter signals, and a display unit configured to present images to an operator. The image capturing system comprises at least two sensor unit assemblies capable of capturing parameters related to essentially overlapping images of the predetermined field of view, and that the processing unit is configured to generate a merged image based upon said overlapping images.
The processing unit is configured to determine a shape and position of an image representation of the working equipment occluding a part of an image in the predetermined field of view, obtained by one of the sensor unit assemblies, by processing image related signals from at least one of the other sensor unit assemblies. The processing unit is further configured to determine if the part occluded by the working equipment is visible in any image in the predetermined field of view obtained by any of the other sensor unit assemblies. If the occluded part is visible by any of the other sensor unit assemblies, an occluded part image, being an image representation of the occluded part, is determined in the field of view obtained by said other sensor unit assembly.
The processing unit is then configured to merge the occluded part image into the merged image in the determined position of the working equipment, and to display the merged image at the display unit.
According to an embodiment at least one of the sensor unit assemblies comprises at least one angle sensor and/or at least one length sensor structured to be arranged at the working equipment, and adapted to measure angles and lengths related to movements of the working equipment, and at least one camera unit mounted at the vehicle and/or at the working equipment (6).
According to another embodiment the sensor unit assemblies comprises at least two camera units that are mounted at separate mounting positions at the vehicle in relation to the working equipment, or at the working equipment, such that different sides of the working equipment are visible at images obtained by the at least two camera units.
According to one embodiment the processing unit is provided with a set of image representations of the working equipment, and is further configured to apply a pattern recognition algorithm to identify any image representation of the working equipment in an image in the predetermined field of view by comparison to the set of image representations. Thereby it is assured that a safe and fast recognition of an occluding object is achieved.
According to another embodiment the processing unit is further configured to identify a predetermined part, e.g. a hook, a load, a fork, of the occluding working equipment to be visible in the merged image, and to display said predetermined part at said display unit. This is advantageous as the operator then easily can manoeuvre the hook or fork as it is clearly visible and highlighted at the display unit. According to still another embodiment the processing unit is configured to determine a presentation mode of the occluded part image, from a set of presentation modes including a transparent mode, wherein the working equipment defining the occluded part image is fully transparent, a semi-transparent mode, wherein the working equipment defining the occluded part image has a variable opacity. This is an important feature as an operator may choose an optimal presentation in dependence of a specific situation.
In a further embodiment of the assistance system the processing unit is
configured to determine a presentation mode where an outer boundary of the occluded part image is indicated in the merged image. By indicating the outer boundary of the working equipment the operator will have a complete overview of the working site and also of the working equipment. The working equipment is a crane, a demountable arm, a boom, or a bucket, or any other tool arranged at a vehicle.
According to a second aspect of the present invention a method is provided that is applied by an operator assistance system for a vehicle provided with a working equipment. The assistance system comprises an image capturing system arranged at the vehicle, and/or at the working equipment, and capable of capturing parameters related to images of the working equipment and of the environment outside the vehicle in a predetermined field of view. The assistance system further comprises a processing unit configured to receive image related parameter signals from the image capturing system and to process the image related parameter signals, and a display unit configured to present images to an operator. The image capturing system comprises at least two sensor unit assemblies capable of capturing parameters related to essentially overlapping images of the predetermined field of view, and that the processing unit is configured to generate a merged image based upon the overlapping images. The method comprises:
- determining a shape and position of an image representation of the working equipment occluding a part of an image in the predetermined field of view, by processing image related signals,
- determining if the part occluded by the working equipment is visible in any image in the predetermined field of view obtained by any of the other sensor unit assemblies, and if the part is visible by any of the other sensor unit assemblies, the method further comprises:
- determining an occluded part image, being an image representation of the occluded part, in the field of view obtained by the other sensor unit assembly,
- merging, by the processing unit, the occluded part image into the merged image in the determined position of the working equipment,
- displaying the merged image at the display unit.
According to one embodiment the method comprises:
- applying a pattern recognition algorithm to identify any image representation of the working equipment in an image in the predetermined field of view by comparison to a set of image representations of the working equipment. Thereby it is assured that a safe and fast recognition of an occluding object is achieved.
According to a further embodiment the method comprises:
- identifying a predetermined part, e.g. a hook, a load, a fork, of the occluding working equipment to be visible in said merged image, and
- displaying the predetermined part at the display unit.
This is advantageous as the operator then easily can manoeuvre the hook or fork as it is clearly visible and highlighted at the display unit.
In still another embodiment the method comprises:
- determining a presentation mode of the occluded part image, from a set of presentation modes including a transparent mode, wherein the working
equipment, defining the occluded part image, is fully transparent, a semi- transparent mode, wherein the working equipment defining the occluded part image has a variable opacity. This is an important feature as an operator may choose an optimal presentation in dependence of a specific situation. According to another further embodiment the method comprises:
- determining a presentation mode where an outer boundary of the occluded part image is indicated in the merged image. By indicating the outer boundary of the working equipment the operator will have a complete overview of the working site and also of the working equipment.
Brief description of the drawings
Figure 1 is a schematic illustration of a vehicle provided with an operator assistance system according to the present invention.
Figure 2 is a schematic illustration of images obtained by sensor unit assemblies, and of a merged image.
Figures 3a-3c are schematic views from above illustrating various aspects of embodiments of the present invention.
Figure 4 is a schematic view from above illustrating another embodiment of the present invention.
Figure 5 is a flow diagram illustrating the method according to the present invention. Detailed description
The operator assistance system, and the method, will now be described in detail with references to the appended figures. Throughout the figures the same, or similar, items have the same reference signs. Moreover, the items and the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.
With references to figure 1 , a block diagram is disclosed schematically illustrating an operator assistance system 2 for a vehicle 4 provided with a working equipment 6. The vehicle may be a cargo vehicle, a truck, a forklift or a working vehicle provided with a working equipment 6 that is e.g. a crane, a demountable arm, a mast boom, a hook-lift equipment provided with a predetermined part 22, e.g. a hook, a fork, or a bucket. The assistance system comprises an image capturing system 8 arranged at the vehicle and/or at the working equipment 6, and capable of capturing parameters related to images of the working equipment 6 and of the environment outside the vehicle in a predetermined field of view 10, a processing unit 12 configured to receive image related parameter signals 14 from the image capturing system 8, to process the image related parameter signals, and a display unit 16 configured to present captured images to an operator.
The image capturing system 8 comprises at least two sensor unit assemblies 18, 20 capable of capturing parameters related to essentially overlapping images of said predetermined field of view 10. The processing unit 12 has a processing capability and is configured to generate a merged image based upon the overlapping images. As will be further discussed below the sensor unit assembly is a camera unit, an angle sensor, a length sensor, or any other sensing unit capable of capturing parameters related to images.
The parameters related to images may be parameters directly related to images, e.g. optical parameters detected by e.g. a camera unit, or indirectly related to images, e.g. various parameters representing movement and position of the working equipment, e.g. angles, lengths, width related to parts of the working equipment.
The processing unit is configured to determine a shape and position of an image representation of the working equipment 6 occluding a part of an image in the predetermined field of view 10, obtained by one of said sensor unit assemblies, by processing image related signals from at least one of the other sensor unit assemblies.
The processing unit 12 is further configured to determine if the part occluded by the working equipment 6 is visible in any image in the predetermined field of view 10 obtained by any of the other sensor unit assemblies. If the occluded part is visible by any of the other sensor unit assemblies, an occluded part image, being an image representation of the occluded part, is determined in the field of view 10 obtained by said other sensor unit assembly.
The processing unit is then configured to merge the determined occluded part image into the merged image in the determined position of the working equipment 6, and to display the merged image at the display unit 16.
According to one embodiment at least one of the sensor unit assemblies comprises at least one angle sensor and at least one length sensor structured to be arranged at the working equipment, and adapted to measure angles and lengths related to movements of said working equipment. In addition at least one camera unit 18, 20 is provided which is mounted at the vehicle 4 and/or at the working equipment 6. The angle sensor(s) and/or the length sensor(s) are arranged at the working equipment and structured to generate sensor signals including angle values and length values which are applied to the processing unit. Based upon those sensor signals and information indicating the type of working equipment the processing unit may determine the shape and position of the working equipment.
According to another embodiment the sensor unit assemblies comprise at least two camera units 18, 20 that are mounted at separate mounting positions at the vehicle 4 in relation to the working equipment 6, or at the working equipment 6, such that different sides of the working equipment are visible at images obtained by the at least two camera units 18, 20.
In one variation, which is schematically illustrated in figure 4, the camera units 18, 20 are mounted at separate mounting positions at the working equipment, e.g. a boom of a crane, and move thereby together with the crane. The definition that the camera units are mounted at different sides of the working equipment should be interpreted broadly, and not being limited to different sides in a horizontal plane. The important aspect is that the view of sights obtained by the camera units mounted at the different sides cover parts that potentially may be occluded by the working equipment during normal use. Thus, the camera units may be mounted at different heights or at other positions where a full overview may be obtained. In one exemplary variation two camera units are mounted at different sides of the working equipment, e.g. at the roof of the operator's cabin, and a third camera unit is mounted at an opposite end of the vehicle. In figure 2 is shown above two images obtained by two camera units arranged at opposite sides of the working equipment, or by a sensor unit assembly at the working equipment, and at least one camera unit. The working equipment is visible at the right image where it occludes parts of the environment and thereby prevents the operator from having complete visual overview. The position and shape of the occluding working equipment is determined, and the positions are then applied in the image obtained by another camera unit to identify the corresponding positions therein. This is schematically illustrated on the image to the left where the image representation of the occluded part is indicated by dashed lines. In this image is also seen the working equipment that occludes a part of the image seen a bit further to the right, which is due to the different positions of the camera units in the horizontal plane. These two images are then merged into the image shown below in figure 2. The merging procedure may be applied by identifying easily identifiable objects in both images and then combine and position the images such that these identified objects correspond to each other. In one embodiment the processing unit is provided with a set of image
representations of the working equipment, and more particularly a set of searchable data representations of images of the working equipment in various views and sizes.
The processing unit then is configured to apply a search procedure to identify any image representation of the working equipment in an image in the predetermined field of view obtained by any of the sensor unit assemblies by comparison to the set of image representations. This is preferably performed by applying a dedicated pattern recognition algorithm capable of comparing captured images with the stored set of data representations of images of the working equipment. The working equipment may occlude a part, e.g. a larger part or a smaller part, of the field of view obtained by one camera unit. Even if only a smaller part of the field of view is occluded the search procedure may be performed. It should be noted that the search procedure preferably is continuously performed for all images obtained by all sensor unit assemblies, e.g. the camera units, and that occluded parts in images obtained from different sensor unit assemblies and in different positions then may be identified, and eventually applied in a merged image.
According to one further embodiment the processing unit is configured to identify a predetermined part 22 of the occluding working equipment (see figure 1 and figures 3a-3c), e.g. a hook, a load, a bucket, or a fork, to be visually presented in the merged image, and to display the predetermined part at the display unit. This is preferably performed by applying another pattern recognition algorithm to be set up in advance dependent on the presently used predetermined part of the working equipment. The merged image is presented to the operator on the display unit. The display unit may be any type of presentation unit configured and arranged such that the operator is provided with the necessary support in order to operate the working equipment safely and efficiently. Various presentation modes are available, where the occluding working
equipment is made invisible, is slightly visible, and optionally having the borders indicated by e.g. dashed lines.
Thus, the processing unit is configured to determine a presentation mode of the occluded part image, from a set of presentation modes including a transparent mode, wherein the working equipment defining the occluded part image is fully transparent and a semi-transparent mode, wherein the working equipment defining the occluded part image has a variable opacity.
In addition the processing unit is configured to determine a presentation mode where an outer boundary of the occluded part image is indicated in the merged image. The boundary of the occluded part image may e.g. be indicated by a dashed line. Figures 3a-3c illustrate various presentation modes.
In figure 3a the boundary of the working equipment 6 and the predetermined part 22 are indicated by dashed lines.
In figure 3b the boundary of the working equipment 6 is indicated by a dashed line and the predetermined part 22 is highlighted by a solid line.
In figure 3c the working equipment 6 is invisible and the predetermined part 22 is highlighted by a solid line.
In the embodiment where two camera units are used they have overlapping field of views. This allows the camera units to simulate human binocular vision, and therefore gives it the ability to capture three-dimensional images, a process known as stereo photography. Stereo cameras may be used for making 3D pictures, or for range imaging. Unlike most other approaches to depth sensing, such as structured light or time-of-flight measurements, stereo vision is a purely passive technology which also works in bright daylight.
As discussed above the image capturing system may comprise at least one camera unit, but may in addition also include one or many sensor units, e.g. angle sensor units, length sensor units, capable of capturing various supporting image data to be supplied to the processing unit.
In one variation the image capturing system applies the Lidar-technology. Lidar is sometimes considered an acronym of Light Detection and Ranging (sometimes Light Imaging, Detection, and Ranging), and is a surveying method that measures distance to a target by illuminating that target with a laser light. Lidar is popularly used to make high-resolution maps, with applications in geodesy, forestry, laser guidance, airborne laser swath mapping (ALSM), and laser altimetry. Lidar sometimes is called laser scanning and 3D scanning, with terrestrial, airborne, and mobile applications.
In still another variation the image capturing system also includes a 3D scanning device. A 3D scanner is a device that analyses a real-world object or environment to collect data on its shape and possibly its appearance (e.g. colour). The collected data can then be used to construct digital three-dimensional models. Many different technologies can be used to build these 3D-scanning devices; each technology comes with its own limitations, advantages and costs. Many limitations in the kind of objects that can be digitised are still present, for example, optical technologies encounter many difficulties with shiny, mirroring or
transparent objects. For example, industrial computed tomography scanning can be used to construct digital 3D models, applying non-destructive testing.
The purpose of a 3D scanner is usually to create a point cloud of geometric samples on the surface of the subject. These points can then be used to extrapolate the shape of the subject (a process called reconstruction). If colour information is collected at each point, then the colours on the surface of the subject can also be determined.
3D scanners share several traits with cameras. Like most cameras, they have a cone-like field of view, and like cameras, they can only collect information about surfaces that are not obscured. While a camera collects colour information about surfaces within its field of view, a 3D scanner collects distance information about surfaces within its field of view. The "picture" produced by a 3D scanner describes the distance to a surface at each point in the picture. This allows the three dimensional position of each point in the picture to be identified.
In still another variation a so-called time-of-flight Lidar scanner may be used, together with the camera units, to produce a 3D model. The Lidar can aim its laser beam in a wide range: its head rotates horizontally, a mirror flips vertically. The laser beam is used to measure the distance to the first object on its path.
The time-of-flight 3D laser scanner is an active scanner that uses laser light to probe the subject. At the heart of this type of scanner is a time-of-flight laser range finder. The laser range finder finds the distance of a surface by timing the round- trip time of a pulse of light. A laser is used to emit a pulse of light and the amount of time before the reflected light is seen by a detector is measured. Since the speed of light c is known, the round-trip time determines the travel distance of the light, which is twice the distance between the scanner and the surface. The accuracy of a time-of-flight 3D laser scanner depends on how precisely we can measure the t; 3.3 picoseconds (approx.) is the time taken for light to travel 1 millimetre. The laser range finder only detects the distance of one point in its direction of view. Thus, the scanner scans its entire field of view one point at a time by changing the range finder's direction of view to scan different points. The view direction of the laser range finder can be changed either by rotating the range finder itself, or by using a system of rotating mirrors. The latter method is commonly used because mirrors are much lighter and can thus be rotated much faster and with greater accuracy. Typical time-of-flight 3D laser scanners can measure the distance of 10,000-100,000 points every second.
In another variation the image capturing system uses a structured-light 3D scanner that projects a pattern of light on the subject and look at the deformation of the pattern on the subject. The pattern is projected onto the subject using either an LCD projector or other stable light source. A camera, offset slightly from the pattern projector, looks at the shape of the pattern and calculates the distance of every point in the field of view. The advantage of structured-light 3D scanners is speed and precision. Instead of scanning one point at a time, structured light scanners scan multiple points or the entire field of view at once. Scanning an entire field of view in a fraction of a second reduces or eliminates the problem of distortion from motion. Some existing systems are capable of scanning moving objects in real-time.
The display unit may be a display arranged e.g. at a control unit or in the vehicle. As an alternative, the display unit 14 is a pair of glasses, for example of the type sold under the trademark Hololens. The pair of glasses is structured to present the 3D representation such that the 3D representation is overlaid on the
transparent glasses through which a user observes the object. Various additional information may also be presented as overlaid information and preferably presented such that the additional information is presented close to an illustrated part of the object. In still another alternative the display unit 14 is a pair of virtual reality goggles. These types of goggles comprise two displays to be arranged in front of the operator's eyes. This variation is particularly advantageous when the operator has no direct line of sight to an object to be handled. Often VR goggles are provided with an orientation sensor that senses the orientation of the VR goggles. It may then be possible for a user to change the field of view to locate potential obstacles close to the load, provided that the object detecting device has a larger field of vision than the image presented at the displays of the VR goggles. The present invention also relates to a method applied by an operator assistance system for a vehicle provided with a working equipment. The method will now be described in detail with reference to the flow diagram shown in figure 5.
The assistance system is described above and comprises thus an image capturing system arranged at the vehicle and/or at the working equipment, and capable of capturing parameters related to images of the working equipment and of the environment outside said vehicle in a predetermined field of view. The assistance system further comprises a processing unit configured to receive image related parameter signals from the image capturing system and to process the image related parameter signals, and a display unit configured to present images to an operator. The image capturing system comprises at least two sensor unit assemblies capable of capturing parameters related to essentially overlapping images of said predetermined field of view. As discussed above the sensor unit assembly may be a camera unit, an angle sensor, a length sensor, or any other sensor unit capable of capturing parameters related to images.
The processing unit is configured to generate a merged image based upon said overlapping images.
The method then comprises determining a shape and position of an image representation of the working equipment occluding a part of an image in the predetermined field of view, by processing image related signals, and determining if the part occluded by the working equipment is visible in any image in the predetermined field of view obtained by any of the other sensor unit assemblies. If the occluded part is visible by any of the other sensor unit assemblies, the method further comprises determining an occluded part image, being an image
representation of the occluded part, in the field of view obtained by the other sensor unit assembly, merging, by said processing unit, the occluded part image into said merged image in said determined position of said working equipment, and finally displaying said merged image at said display unit.
According to one embodiment the method comprises applying a pattern
recognition algorithm to identify any image representation of the working equipment in an image in the predetermined field of view by comparison to a set of image representations of the working equipment.
The method preferably comprises identifying a predetermined part, e.g. a hook, a load, a fork, of the occluding working equipment to be visible in the merged image, and then displaying the predetermined part at the display unit.
In a further embodiment the method comprises determining a presentation mode of the occluded part image. The presentation mode is determined from a set of presentation modes including a transparent mode, wherein the working equipment defining the occluded part image is fully transparent, and a semi-transparent mode, wherein the working equipment defining the occluded part image has a variable opacity.
In addition the method further comprises determining a presentation mode where an outer boundary of the occluded part image is indicated in the merged image, e.g. by a dashed line.
The present invention is not limited to the above-described preferred
embodiments. Various alternatives, modifications and equivalents may be used. Therefore, the above embodiments should not be taken as limiting the scope of the invention, which is defined by the appending claims.

Claims

Claims
1 . An operator assistance system (2) for a vehicle (4) provided with a working equipment (6), the assistance system comprises an image capturing system (8) arranged at said vehicle and/or at said working equipment (6), and capable of capturing parameters related to images, of said working equipment (6) and of the environment outside said vehicle in a predetermined field of view (10), a processing unit (12) configured to receive image related parameter signals (14) from said image capturing system (8) and to process said image related
parameter signals (14), and a display unit (16) configured to present images to an operator, said image capturing system (8) comprises at least two sensor unit assemblies (18, 20) capable of capturing parameters related to essentially overlapping images of said predetermined field of view (10), and that said processing unit (12) is configured to generate a merged image based upon said overlapping images,
c h a r a c t e r i z e d i n that said processing unit (12) is configured to determine a shape and position of an image representation of said working equipment (6) occluding a part of an image in the predetermined field of view (10), obtained by one of said sensor unit assemblies, by processing image related signals from at least one of the other sensor unit assemblies, the processing unit (12) is further configured to determine if the part occluded by said working equipment (6) is visible in any image in the predetermined field of view (10) obtained by any of the other sensor unit assemblies, and if said part is visible by any of the other sensor unit assemblies, an occluded part image, being an image representation of said occluded part, is determined in the field of view (10) obtained by said other sensor unit assembly,
wherein said processing unit is configured to merge said occluded part image into said merged image in said determined position of said working equipment (6), and to display said merged image at said display unit (16).
2. The operator assistance system according to claim 1 , wherein at least one of said sensor unit assemblies comprises at least one angle sensor and/or at least one length sensor structured to be arranged at said working equipment, and adapted to measure angles and lengths related to movements of said working equipment, and at least one camera unit (18, 20) mounted at said vehicle (4) and/or at said working equipment (6).
3. The operator assistance system according to claim 1 or 2, wherein said sensor unit assemblies comprises at least two camera units (18, 20) that are mounted at separate mounting positions at said vehicle (4) in relation to said working equipment (6), or at said working equipment (6), such that different sides of said working equipment are visible at images obtained by said at least two camera units (18, 20).
4. The assistance system according to any of claims 1 -3, wherein said processing unit is provided with a set of image representations of the working equipment, and is further configured to apply a pattern recognition algorithm to identify any image representation of the working equipment in an image in the predetermined field of view by comparison to the set of image representations.
5. The assistance system according to any of claims 1 -4, wherein said processing unit is further configured to identify a predetermined part, e.g. a hook, a load, a fork, of said occluding working equipment to be visible in said merged image, and to display said predetermined part at said display unit.
6. The assistance system according to any of claims 1 -5, wherein said processing unit is configured to determine a presentation mode of said occluded part image, from a set of presentation modes including a transparent mode, wherein said working equipment defining the occluded part image is fully transparent, a semi-transparent mode, wherein said working equipment defining said occluded part image has a variable opacity.
7. The assistance system according to any of claims 1 -6, wherein said processing unit is configured to determine a presentation mode where an outer boundary of the occluded part image is indicated in the merged image.
8. The assistance system according to any of claims 1 -7, wherein said working equipment is a crane, a demountable arm, a boom, or a bucket.
9. A method applied by an operator assistance system (2) for a vehicle
(4) provided with a working equipment (6), the assistance system comprises an image capturing system (8) arranged at said vehicle and/or at the working equipment (6), and capable of capturing parameters related to images of said working equipment (6) and of the environment outside said vehicle in a
predetermined field of view (10), a processing unit (12) configured to receive image related parameter signals (14) from said image capturing system (8) and to process said image related parameter signals (14), and a display unit (16) configured to present images to an operator, said image capturing system (8) comprises at least two sensor unit assemblies (18, 20) capable of capturing parameters related to essentially overlapping images of said predetermined field of view (10), and that said processing unit (12) is configured to generate a merged image based upon said overlapping images,
c h a r a c t e r i z e d i n that method comprises:
- determining a shape and position of an image representation of said working equipment (6) occluding a part of an image in the predetermined field of view (10), by processing image related signals,
- determining if the part occluded by said working equipment (6) is visible in any image in the predetermined field of view (10) obtained by any of the other sensor unit assemblies, and if said part is visible by any of the other sensor unit assemblies, the method further comprises:
- determining an occluded part image, being an image representation of said occluded part, in the field of view (10) obtained by said other sensor unit assembly,
- merging, by said processing unit, said occluded part image into said merged image in said determined position of said working equipment (6),
- displaying said merged image at said display unit (16).
10. The method according to claim 9 comprising:
- applying a pattern recognition algorithm to identify any image representation of the working equipment in an image in the predetermined field of view by comparison to a set of image representations of the working equipment.
1 1 . The method according to claim 9 or 10, comprising:
- identifying a predetermined part, e.g. a hook, a load, a fork, of said occluding working equipment to be visible in said merged image, and
- displaying said predetermined part at said display unit.
12. The method according to any of claims 9-1 1 , comprising:
- determining a presentation mode of said occluded part image, from a set of presentation modes including a transparent mode, wherein said working equipment defining the occluded part image is fully transparent, a semi- transparent mode, wherein said working equipment defining said occluded part image has a variable opacity.
13. The method according to any of claims 9-12, comprising:
- determining a presentation mode where an outer boundary of the occluded part image is indicated in the merged image.
EP18762407.7A 2017-09-26 2018-08-16 Operator assistance system and a method in relation to the system Active EP3687937B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
SE1751193 2017-09-26
PCT/SE2018/050829 WO2019066693A1 (en) 2017-09-26 2018-08-16 Operator assistance system and a method in relation to the system

Publications (2)

Publication Number Publication Date
EP3687937A1 true EP3687937A1 (en) 2020-08-05
EP3687937B1 EP3687937B1 (en) 2021-10-06

Family

ID=63442763

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18762407.7A Active EP3687937B1 (en) 2017-09-26 2018-08-16 Operator assistance system and a method in relation to the system

Country Status (2)

Country Link
EP (1) EP3687937B1 (en)
WO (1) WO2019066693A1 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102019112954A1 (en) * 2019-05-16 2020-11-19 Jungheinrich Aktiengesellschaft Method for supporting the position of an industrial truck and industrial truck
JP7613609B2 (en) * 2021-11-30 2025-01-15 株式会社タダノ Pilot assistance system and work vehicle

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4951639B2 (en) * 2009-03-02 2012-06-13 日立建機株式会社 Work machine with ambient monitoring device
JP5941663B2 (en) * 2011-11-30 2016-06-29 住友建機株式会社 Construction machine monitoring system
JP6258594B2 (en) * 2013-03-28 2018-01-10 三井造船株式会社 Crane cab and crane

Also Published As

Publication number Publication date
WO2019066693A1 (en) 2019-04-04
EP3687937B1 (en) 2021-10-06

Similar Documents

Publication Publication Date Title
US11292700B2 (en) Driver assistance system and a method
EP3589575B1 (en) A vehicle provided with an arrangement for determining a three dimensional representation of a movable member
WO2018169467A1 (en) A vehicle with a crane with object detecting device
EP3235773B1 (en) Surrounding information-obtaining device for working vehicle
US10132611B2 (en) Laser scanner
US7974461B2 (en) Method and apparatus for displaying a calculated geometric entity within one or more 3D rangefinder data sets
US7777761B2 (en) Method and apparatus for specifying and displaying measurements within a 3D rangefinder data set
US7403268B2 (en) Method and apparatus for determining the geometric correspondence between multiple 3D rangefinder data sets
US20060193521A1 (en) Method and apparatus for making and displaying measurements based upon multiple 3D rangefinder data sets
EP3657455B1 (en) Methods and systems for detecting intrusions in a monitored volume
JPH06293236A (en) Travel environment monitoring device
EP2805180A1 (en) Laser tracker with functionality for graphical target preparation
JP2020142903A (en) Image processing apparatus and control program
CN104228683A (en) Method for operating a driver assist system for maneuvering and/or parking a motor vehicle
EP3687937B1 (en) Operator assistance system and a method in relation to the system
EP4227708A1 (en) Augmented reality alignment and visualization of a point cloud
KR102332616B1 (en) Display device for construction equipment using LiDAR and AR
JP2002354466A (en) Ambient monitoring device for vehicles
US20230300459A1 (en) Software camera view lock allowing editing of drawing without any shift in the view
US20240320933A1 (en) Systems and methods for visualizing floor data in mixed reality environment
US20240161435A1 (en) Alignment of location-dependent visualization data in augmented reality
EP4655612A1 (en) Image-based localization and tracking using three-dimensional data
JP2000171222A (en) Method and device for three-dimensional input
WO2024102428A1 (en) Alignment of location-dependent visualization data in augmented reality
JP2025004443A (en) Underwater work equipment

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20200228

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20210325

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: HIAB AB

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20210615

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1436070

Country of ref document: AT

Kind code of ref document: T

Effective date: 20211015

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018024707

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20211006

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220106

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220206

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220207

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220106

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220107

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602018024707

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20220707

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220816

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220831

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220831

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20220831

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220816

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20220831

REG Reference to a national code

Ref country code: AT

Ref legal event code: UEP

Ref document number: 1436070

Country of ref document: AT

Kind code of ref document: T

Effective date: 20211006

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20180816

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 602018024707

Country of ref document: DE

Owner name: HIAB AB, SE

Free format text: FORMER OWNER: HIAB AB, KISTA, SE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20250827

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20250826

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20250825

Year of fee payment: 8

Ref country code: AT

Payment date: 20250819

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211006