EP3258443A1 - Informationsverarbeitungsvorrichtung und -verfahren - Google Patents

Informationsverarbeitungsvorrichtung und -verfahren Download PDF

Info

Publication number
EP3258443A1
EP3258443A1 EP17158231.5A EP17158231A EP3258443A1 EP 3258443 A1 EP3258443 A1 EP 3258443A1 EP 17158231 A EP17158231 A EP 17158231A EP 3258443 A1 EP3258443 A1 EP 3258443A1
Authority
EP
European Patent Office
Prior art keywords
movement amount
information
reliability
moving object
processing device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP17158231.5A
Other languages
English (en)
French (fr)
Other versions
EP3258443B1 (de
Inventor
Tsuyoshi Takatani
Tsuyoshi Tasaki
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Toshiba Corp
Original Assignee
Toshiba Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Toshiba Corp filed Critical Toshiba Corp
Publication of EP3258443A1 publication Critical patent/EP3258443A1/de
Application granted granted Critical
Publication of EP3258443B1 publication Critical patent/EP3258443B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C22/00—Measuring distance traversed on the ground by vehicles, persons, animals or other moving solid bodies, e.g. using odometers, using pedometers
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C19/00—Gyroscopes; Turn-sensitive devices using vibrating masses; Turn-sensitive devices without moving masses; Measuring angular rate using gyroscopic effects
    • G01C19/56—Turn-sensitive devices using vibrating masses, e.g. vibratory angular rate sensors based on Coriolis forces
    • G01C19/5776—Signal processing not specific to any of the devices covered by groups G01C19/5607 - G01C19/5719
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C3/00—Measuring distances in line of sight; Optical rangefinders
    • G01C3/02—Details
    • G01C3/06—Use of electric means to obtain final indication
    • G01C3/08—Use of electric radiation detectors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

  • the present disclosure relates to an information processing device and an information processing method.
  • a technique for calculating a current movement amount of a moving object For example, there is known a technique for calculating a traveled course and a traveled distance by using information provided by an inertial sensor and information obtained from a camera.
  • an information processing device includes an internal sensor, a first derivation unit, a second derivation unit, a reliability calculation unit, a movement amount calculation unit.
  • the internal sensor monitors monitor information including acceleration of a moving object.
  • the first derivation unit derives a first movement amount of the moving object from the monitor information.
  • the second derivation unit derives a second movement amount of the moving object using surrounding information of the moving object monitored by an external sensor.
  • the reliability calculation unit calculates reliability of the second movement amount.
  • the movement amount calculation unit calculates a current movement amount of the moving object using the first movement amount and the second movement amount when the reliability meets a specific criterion, and calculates the first movement amount as the current movement amount when the reliability does not meet the specific criterion.
  • FIG. 1 is a diagram illustrating an example of an information processing device 10 of an arrangement.
  • the information processing device 10 calculates a current movement amount of a moving object.
  • the arrangement describes a case in which the information processing device 10 is mounted on a moving object 12 as an example.
  • the moving object 12 is a movable object. In the example illustrated in FIG. 1 , the moving object 12 travels in the direction of arrow X.
  • Examples of the moving object 12 include an automobile, a truck, a flying object (manned or unmanned aerial vehicle such as a drone), and a robot.
  • the arrangement describes a case in which the moving object 12 is an automobile.
  • the automobile includes a two-wheeled automobile, a three-wheeled automobile, and a four-wheeled automobile.
  • the arrangement describes a case in which the automobile is the four-wheeled automobile as an example.
  • a movement amount of the moving object 12 includes at least one of a change amount of velocity, a change amount of acceleration, a change amount of orientation, and a change amount of position (travel distance) of the moving object 12.
  • the change amount of position is represented by, for example, a translation vector.
  • the change amount of orientation can be represented by, for example, a rotation matrix, roll, pitch, and yaw angles, a rotation vector, Euler angles, and a quaternion.
  • the arrangement describes a case in which the movement amount is represented by the change amount of position of the moving object 12 and the change amount of orientation of the moving object 12, as an example.
  • the information processing device 10 includes a processing circuit 10A, an output circuit 10C, an external sensor 10G, and an internal sensor 10J.
  • the internal sensor 10J is a sensor for monitoring monitor information.
  • the internal sensor 10J is mounted on the moving object 12 the movement amount of which is subject to calculation.
  • the internal sensor 10J is independent of the environment around the moving object 12, and monitors the monitor information in a stable manner regardless of variations in the environment.
  • the monitor information is information that includes at least one of acceleration of the moving object 12, velocity of the moving object 12, and angular velocity of the moving object 12.
  • the internal sensor 10J is, for example, an inertial measurement unit (IMU), an acceleration sensor, a velocity sensor, or a rotary encoder.
  • IMU inertial measurement unit
  • acceleration sensor acceleration sensor
  • velocity sensor velocity sensor
  • rotary encoder rotary encoder
  • the arrangement describes a case in which the internal sensor 10J is the IMU as an example. Accordingly, in the arrangement, the internal sensor 10J monitors the monitor information including three-axis acceleration and three-axis angular velocity of the moving object 12.
  • the external sensor 10G monitors surrounding information of the moving object 12.
  • the external sensor 10G may be mounted on the moving object 12 the movement amount of which is subject to calculation or mounted on an external of the moving object 12 such as on another moving object and on an external device.
  • the surrounding information is information indicating a status of surroundings including information around the moving object 12 the movement amount of which is subject to calculation.
  • the surroundings of the moving object 12 the movement amount of which is subject to calculation is a region within a predetermined range from the moving object 12.
  • the range is a range that can be monitored by the external sensor 10G. The range may be set in advance.
  • the surrounding information is at least one of a photographic image and distance information of a periphery of the moving object 12.
  • the surrounding information may include position information of the moving object 12.
  • the photographic image is image data obtained by photographing (Hereinafter simply referred to a photographic image).
  • the distance information is information indicating a distance from the moving object 12 to a target.
  • the target is an external point that can be monitored by the external sensor 10G.
  • the position information may use relative positioning or absolute positioning.
  • the external sensor 10G is, for example, a photographing device with which a photographic image is obtained by photographing, a distance sensor (millimeter wave radar, laser sensor, distance image sensor), and a position sensor (global navigation satellite system (GNSS), radio communication device).
  • a distance sensor millimeter wave radar, laser sensor, distance image sensor
  • GNSS global navigation satellite system
  • the arrangement describes a case, as an example, in which the external sensor 10G is the photographing device with which a photographic image is obtained as surrounding information.
  • the external sensor 10G monitors the surrounding information at each predetermined timing and outputs the monitored surrounding information sequentially to the processing circuit 10A.
  • the internal sensor 10J likewise monitors the monitor information at each predetermined timing and outputs the monitored monitor information sequentially to the processing circuit 10A.
  • the timing of the external sensor 10G monitoring and outputting the surrounding information to the processing circuit 10A and the timing of the internal sensor 10J monitoring and outputting the monitor information to the processing circuit 10A may or may not always be the same. When different timings are included, the surrounding information and the monitor information may be monitored and output to the processing circuit 10A at the same timing at least at a predetermined interval.
  • the arrangement describes a case, as an example, in which the external sensor 10G and the internal sensor 10J respectively monitor the surrounding information and the monitor information at the same timings and outputs the information to the processing circuit 10A at the same timings.
  • the processing circuit 10A calculates a current movement amount of the moving object 12 using the surrounding information (a photographic image is used in the arrangement) monitored by the external sensor 10G and the monitor information monitored by the internal sensor 10J. The details of the processing circuit 10A are described further below.
  • the output circuit 10C outputs output information.
  • the output information includes information relating to the current movement amount of the moving object 12. The details of the output information are described further below.
  • the output circuit 10C includes a display function for displaying output information, a sound output function for outputting sound indicating the output information, and a communication function for transmitting the output information.
  • the output circuit 10C includes a communication circuit 10D, a display device 10E, and a speaker 10F.
  • the output circuit 10C is preferably provided in a position from which the output information can be output to the driver of the moving object 12.
  • the communication circuit 10D transmits information such as the output information to another device.
  • the communication circuit 10D transmits the output information to a terminal device carried by the driver of the moving object 12 through a known communication line.
  • the display device 10E displays output information.
  • the display device 10E is, for example, a known liquid crystal display (LCD), a projection device, and a lighting device.
  • the speaker 10F outputs a sound indicating the output information.
  • FIG. 2 is a block diagram illustrating a configuration example of the information processing device 10.
  • the information processing device 10 is, for example, a special or general-purpose computer.
  • the information processing device 10 includes the processing circuit 10A, a storage circuit 10B, the output circuit 10C, the external sensor 10G, the internal sensor 10J, and an input device 10H.
  • the output circuit 10C includes, as described above, the communication circuit 10D, the display device 10E, and the speaker 10F.
  • the processing circuit 10A, the storage circuit 10B, the output circuit 10C, the external sensor 10G, the internal sensor 10J, and the input device 10H are connected through a bus 10I.
  • the storage circuit 10B, the output circuit 10C (communication circuit 10D, display device 10E, and speaker 10F), the external sensor 10G, the internal sensor 10J, and the input device 10H may be connected wirelessly or wired to the processing circuit 10A.
  • At least one of the storage circuit 10B, the output circuit 10C (communication circuit 10D, display device 10E, and speaker 10F), the external sensor 10G, the internal sensor 10J, and the input device 10H may be connected to the processing circuit 10A via a network.
  • At least the internal sensor 10J needs to be mounted on the moving object 12 the movement amount of which is subject to calculation. Meanwhile, at least one of the other components (processing circuit 10A, storage circuit 10B, output circuit 10C (communication circuit 10D, display device 10E, and speaker 10F), external sensor 10G, and input device 10H), except for the internal sensor 10J, may be mounted on an external of the moving object 12.
  • the arrangement describes a case, as an example, in which the processing circuit 10A, the storage circuit 10B, the output circuit 10C, the external sensor 10G, the internal sensor 10J, and the input device 10H are mounted on the moving object 12.
  • the input device 10H receives various instructions and information input from a user.
  • the input device 10H is, for example, a pointing device such as a mouse and a trackball, or an input device such as a keyboard.
  • the storage circuit 10B stores therein various data.
  • the storage circuit 10B is, for example, a semiconductor memory element such as a random access memory (RAM) and a flash memory, a hard disk, and an optical disk.
  • the storage circuit 10B may be a storage device provided in an external of the information processing device 10.
  • the storage circuit 10B may be a storage medium.
  • the storage medium may be the storage medium that stores or temporarily stores therein computer programs or various information downloaded via a local area network (LAN) or the Internet.
  • the storage circuit 10B may include a plurality of storage media.
  • the processing circuit 10A includes a derivation function 10K, a derivation function 10S, a reliability calculation function 10L, a movement amount calculation function 10M, an output control function 10N, and a change function 10Q.
  • Each processing function of the processing circuit 10A is stored in the storage circuit 10B in a form of a computer program that can be executed by a computer.
  • the processing circuit 10A is a processor that implements functions corresponding to the computer programs by reading the computer programs from the storage circuit 10B and executing the computer programs.
  • FIG. 2 describes a case in which a single processing circuit 10A implements the derivation function 10K, the derivation function 10S, the reliability calculation function 10L, the movement amount calculation function 10M, the output control function 10N, and the change function 10Q.
  • a plurality of separate processors for implementing corresponding functions may be combined to constitute the processing circuit 10A.
  • each processor executes a computer program to implement individual function.
  • each processing function may be configured as a computer program to be executed by a single processing circuit or specific functions may be implemented in a dedicated and independent computer program execution circuit.
  • processor refers to a circuit in, for example, a central processing unit (CPU), a graphical processing unit (GPU), an application specific integrated circuit (ASIC), or a programmable logic device such as a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA).
  • CPU central processing unit
  • GPU graphical processing unit
  • ASIC application specific integrated circuit
  • SPLD simple programmable logic device
  • CPLD complex programmable logic device
  • FPGA field programmable gate array
  • the processor implements a function by reading and executing a computer program stored in the storage circuit 10B.
  • the computer program may be directly incorporated into a processor circuit in place of being stored in the storage circuit 10B.
  • the processor implements a function by reading and executing a computer program stored in the circuit.
  • the derivation function 10S is an example of a first derivation unit.
  • the derivation function 10S derives a movement amount of the moving object 12 using the monitor information monitored by the internal sensor 10J.
  • the movement amount derived by the derivation function 10S is referred to as a first movement amount herein.
  • the first movement amount indicates the amount by which the moving object 12 travels during a period from the timing at which the monitor information used to derive the last first movement amount is monitored to the timing at which the monitor information used to derive the current first movement amount is monitored.
  • the first movement amount may be the movement amount of the moving object 12 per unit time at the timing when the monitor information used to derive the current first movement amount is monitored.
  • the derivation function 10S derives the first movement amount including a change amount of position of the moving object 12 and a change amount of orientation of the moving object 12, from the monitor information monitored by the internal sensor 10J.
  • the internal sensor 10J monitors the monitor information including three-axis acceleration and three-axis angular velocity of the moving object 12.
  • the accelerations in x-axis, y-axis, and z-axis included in the monitor information is respectively defined as a x , a y , and a z.
  • the angular velocities in x-axis, y-axis, and z-axis included in the monitor information is respectively defined as ⁇ x , ⁇ y , and ⁇ z .
  • the acceleration vector a indicating the accelerations in the three axes is represented by Expression (1).
  • the angular velocity vector ⁇ indicating the angular velocities in the three axes is represented by Expression (2).
  • T represents a transpose operator.
  • v (v x , v y , v z ) T .
  • a change amount of position ⁇ t 2 of the moving object 12 a change amount of orientation ⁇ r 2 of the moving object 12, and a change amount of velocity ⁇ v 2 of the moving object 12 can be calculated by Expressions (3) to (5), respectively.
  • a predetermined unit of time may be used for ⁇ . More specifically, in the case in which the first movement amount is calculated by using a movement amount of the moving object 12 per unit time, the unit time thereof may be used as ⁇ . In the case in which the first movement amount is calculated by using a period from the last timing of monitoring the monitor information to the current timing of monitoring the monitor information, the period thereof may be used as ⁇ .
  • the derivation function 10S derives, by calculating Expressions (3) to (5), the first movement amount including the change amount of position ⁇ t 2 of the moving object 12, the change amount of orientation ⁇ r 2 of the moving object 12, and the change amount of velocity ⁇ v 2 of the moving object 12.
  • the derivation function 10K is an example of a second derivation unit.
  • the derivation function 10K derives a movement amount of the moving object 12 using surrounding information monitored by the external sensor 10G.
  • the movement amount of the moving object 12 derived by the derivation function 10K is referred to as a second movement amount herein.
  • the second movement amount indicates the amount by which the moving object 12 travels during a period from the timing at which the surrounding information used to derive the last second movement amount is monitored to the timing at which the surrounding information used to derive the current second movement amount is monitored.
  • the period is the same as that indicated by the first movement amount.
  • the second movement amount may be the movement amount of the moving object 12 per unit time at the timing at which the surrounding information used to derive the current second movement amount is monitored.
  • the derivation function 10K derives the second movement amount including the change amount of position of the moving object 12 and the change amount of orientation of the moving object 12, from the surrounding information monitored by the external sensor 10G.
  • the derivation function 10K derives the second movement amount of the moving object 12 using two pieces of surrounding information monitored at different timings by the external sensor 10G.
  • the derivation function 10K derives the second movement amount of the moving object 12 at the first timing.
  • the derivation function 10K derives the second movement amount of the moving object 12 at the first timing by using the surrounding information monitored at the first timing and the surrounding information monitored the second timing.
  • the first timing and the second timing may be any timing as long as they are different from each other.
  • the second timing may be before the first timing (in the past) or after the first timing (in the future).
  • the derivation function 10K stores, in advance, a plurality of pieces of monitor information monitored at different timings in the storage circuit 10B or any other storage medium.
  • the derivation function 10K reads any two pieces of monitor information from the monitor information and uses one piece of the information as the monitor information at the first timing and the other piece of the information as the monitor information at the second timing.
  • the derivation function 10K derives the second movement amount of the moving object 12 at the first timing by using the surrounding information monitored at the first timing and the surrounding information monitored at the second timing.
  • the derivation function 10K detects object points in the surrounding information monitored at the first timing.
  • the object points are points in the surrounding information monitored by the external sensor 10G.
  • the object points include, for example, feature points representing distinctive regions.
  • the feature points are points that identify boundaries between regions in which at least one of luminance, color, and distance therebetween differs by a predetermined value or more.
  • the feature points are, for example, edges in which luminance changes drastically, and corners in which black points of a line segment or edges bend sharply.
  • the object points may be any portions (points) in the surrounding information and are not limited to the feature points.
  • FIG. 3 is an explanatory diagram illustrating derivation of the second movement amount.
  • the moving object 12 bearing the external sensor 10G travels from a position A to a position B in a three-dimensional space S, as illustrated in FIG. 3 , and that the external sensor 10G monitors surrounding information 31 at the timing of the position B (the first timing) and at the timing of the position A (the second timing).
  • the following describes a case in which the derivation function 10K derives the second movement amount from the position A to the position B.
  • the external sensor 10G monitors the surrounding information 31 including various object points 40 by monitoring a real space.
  • the object points 40 include feature points such as edges and corners as described above.
  • the derivation function 10K detects the object points 40 included in the surrounding information 31 to derive the second movement amount.
  • the derivation function 10K derives the second movement amount of the moving object 12 at the first timing using an error between the object points 40 in the surrounding information 31 monitored at the first timing and estimation points that are the estimated object points 40 at the first timing estimated using the surrounding information 31 monitored at the second timing.
  • FIGS. 4A to 4C are explanatory diagrams illustrating the derivation of the second movement amount in the case in which the surrounding information 31 is a photographic image 30.
  • FIG. 4A is an example of the photographic images 30A and 30B monitored by the external sensor 10G.
  • the photographic image 30B is an example of the photographic image monitored at the first timing at the position B (Refer to FIG. 3 ).
  • the photographic image 30A is an example of the photographic image 30 monitored at the second timing at the position A ahead of the position B (Refer to FIG. 3 ).
  • the derivation function 10K detects object points 42b in the photographic image 30B monitored at the first timing.
  • the derivation function 10K detects points that indicate edges and corners, as object points 42b, in the photographic image 30B monitored at the first timing by a known image processing technique. For example, the derivation function 10K detects the object points 42b through corner detection by the Harris operator and edge detection by the Kirsch operator.
  • the derivation function 10K searches for corresponding points 42a that correspond to the object points 42b described above in the photographic image 30A monitored at the second timing different from the first timing (Refer to FIG. 4A ).
  • a known method may be used to find the corresponding points 42a in the photographic image 30A that correspond to the object points 42b in the photographic image 30B.
  • the derivation function 10K may search for the corresponding points 42a by mapping image patches through template matching.
  • the derivation function 10K may use, for example, the speeded up robust features (SURF), the scale invariant feature transform (SIFT), and the accelerated KAZE (AKAZE) to find the corresponding points 42a each feature amount of which matches or is similar to that of the object points 42b in the photographic image 30B.
  • SURF speeded up robust features
  • SIFT scale invariant feature transform
  • AKAZE accelerated KAZE
  • the derivation function 10K derives the second movement amount by using the object points 42b detected from the photographic image 30B of the first timing and the corresponding points 42a found in the photographic image 30A of the second timing.
  • the derivation function 10K derives the second movement amount by using a technique using the essential matrix estimation. Described in the arrangement is an example that uses a five-point algorithm.
  • the five-point algorithm requires five or more pairs of an object point 42b and a corresponding point 42a.
  • the essential matrix is a matrix that includes a relation of position and orientation between the external sensor 10G at the first timing and the external sensor 10G at the second timing. In other words, the essential matrix represents the relation between the position of the object points 42b in the photographic image 30B monitored at the first timing and the position of the corresponding points 42a in the photographic image 30A monitored at the second timing.
  • the technique based on the essential matrix performs singular value decomposition to estimate an essential matrix that satisfies a plurality of pairs of an object point 42b and a corresponding point 42a and calculates, using the result of the singular value decomposition obtained in the course of estimation, a translation vector ⁇ t 1 and a rotation matrix ⁇ R 1 that respectively indicate the change amount of position and the change amount of orientation of the external sensor 10G at the first timing.
  • the derivation function 10K then projects object points 42b detected from the photographic image 30B of the first timing and corresponding points 42a found in the photographic image 30A of the second timing to a three-dimensional virtual space S' in which the external sensor 10G is arranged (Refer to FIG. 4B ).
  • the difference between the object point 42b and its corresponding point 42a in the three-dimensional virtual space S' indicates a projection error 42D illustrated in FIG. 4B .
  • the external sensors 10G are arranged in the three-dimensional virtual space S' at two predetermined positions with each of the external sensors 10G set in a predetermined orientation.
  • the derivation function 10K performs triangulation by using the position and the orientation of the two external sensors 10G arranged in the three-dimensional virtual space S' to calculate a three-dimensional position of each object point 42b in the photographic image 30B and each corresponding point 42a in the photographic image 30A in the three-dimensional virtual space S'.
  • the derivation function 10K then projects, again, the corresponding points 42a in the three-dimensional virtual space S' to the photographic image 30B as the corresponding object points 42b'.
  • the derivation function 10K determines the object points 42b in the photographic image 30B at the first timing and the estimation points 42b', which are the object points 42b estimated using the photographic image 30A monitored at the second timing.
  • the derivation function 10K repeats projection to the three-dimensional virtual space S' and reprojection to the photographic image 30B until a reprojection error 42D', which is the difference between the object points 42b and corresponding estimation points 42b' in the photographic image 30B, becomes equal to or smaller than a set value while changing the orientation and the position of the external sensors 10G in the three-dimensional virtual space S' (Refer to FIG. 4C ).
  • the set value may be defined in advance.
  • a plurality of pairs of an object point 42b and a corresponding point 42a exist in the photographic images 30A and 30B.
  • the derivation function 10K may repeat the projection and reprojection until the largest reprojection error 42D' among the reprojection errors 42D' of the pairs of a corresponding point 42a and an object point 42b becomes equal to or smaller than the set value.
  • the derivation function 10K may repeat the projection and reprojection until the average of the reprojection errors 42D' of the pairs of a corresponding point 42a and an object point 42b becomes equal to or smaller than the set value.
  • the derivation function 10K then derives, as the second movement amount, the amount of change in the position and the orientation of the external sensor 10G at the first timing with respect to the position and the orientation of the external sensor 10G at the second timing, which are set at the point where the reprojection error 42D' becomes equal to or smaller than the set value.
  • the reprojection error 42D' that is equal to or smaller than the set value is referred to as an error 47. That is, the error 47 indicates the reprojection error 42D' used by the derivation function 10K to derive the second movement amount.
  • the reprojection error is calculated by using the initial second movement based on the essential matrix and refines the second movement amount.
  • the reprojection error can be also calculated by using the first movement amount as the initial value of the second movement amount.
  • the change amount of the position and the change amount of the orientation of the external sensor 10G at the first timing with respect to the position and the orientation of the external sensor 10G at the second timing, which are set at the time point where the reprojection error 42D' becomes equal to or smaller than the set value, are respectively represented by a translation vector ⁇ t 1 and a rotation matrix ⁇ R 1 .
  • the derivation function 10K converts the rotation matrix ⁇ R 1 , which indicates the change amount of orientation, to roll, pitch, and yaw angles.
  • the derivation function 10K then derives, as the second movement amount, the change amount of position ⁇ t 1 represented by the translation vector and the change amount of orientation ⁇ r 1 represented by roll, pitch, and yaw angles.
  • the following describes a method for deriving the second movement amount in the case in which the surrounding information 31 monitored by the external sensor 10G is distance information 32.
  • FIGS. 5A and 5B are explanatory diagrams illustrating the derivation of the second movement amount in the case in which the surrounding information 31 is the distance information 32.
  • FIGS. 5A and 5B illustrate a case, as an example, in which the distance information 32 is a distance image that defines a distance from the external sensors 10G for each pixel.
  • Distance information 32B illustrated in FIG. 5B is an example of the distance information 32 monitored at the first timing at the position B (Refer to FIG. 3 ).
  • Distance information 32A illustrated in FIG. 5A is an example of the distance information 32 monitored at the second timing at the position A ahead of the position B (Refer to FIG. 3 ).
  • the derivation function 10K detects object points 44b in the distance information 32B monitored at the first timing (Refer to FIG. 5B ).
  • the object points 44b are detected in the same manner as in the detection of the object points 42b.
  • the derivation function 10K finds corresponding points 44a that correspond to the object points 44b described above in the distance information 32A monitored at the second timing different from the first timing (Refer to FIG. 5A ).
  • the corresponding points 44a may be found in the same manner that the corresponding points 42a are found.
  • the distance information 32A and 32B is the information indicating the distance in a three-dimensional space. Accordingly, the derivation function 10K places the corresponding points 44a of the distance information 32A in the distance information 32B, identifies differences between the placed corresponding points 44a and the corresponding object points 44b, and determines estimation points 44b' of the corresponding object points 44b on the basis of the differences.
  • the derivation function 10K then derives an estimation error 44D' between the object points 44b and the estimation points 44b'. More specifically, the derivation function 10K derives the error between the object points 44b and the estimation points 44b', the object points 44b being in the distance information 32B monitored at the first timing and the estimation points 44b' being estimated using the corresponding points 44a, which correspond to the object points 44b, in the distance information 32A monitored at the second timing.
  • the derivation function 10K converts the rotation matrix ⁇ R 1 to roll, pitch, and yaw angles.
  • the derivation function 10K then derives, as the second movement amount, the change amount of position ⁇ t 1 represented by a translation vector and the change amount of orientation ⁇ r 1 represented by roll, pitch, and yaw angles.
  • the derivation function 10K may use the largest estimation error 44D' among the estimation errors 44D' of the pairs of an object point 44b and an estimation point 44b' to derive the second movement amount.
  • the derivation function 10K may use an average of the estimation errors 44D' of the pairs of an object point 44b and an estimation point 44b' to derive the second movement amount.
  • the estimation error 44D' used by the derivation function 10K to derive the second movement amount is referred to as an error 48.
  • the error 48, the estimation error 44D' used by the derivation function 10K to derive the second movement amount using the distance information 32, and the error 47, the reprojection error 42D' used by the derivation function 10K to derive the second movement amount using the photographic images 30, may be collectively referred to as an error 46 herein.
  • the monitor information monitored by the internal sensor 10J is independent of the environment around the moving object 12 as described above.
  • the surrounding information 31 monitored by the external sensor 10G is dependent on the environment around the moving object 12.
  • the processing circuit 10A may have difficulty in obtaining the surrounding information 31 from the external sensor 10G.
  • the processing circuit 10A may not be able to obtain the surrounding information 31 that is sufficient to accurately search for the object points 40.
  • the processing circuit 10A may not be able to obtain the surrounding information 31 that is sufficient to accurately search for the object points 40.
  • the accuracy of calculation may be decreased if the current movement amount of the moving object 12 is calculated by simply using the second movement amount derived from the surrounding information monitored by the external sensor 10G and the first movement amount derived from the monitor information monitored by the internal sensor 10J.
  • the reliability calculation function 10L calculates reliability of the second movement amount. More specifically, the reliability calculation function 10L calculates the reliability of the second movement amount derived by the derivation function 10K.
  • Reliability means the accuracy of the second movement amount. A higher reliability means a higher accuracy of the second movement amount is (that is, the second movement amount is closer to the amount by which the moving object 12 has traveled in real life).
  • the reliability calculation function 10L calculates the reliability of the second movement amount using at least one of the error 46 mentioned above, the difference between the first and the second movement amount, and the number of times that the external sensor 10G has succeeded in obtaining the surrounding information successively.
  • the following describes a method for calculating the reliability using the error 46.
  • the error 46 includes, as described above, the reprojection error 42D' used to derive the second movement amount (error 47, refer to FIG. 4C ) and the estimation error 44D' used to derive the second movement amount (error 48, refer to FIG. 5B ).
  • the reliability calculation function 10L calculates reliability using the error 46 (error 47, error 48) between the object points 40 (object points 42b, object points 44b) in the surrounding information 31 (photographic image 30, distance information 32) at the first timing used to derive the second movement amount and the object points 40 (object points 42b, object points 44b) estimated as the estimation points (estimation points 42b', estimation points 44b') using the surrounding information 31 (photographic image 30, distance information 32) monitored at the second timing.
  • the reliability calculation function 10L reads a plurality of errors 46 between a plurality of object points 40 (object points 42b, object points 44b) in the surrounding information 31 (photographic image 30, distance information 32) at the first timing used to derive the second movement amount and the corresponding estimation points (estimation points 42b', estimation points 44b'). More specifically, the reliability calculation function 10L reads a plurality of reprojection errors 42D' or a plurality of estimation errors 44D' as a plurality of errors 46. The reliability calculation function 10L then counts the number of the errors 46 that satisfy a specific condition. The reliability calculation function 10L then calculates the number as the reliability of the second movement amount at the first timing.
  • the specific condition may be defined in advance.
  • the specific condition for example, is that the value of the error 46 (reprojection error 42D', estimation error 44D') is equal to or less than a reference value.
  • the reliability calculation function 10L counts, as the reliability, the number of the errors 46 between the object points 40 and the corresponding estimation points the value of which is less than the reference value.
  • the specific condition may be that the difference between the object points 40 and the corresponding estimation points in the feature amount is equal to or less than a reference value.
  • the reliability calculation function 10L derives the feature amount for each of the object points 40 and the corresponding estimation points.
  • the reliability calculation function 10L may calculate, as the reliability, the number of pairs of an object point 40 and an estimation point that have the difference in the feature amount being equal to or lower than the reference value.
  • a feature amount for each of object points 40 and estimation points may be defined in accordance with the edges and corners indicated in each of object points 40 and the estimation points. Specifically, the reliability calculation function 10L derives higher feature amounts for sharper edges indicated in the object points 40 and the estimation points. Likewise, the reliability calculation function 10L derives higher feature amounts for wider corners. The reliability calculation function 10L may take the number of pairs of an object point 40 and a corresponding estimation point that have the difference in the feature amount being equal to or lower than the reference value.
  • the reliability calculation function 10L may calculate reliability using the difference between the first movement amount and the second movement amount.
  • the reliability calculation function 10L calculates the reliability by performing subtraction between the first movement amount and the second movement amount.
  • the reliability calculation function 10L may use the inverse of the difference between the first movement amount and the second movement amount to calculate the reliability. In this case, the reliability calculation function 10L can calculate the reliability by using Expression (6).
  • a 1 ⁇ ⁇ t 1 ⁇ ⁇ t 2 ⁇ + ⁇ ⁇ r 1 ⁇ ⁇ r 2 ⁇
  • a represents the reliability.
  • ⁇ t 1 , ⁇ t 2 , ⁇ r 1 , and ⁇ r 2 are as mentioned above.
  • the reliability calculation function 10L may calculate the reliability by first multiplying the second movement amount and the first movement amount by their respective weighting values and then calculating the difference therebetween.
  • the reliability calculation function 10L sets the respective predetermined weighting values for the displacement amount of the position and the displacement amount of the orientation included in the movement amount.
  • the reliability calculation function 10L adds the respective predetermined weighting values to the displacement amount of position and the displacement amount of orientation included in the second movement amount, or multiplies the displacement amount of position and the displacement amount of orientation included in the second movement amount by the respective weighting values.
  • the reliability calculation function 10L adds respective predetermined weighting values to the displacement amount of position and the displacement amount of orientation included in the first movement amount, or multiplies the displacement amount of position and the displacement amount of orientation included in the first movement amount by the respective predetermined weighting values.
  • the reliability calculation function 10L may use the differences in the displacement amount of position and the displacement amount of orientation, after the respective weighting values added or multiplied, between each pair to determine the reliability.
  • the derivation function 10K derives the second movement amount of the moving object 12 each time the surrounding information is obtained from the external sensor 10G. Accordingly, the reliability calculation function 10L may use, as the reliability, the number of times the derivation function 10K has successively obtained the surrounding information from the time it failed to obtain the surrounding information last time until the time it obtained the surrounding information this time.
  • the movement amount calculation function 10M uses the second movement amount and the first movement amount to calculate the current movement amount of the moving object 12 when the reliability of the second movement amount meets a specific criterion. When the reliability of the second movement amount does not meet the specific criterion, the movement amount calculation function 10M uses the first movement amount as the current movement amount of the moving object 12.
  • Meeting the specific criterion means that the level of reliability of the second movement derived by the derivation function 10K is high enough to be used for the calculation of the current movement amount of the moving object 12.
  • the specific criterion may be defined in advance according to the type of reliability. As described above, the reliability calculation function 10L uses various methods to calculate the reliability. Thus, different criteria may be set in advance for different types of reliability.
  • the movement amount calculation function 10M determines that the reliability meets the specific criterion when the reliability is equal to or higher than a threshold.
  • the movement amount calculation function 10M determines that the reliability meets the specific criterion when the reliability is equao to or less than a threshold.
  • a value indicating the reliability being in a certain range indicates high reliability.
  • the movement amount calculation function 10M may determine that the reliability meets the specific criterion when the reliability is within the range.
  • the value indicating the reliability being out of a certain range indicates high reliability.
  • the movement amount calculation function 10M may determine that the reliability meets the specific criterion when the reliability is out of the range.
  • the threshold used as the specific criterion can be changed.
  • the change function 10Q changes the threshold.
  • the change function 10Q changes the threshold to the value instructed by a user through the operation of an input device 10H.
  • the change function 10Q may change the threshold in accordance with the error 46 used to derive the second movement amount.
  • the movement amount calculation function 10M may use a known technique to calculate the current movement amount of the moving object 12 using the second movement amount and the first movement amount. For example, the movement amount calculation function 10M calculates the current movement amount of the moving object 12 by using, for example, the Gauss-Newton algorithm, the Levenberg-Marquardt algorithm, and the conjugate gradient algorithm.
  • the following describes an example method in which the movement amount calculation function 10M uses the Gauss-Newton algorithm to calculate the current movement amount of the moving object 12.
  • Expression (7) represents a current movement amount x subject to the calculation, where ⁇ t 3 indicates a change amount of position and ⁇ r 3 indicates a change amount of orientation.
  • the current movement amount x is calculated by Expression (8).
  • ⁇ x represents an update amount.
  • the movement amount calculation function 10M updates the movement amount x by using Expression (8).
  • the movement amount calculation function 10M then repeats to update the update amount ⁇ x until the update of the movement amount x converges.
  • the movement amount calculation function 10M determines that the movement amount x has been converged when the update amount ⁇ x or the magnitude of the abovementioned error 46 used to derive the second movement amount becomes equal to or less than a specific value.
  • An error vector indicating the error 46 is referred to as e.
  • the update amount ⁇ x can be represented by Expression (10) following the Jacobian matrix J indicated in Expression (9).
  • ⁇ x ⁇ J T J ⁇ 1 J T e
  • the error vector e can be represented by Expressions (11) and (12).
  • e t ⁇ t 3 ⁇ ⁇ t 2
  • e r ⁇ r 3 ⁇ ⁇ r 2
  • ⁇ t 2 in Expression (11) represents the change amount of position of the moving object 12 indicated by the first movement amount.
  • ⁇ t 3 in Expression (11) represents the change amount of position of the moving object 12 indicated by the current movement amount of the moving object 12.
  • ⁇ r 2 in Expression (12) represents the change amount of orientation of the moving object 12 indicated by the first movement amount.
  • ⁇ r 3 in Expression (12) represents the change amount of orientation of the moving object 12 indicated by the current movement amount of the moving object 12.
  • the movement amount calculation function 10M uses Expression (13) to calculate an essential matrix E 3 for each change amount of position ⁇ t 3 of the moving object 12 and each change amount of orientation ⁇ r 3 of the moving object 12 in the current movement amount x subject to the calculation.
  • the movement amount calculation function 10M uses Expression (16) to calculate object points 40 (object points 42b, object points 44b), uses Expression (17) to calculate corresponding points (corresponding points 42a, corresponding points 44a) corresponding to the object points 40, and uses Expression (18) to calculate the errors 46 between the corresponding pairs of an object point 40 and a corresponding point.
  • p 1 i x 1 i y 1 i 1 T
  • i is an index for each object point 40 (object point 42b, object point 44b) and each corresponding point (corresponding point 42a, corresponding point 44a).
  • the number of object points 40 included in the surrounding information 31 (photographic image 30 or distance information 32) monitored at the first timing is defined as n.
  • the error vector e can be represented by Expression (19).
  • e e t T , e r T , e p 1 , e p 2 , ... , e pn T
  • the movement amount calculation function 10M calculates the current movement amount x of the moving object 12 by using Expression (8).
  • the movement amount calculation function 10M may add weighting values before calculating the current movement amount of the moving object 12 using the second movement amount and the first movement amount.
  • the movement amount calculation function 10M may also calculate the current movement amount of the moving object 12 by adding the predetermined weighting value to at least one of the change amount of position and the change amount of orientation included in the second movement amount and the first movement amount.
  • the movement amount calculation function 10M may further calculate at least one of current position, velocity, acceleration, moving direction, planned moving direction, and trajectory of the moving object 12 by using the calculated movement amount.
  • a known method can be used as a method for the calculation.
  • the output control function 10N is an example of an output control unit.
  • the output control function 10N outputs output information.
  • the output information includes information relating to the current movement amount of the moving object 12.
  • the output information may be the information indicating the current movement amount of the moving object 12 or the information indicating various values calculated from the current movement amount of the moving object 12, such as the current location of the moving object 12, the direction in which the moving object 12 is currently moving forward, and the current orientation of the moving object 12.
  • the output information may further include at least one of current position, velocity, acceleration, moving direction, planned moving direction, and trajectory of the moving object 12 calculated by using the current movement amount of the moving object 12.
  • the output control function 10N generates the output information by applying the current movement amount of the moving object 12 calculated by the movement amount calculation function 10M to the movement amount calculation method in the movement amount calculation function 10M.
  • the output control function 10N controls the output circuit 10C to output the output information.
  • the output circuit 10C accordingly outputs the output information.
  • the communication circuit 10D in the output circuit 10C transmits the output information to an output target, such as a mobile terminal held by the driver of the moving object 12.
  • the output information is, for example, displayed on the display device 10E in the output circuit 10C.
  • the display device 10E illuminates in different colors, strength, and lighting state (illuminated or blinked) for different output information.
  • the output control function 10N controls the speaker 10F to output sound in accordance with the output information.
  • the sound in accordance with the output information may be a sound indicating the output information or an alarm sound corresponding to the output information.
  • FIGS. 6A and 6B are schematic views illustrating examples of output information 50 displayed on the display device 10E.
  • FIGS. 6A is an example of output information 50A in the case in which the movement amount calculation function 10M calculates the first movement amount as the current movement amount of the moving object 12.
  • the output information 50A includes, for example, the information that indicates the current movement amount of the moving object 12 is calculated by only using an IMU, that is, the internal sensor 10J, and the information that specifies the current movement amount of the moving object 12.
  • FIGS. 6B is an example of output information 50B in the case in which the movement amount calculation function 10M calculates the current movement amount of the moving object 12 using the second movement amount and the first movement amount.
  • the output information 50B includes, for example, the information that indicates the current movement amount of the moving object 12 is calculated by using an IMU, that is, the internal sensor 10J, and the external sensor 10G and the information that specifies the current movement amount of the moving object 12.
  • FIG. 7 is a flow chart illustrating an example of a procedure executed by the processing circuit 10A.
  • the derivation function 10S obtains the monitor information from the internal sensor 10J (Step S100). Then, the derivation function 10S derives the first movement amount from the monitor information obtained at Step S100 (Step S102).
  • Step S108 calculates reliability of the second movement amount derived at Step S106 (Step S108). Then, the movement amount calculation function 10M determines whether the reliability calculated at Step S108 meets a specific criterion (Step S110). If the determination is Yes at Step S110 (Yes at Step S110), the process proceeds to Step S112.
  • the movement amount calculation function 10M calculates the current movement amount of the moving object 12 using the first movement amount derived at Step S102 and the second movement amount derived at Step S106 (Step S112). The process proceeds to Step S116.
  • Step S114 the movement amount calculation function 10M calculates the first movement amount derived at Step S102 as the current movement amount of the moving object 12 (Step S114).
  • the output control function 10N outputs, from the output circuit 10C, the output information relating to the current movement amount of the moving object 12 calculated at Step S112 or S114 (Step S116).
  • Step S118 the processing circuit 10A determines whether the process should be ended.
  • the processing circuit 10A makes a decision at Step S118, for example, by determining whether a signal indicating the end of the process is input by a user through an operation instruction of the input device 10H.
  • FIG. 8 is a flow chart illustrating an example of a procedure for deriving the second movement amount at Step S106 (Refer to FIG. 7 )
  • the derivation function 10K obtains the surrounding information 31 monitored at the first timing (Step S200).
  • the surrounding information 31 monitored at the first timing is the surrounding information 31 obtained at Step S104 (Refer to FIG. 7 ).
  • the derivation function 10K detects object points 40 in the surrounding information 31 obtained at Step S200 (Step S202).
  • the derivation function 10K obtains the surrounding information 31 monitored at the second timing different from the first timing (Step S204). The derivation function 10K then searches the surrounding information 31 at the second timing obtained at Step S204 for corresponding points corresponding one-to-one to object points 40 (Step S206).
  • the derivation function 10K estimates, using the object points 40 detected at Step S202 and the corresponding points found at Step S206, a translation vector that indicates the change amount of position of the external sensor 10G and a rotation matrix that indicates the change amount of orientation of the external sensor 10G at the first timing with respect to the position and the orientation of the external sensor 10G at the second timing (Step S208).
  • the derivation function 10K converts the rotation matrix indicating the change amount of orientation, which is estimated at Step S208, to roll, pitch, and yaw angles (Step S210).
  • the derivation function 10K uses the change amount of position represented by the translation vector, which is estimated at Step S208, and the change amount of orientation represented by the roll, pitch, and yaw angles, which is derived at Step S210, to derive the second movement amount (Step S212). The routine is then ended.
  • the information processing device 10 of the arrangement includes the internal sensor 10J, the derivation function 10S (the first derivation unit), the derivation function 10K (the second derivation unit), the reliability calculation function 10L (reliability calculation unit), and the movement amount calculation function 10M (movement amount calculation unit).
  • the internal sensor 10J monitors the monitor information of the moving object 12 including acceleration.
  • the derivation function 10S derives the first movement amount of the moving object 12 from the monitor information.
  • the derivation function 10K derives the second movement amount of the moving object 12 using the surrounding information 31 of the moving object 12 monitored by the external sensor 10G.
  • the reliability calculation function 10L calculates reliability of the second movement amount. When the reliability of the second movement amount meets a specific criterion, the movement amount calculation function 10M calculates the current movement amount of the moving object 12 using the second movement amount and the first movement amount. When the reliability of the second movement amount does not meet the specific criterion, the movement amount calculation function 10M uses the first movement amount as the current movement amount of the moving object 12.
  • the information processing device 10 calculates the current movement amount of the moving object 12 using the second movement amount and the first movement amount when the reliability of the second movement amount derived using the surrounding information 31 monitored at the external sensor 10G meets a specific criterion.
  • the information processing device 10 uses the first movement amount derived using the monitor information monitored by the internal sensor 10J to calculate the current movement amount of the moving object 12.
  • the information processing device 10 can perform an accurate and stable calculation of the movement amount of the moving object 12 even when accuracy of the surrounding information monitored by the external sensor 10G is low.
  • the information processing device 10 can reduce deterioration in calculation accuracy of the movement amount of the moving object 12.
  • FIG. 9 is an example of the hardware configuration of the information processing device 10 according to the arrangement.
  • the information processing device 10 includes a control device such as a central processing unit (CPU) 86, a storage device such as a read only memory (ROM) 88, a random access memory (RAM) 90, a hard disk drive (HDD) 92, an I/F unit 82 serving as an interface among various devices, an output unit 80 for outputting various information such as output information, an input unit 94 for accepting operations of a user, and a bus 96 for connecting each unit.
  • a control device such as a central processing unit (CPU) 86, a storage device such as a read only memory (ROM) 88, a random access memory (RAM) 90, a hard disk drive (HDD) 92, an I/F unit 82 serving as an interface among various devices, an output unit 80 for outputting various information such as output information, an input unit 94 for accepting operations of a user, and a bus 96 for connecting each unit.
  • CPU central processing unit
  • ROM read only memory
  • RAM random access memory
  • HDD hard disk drive
  • each of the functions described above is implemented on the computer when the CPU 86 reads computer programs from the ROM 88 and runs the computer programs on the RAM 90.
  • the computer programs for performing each of the functions described above to be executed on the information processing device 10 of the arrangement may be stored in the HDD 92.
  • the computer programs for performing each of the functions described above to be executed on the information processing device 10 of the arrangement may be provided by being installed in the ROM 88.
  • the computer programs for performing each of the functions described above to be executed on the information processing device 10 of the arrangement may be provided as computer program products.
  • the computer programs may be stored in a computer-readable storage medium such as a compact disc read only memory (CD-ROM), a compact disc-recordable (CD-R), a memory card, a digital versatile disc (DVD), and a flexible disk (FD) in an installable form or in an executable form.
  • the computer programs for performing each of the functions described above to be executed on the information processing device 10 of the arrangement may be stored in a computer connected to a network, such as the Internet, and provided through downloads via the network.
  • the computer programs for performing each of the functions described above to be executed on the information processing device 10 of the arrangement may be provided or distributed through a network such as the Internet.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Electromagnetism (AREA)
  • Image Analysis (AREA)
  • Traffic Control Systems (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
EP17158231.5A 2016-06-14 2017-02-27 Informationsverarbeitungsvorrichtung und -verfahren Active EP3258443B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2016117705A JP6685847B2 (ja) 2016-06-14 2016-06-14 情報処理装置、および、情報処理方法

Publications (2)

Publication Number Publication Date
EP3258443A1 true EP3258443A1 (de) 2017-12-20
EP3258443B1 EP3258443B1 (de) 2020-03-25

Family

ID=58231380

Family Applications (1)

Application Number Title Priority Date Filing Date
EP17158231.5A Active EP3258443B1 (de) 2016-06-14 2017-02-27 Informationsverarbeitungsvorrichtung und -verfahren

Country Status (3)

Country Link
US (1) US10612937B2 (de)
EP (1) EP3258443B1 (de)
JP (1) JP6685847B2 (de)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102095267B1 (ko) * 2018-10-22 2020-03-31 한국항공대학교산학협력단 중력 기반 정자 분리 장치 및 분리 방법
CN109708658B (zh) * 2019-01-14 2020-11-24 浙江大学 一种基于卷积神经网络的视觉里程计方法
JP7332403B2 (ja) 2019-09-11 2023-08-23 株式会社東芝 位置推定装置、移動体制御システム、位置推定方法およびプログラム
JP6988873B2 (ja) * 2019-11-18 2022-01-05 トヨタ自動車株式会社 位置推定装置および位置推定用コンピュータプログラム

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1906140A2 (de) * 2006-09-29 2008-04-02 Topcon Corporation Vorrichtung und Verfahren zur Positionsmessung
US9031782B1 (en) * 2012-01-23 2015-05-12 The United States Of America As Represented By The Secretary Of The Navy System to use digital cameras and other sensors in navigation

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4639140B2 (ja) * 2005-11-17 2011-02-23 株式会社キーエンス 画像処理装置
JP4561732B2 (ja) * 2006-11-20 2010-10-13 トヨタ自動車株式会社 移動体位置測位装置
CN101583844B (zh) * 2007-01-18 2011-11-09 三菱电机株式会社 车载导航装置
JP2010151459A (ja) * 2008-12-24 2010-07-08 Seiko Epson Corp 位置算出方法及び位置算出装置
WO2011068184A1 (ja) * 2009-12-03 2011-06-09 独立行政法人産業技術総合研究所 移動体の測位装置
CA2812723C (en) 2010-09-24 2017-02-14 Evolution Robotics, Inc. Systems and methods for vslam optimization
JP6003673B2 (ja) * 2013-01-23 2016-10-05 株式会社デンソー ３次元位置推定装置、車両制御装置、および３次元位置推定方法
US9121866B2 (en) * 2013-03-15 2015-09-01 Autoliv Asp, Inc. System and method for inertial sensor offset compensation
JP6702543B2 (ja) 2016-05-31 2020-06-03 株式会社東芝 情報処理装置、方法およびプログラム

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1906140A2 (de) * 2006-09-29 2008-04-02 Topcon Corporation Vorrichtung und Verfahren zur Positionsmessung
US9031782B1 (en) * 2012-01-23 2015-05-12 The United States Of America As Represented By The Secretary Of The Navy System to use digital cameras and other sensors in navigation

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
BRIAN WILLIAMS ET AL: "Feature and pose constrained visual Aided Inertial Navigation for computationally constrained aerial vehicles", ROBOTICS AND AUTOMATION (ICRA), 2011 IEEE INTERNATIONAL CONFERENCE ON, IEEE, 9 May 2011 (2011-05-09), pages 431 - 438, XP032033858, ISBN: 978-1-61284-386-5, DOI: 10.1109/ICRA.2011.5979997 *
MOURIKIS A I ET AL: "A Multi-State Constraint Kalman Filter for Vision-aided Inertial Navigation", 2007 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION - 10-14 APRIL 2007 - ROMA, ITALY, IEEE, PISCATAWAY, NJ, USA, 10 April 2007 (2007-04-10), pages 3565 - 3572, XP031389349, ISBN: 978-1-4244-0601-2 *

Also Published As

Publication number Publication date
EP3258443B1 (de) 2020-03-25
US20170356756A1 (en) 2017-12-14
JP6685847B2 (ja) 2020-04-22
US10612937B2 (en) 2020-04-07
JP2017224071A (ja) 2017-12-21

Similar Documents

Publication Publication Date Title
US20210012520A1 (en) Distance measuring method and device
US10884110B2 (en) Calibration of laser and vision sensors
US11397088B2 (en) Simultaneous localization and mapping methods and apparatus
CN110869700B (zh) 用于确定车辆位置的系统和方法
US10634777B2 (en) Radar odometry for vehicle
US11151741B2 (en) System and method for obstacle avoidance
US11373418B2 (en) Information processing apparatus, information processing method, program, and mobile object
US8320616B2 (en) Image-based system and methods for vehicle guidance and navigation
EP3258443B1 (de) Informationsverarbeitungsvorrichtung und -verfahren
US10895458B2 (en) Method, apparatus, and system for determining a movement of a mobile platform
CN113899375B (zh) 车辆定位方法和装置、存储介质及电子设备
US20190385324A1 (en) Three-dimensional measurement apparatus
US20140286537A1 (en) Measurement device, measurement method, and computer program product
JP2017138660A (ja) 物体検出方法、物体検出装置、およびプログラム
EP2998930A1 (de) Vorrichtung zur schätzung der position beweglicher körper, verfahren zur schätzung der position beweglicher körper und nichttransitorisches speichermedium
JP2020003463A (ja) 自車位置推定装置
KR101821992B1 (ko) 무인비행체를 이용한 목표물의 3차원 위치 산출 방법 및 장치
JP2022190173A (ja) 位置推定装置
JP7179687B2 (ja) 障害物検知装置
CN112486158A (zh) 位置推定装置、移动体控制系统、位置推定方法以及程序
WO2022033139A1 (zh) 一种自运动估计方法及相关装置
US20220309693A1 (en) Adversarial Approach to Usage of Lidar Supervision to Image Depth Estimation
EP4134625A1 (de) Fahrvorrichtung, fahrzeug und verfahren zum automatisierten fahren und/oder assistierten fahren
Nie et al. Real-Time Dense Visual Odometry for RGB-D Cameras
Huber Experimental Analysis of Measurements Fusion for Pose Estimation Using PMD Sensor

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20170227

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20190403

RIC1 Information provided on ipc code assigned before grant

Ipc: G01C 22/00 20060101ALI20190812BHEP

Ipc: G06T 7/73 20170101ALI20190812BHEP

Ipc: G01C 3/08 20060101ALI20190812BHEP

Ipc: G01C 19/5776 20120101ALI20190812BHEP

Ipc: G06T 7/246 20170101AFI20190812BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190925

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: KABUSHIKI KAISHA TOSHIBA

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1249437

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200415

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017013433

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200625

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200626

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200625

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20200325

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200818

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200725

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1249437

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200325

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602017013433

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

26N No opposition filed

Effective date: 20210112

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20210228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210228

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210228

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210227

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210227

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20170227

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231229

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20231229

Year of fee payment: 8

Ref country code: GB

Payment date: 20240108

Year of fee payment: 8