EP3805702B1 - Information processing device - Google Patents

Information processing device Download PDF

Info

Publication number
EP3805702B1
EP3805702B1 EP19812343.2A EP19812343A EP3805702B1 EP 3805702 B1 EP3805702 B1 EP 3805702B1 EP 19812343 A EP19812343 A EP 19812343A EP 3805702 B1 EP3805702 B1 EP 3805702B1
Authority
EP
European Patent Office
Prior art keywords
vehicle
point group
processing apparatus
map
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP19812343.2A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3805702A4 (en
EP3805702A1 (en
Inventor
Morihiko Sakano
Shinya Tagawa
Masahiro Kiyohara
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Faurecia Clarion Electronics Co Ltd
Original Assignee
Clarion Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Clarion Co Ltd filed Critical Clarion Co Ltd
Publication of EP3805702A1 publication Critical patent/EP3805702A1/en
Publication of EP3805702A4 publication Critical patent/EP3805702A4/en
Application granted granted Critical
Publication of EP3805702B1 publication Critical patent/EP3805702B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
    • G01C21/3804—Creation or updating of map data
    • G01C21/3833—Creation or updating of map data characterised by the source of data
    • G01C21/3848—Data obtained from both position sensors and additional sensors
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
    • B60W60/001—Planning or execution of driving tasks
    • B60W60/0027—Planning or execution of driving tasks using trajectory prediction for other traffic participants
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/28—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network with correlation of data from several navigational instruments
    • G01C21/30—Map- or contour-matching
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34—Route searching; Route guidance
    • G01C21/36—Input/output arrangements for on-board computers
    • G01C21/3602—Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34—Route searching; Route guidance
    • G01C21/36—Input/output arrangements for on-board computers
    • G01C21/3679—Retrieval, searching and output of POI information, e.g. hotels, restaurants, shops, filling stations, parking facilities
    • G01C21/3685—Retrieval, searching and output of POI information, e.g. hotels, restaurants, shops, filling stations, parking facilities the POI's being parking facilities
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
    • G01C21/3804—Creation or updating of map data
    • G01C21/3807—Creation or updating of map data characterised by the type of data
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
    • G01C21/3804—Creation or updating of map data
    • G01C21/3807—Creation or updating of map data characterised by the type of data
    • G01C21/3811—Point data, e.g. Point of Interest [POI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G06V20/586—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of parking space
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2520/00—Input parameters relating to overall vehicle dynamics
    • B60W2520/06—Direction of travel
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2520/00—Input parameters relating to overall vehicle dynamics
    • B60W2520/10—Longitudinal speed
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2520/00—Input parameters relating to overall vehicle dynamics
    • B60W2520/12—Lateral speed
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2554/00—Input parameters relating to objects
    • B60W2554/40—Dynamic objects, e.g. animals, windblown objects
    • B60W2554/404—Characteristics
    • B60W2554/4041—Position
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2554/00—Input parameters relating to objects
    • B60W2554/40—Dynamic objects, e.g. animals, windblown objects
    • B60W2554/404—Characteristics
    • B60W2554/4042—Longitudinal speed
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2554/00—Input parameters relating to objects
    • B60W2554/40—Dynamic objects, e.g. animals, windblown objects
    • B60W2554/404—Characteristics
    • B60W2554/4043—Lateral speed
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2554/00—Input parameters relating to objects
    • B60W2554/40—Dynamic objects, e.g. animals, windblown objects
    • B60W2554/404—Characteristics
    • B60W2554/4044—Direction of movement, e.g. backwards

Definitions

  • the present invention relates to an information processing apparatus.
  • Automated parking is a form of such automated driving.
  • automated driving limited to a car park is performed to automatically park the vehicle at a desired parking partition.
  • Patent Literature 1 discloses a robot self-position estimation method that executes at least one of first processing of acquiring map data of a space in which a robot moves, calculating, by simulation, a self-position estimation easiness parameter indicating easiness of estimation of the self-position of the robot for each area in the map data, and presenting information related to the self-position estimation easiness parameter to a user before autonomous travel of the robot, and second processing of causing the robot to estimate the self-position based on the self-position estimation easiness parameter.
  • Patent Literature 1 has a problem that it is difficult to accurately estimate the self-position when the amount of acquired map data is insufficient.
  • FIG. 1 is a configuration diagram of an automated parking system 100 including the information processing apparatus according to the embodiment of the present invention.
  • the automated parking system 100 is mounted on a vehicle 1.
  • the automated parking system 100 includes an external sensor group 102, a car navigation system 107, a vehicle speed sensor 108, a rudder angle sensor 109, an input apparatus 110, a display apparatus 111, a communication apparatus 114, an on-board processing apparatus 120, a vehicle control apparatus 130, a steering apparatus 131, a drive apparatus 132, a braking apparatus 133, a sensor error database 150, and an environment database 151.
  • the external sensor group 102, the car navigation system 107, the vehicle speed sensor 108, the rudder angle sensor 109, the input apparatus 110, the display apparatus 111, the communication apparatus 114, the vehicle control apparatus 130, the sensor error database 150, and the environment database 151 are each coupled with the on-board processing apparatus 120 through a signal line to communicate various signals and various kinds of data with the on-board processing apparatus 120.
  • the external sensor group 102 detects an object existing in surroundings of the vehicle 1 and outputs detection information thereof to the on-board processing apparatus 120.
  • the external sensor group 102 includes a camera 102A configured to perform image capturing of surroundings of the vehicle 1, a sonar 102B configured to observe an object in surroundings of the vehicle 1 by using ultrasonic wave, and a radar 102C configured to observe an object in surroundings of the vehicle 1 by using radio wave or infrared. Not all sensors necessarily need to be included in the external sensor group 102, and another sensor may be included in the external sensor group 102. Any optional sensor capable of detecting an object existing in surroundings of the vehicle 1 may be included in the external sensor group 102.
  • the car navigation system 107 includes a GPS receiver 107A and a storage unit 107B.
  • the GPS receiver 107A receives signals from a plurality of satellites included in a satellite navigation system.
  • the storage unit 107B is a nonvolatile storage apparatus and stores a road map 107C.
  • the road map 107C includes information related to a road coupling structure, and latitude-longitude information corresponding to a road position.
  • the car navigation system 107 calculates, based on information included in signals received by the GPS receiver 107A, a latitude and a longitude indicating the position of the car navigation system 107 on the Earth, in other words, the position of the vehicle 1.
  • a road corresponding to the position of the vehicle 1 is specified by referring to the road map 107C as necessary, and the position of the vehicle 1 is corrected in accordance with the road.
  • the accuracy of the latitude and longitude calculated by the car navigation system 107 does not necessarily need to be high but may include, for example, an error of several m to 10 m approximately.
  • the car navigation system 107 outputs information of the calculated latitude and longitude to the on-board processing apparatus 120.
  • the vehicle speed sensor 108 and the rudder angle sensor 109 measures the vehicle speed and rudder angle, respectively, of the vehicle 1 and outputs the measured vehicle speed and rudder angle to the on-board processing apparatus 120.
  • the on-board processing apparatus 120 can calculate the movement amount and movement direction of the vehicle 1 by a well-known dead reckoning technology using the outputs from the vehicle speed sensor 108 and the rudder angle sensor 109.
  • the input apparatus 110 receives operation command inputting on the on-board processing apparatus 120 by a user and outputs, to the on-board processing apparatus 120, an operation command signal in accordance with the contents of the inputting.
  • the input apparatus 110 includes, for example, a response button 110A and an automated parking button 110B. Details of these buttons will be described later.
  • the display apparatus 111 is, for example, a liquid crystal display and displays an image output from the on-board processing apparatus 120.
  • the input apparatus 110 and the display apparatus 111 may be integrated as, for example, a touch-panel display configured to support a touch operation. In this case, it can be determined that the response button 110A or the automated parking button 110B is pressed when the user touches a predetermined area on the display in accordance with an image displayed on the display.
  • the communication apparatus 114 is used to wirelessly transmit and receive information between an instrument outside the vehicle 1 and the on-board processing apparatus 120.
  • the on-board processing apparatus 120 uses the communication apparatus 114 to transmit and receive information through communication with a portable terminal being brought with the user.
  • the target of communication by the communication apparatus 114 is not limited to the portable terminal of the user, but the communication apparatus 114 may perform communication with an optional instrument capable of performing wireless communication.
  • the vehicle control apparatus 130 controls the steering apparatus 131, the drive apparatus 132, and the braking apparatus 133 based on a vehicle control command from the on-board processing apparatus 120.
  • the steering apparatus 131 operates steering of the vehicle 1.
  • the drive apparatus 132 provides drive power to the vehicle 1.
  • the drive apparatus 132 increases drive power of the vehicle 1 by, for example, increasing a target rotation speed of an engine included in the vehicle 1.
  • the braking apparatus 133 provides braking force to the vehicle 1.
  • the vehicle control apparatus 130 performs these kinds of control in accordance with the vehicle control command from the on-board processing apparatus 120, thereby automatically moving the vehicle 1 without a driving operation by the user to park the vehicle 1 at an optional parking position.
  • the on-board processing apparatus 120 includes a calculation unit 121, a RAM 122, a ROM 123, a storage unit 124, and an interface 125.
  • the calculation unit 121 is, for example, a CPU and performs various kinds of arithmetic processing by executing computer programs. Another arithmetic processing apparatus such as a FPGA may be used to achieve all or some functions of the calculation unit 121.
  • the RAM 122 is a readable and writable storage area and operates as a main storage apparatus of the on-board processing apparatus 120.
  • the ROM 123 is a read-only storage area and stores a computer program executed by the calculation unit 121. The computer program is loaded onto the RAM 122 and executed by the calculation unit 121.
  • the calculation unit 121 reads and executes the computer program loaded onto the RAM 122 to operate as functional blocks of a point group data acquisition unit 121A, a point group data verification unit 121B, a point group map generation unit 121C, an absolute position acquisition unit 121D, a movement amount estimation unit 121E, a position estimation unit 121F, a display control unit 121G, a vehicle control unit 121H, and a mode switching unit 121I.
  • the pieces of information of a point group data 122A, a local surrounding information 122B, and an outlier list 122C are generated by the computer program executed by the calculation unit 121 and are temporarily stored in the RAM 122. Details of these functional blocks and pieces of information will be described later.
  • the storage unit 124 is a nonvolatile storage apparatus and operates as an auxiliary storage apparatus of the on-board processing apparatus 120.
  • the storage unit 124 stores a point group map 124A.
  • the point group map 124A is information in which coordinate values of a plurality of points representing an object existing in surroundings of the vehicle 1 at a place where the vehicle 1 traveled in the past are recorded in association with position information of the place.
  • car park data related to car parks in various locations is stored as the point group map 124A in the storage unit 124.
  • the car park data is a set of a latitude and a longitude indicating position information of each car park, a coordinate value indicating a parking area in the car park, and coordinate values of a plurality of points representing a landmark existing in the car park. Specific configuration and landmarks of the car park data in the point group map 124A will be described later.
  • the interface 125 performs interface processing of information input and output between the on-board processing apparatus 120 and another instrument included in the automated parking system 100.
  • the sensor error database 150 stores error information of the vehicle speed sensor 108 and the rudder angle sensor 109.
  • the environment database 151 stores information on an allowable accuracy of control of the vehicle 1, which is performed by the vehicle control apparatus 130. Values measured or calculated in advance are used as these pieces of information.
  • the on-board processing apparatus 120 can perform verification related to formability of the point group map 124A by using information in these databases. This point will be described later.
  • the landmark is an object existing in surroundings of the vehicle 1 and having a characteristic identifiable by the external sensor group 102 and is, for example, a parking frame line as a kind of road surface paint or a wall of a building as an obstacle that would interfere with traveling of the vehicle 1.
  • moving objects such as a vehicle and a person are not included in landmarks.
  • the on-board processing apparatus 120 detects, based on the detection information input from the external sensor group 102, a landmark existing in surroundings of the vehicle 1, in other words, a point having a characteristic identifiable by the external sensor group 102.
  • the landmark detection performed based on the detection information input from the external sensor group 102 is referred to as "landmark positioning".
  • the camera 102A outputs an image (hereinafter referred to as a captured image) obtained through image capturing of surroundings of the vehicle 1 to the on-board processing apparatus 120.
  • the on-board processing apparatus 120 performs landmark positioning by using the captured image from the camera 102A. Internal parameters such as the focal length and image sensor size of the camera 102A, and external parameters such as the position and posture of attachment of the camera 102A to the vehicle 1 are known and stored in the ROM 123 in advance.
  • the on-board processing apparatus 120 can perform landmark positioning by calculating the positional relation between an object and the camera 102A by using the internal and external parameters stored in the ROM 123.
  • the sonar 102B and the radar 102C radiate sound waves and radio waves, respectively, having specific wavelengths and measure durations until returning reflected waves having reflected on an object in surroundings of the vehicle 1 are received, thereby observing the position of the object. Then, the obtained position of the object is output to the on-board processing apparatus 120.
  • the on-board processing apparatus 120 can perform landmark positioning by acquiring the object position input from the sonar 102B and the radar 102C.
  • the on-board processing apparatus 120 may detect a road surface paint such as a parking frame by operating an image recognition program for a captured image of the camera 102A as described below.
  • a road surface paint such as a parking frame
  • the parking frame detection first, an edge is extracted from an input image through a line extraction filter or the like. Subsequently, for example, a pair of an edge rise that is a change from white to black and an edge fall that is a change from black to white is extracted. Then, when the interval between the pair is substantially equal to a first predetermined distance determined in advance, in other words, the thickness of a white line included in a parking frame, the pair is set as a parking frame candidate.
  • a plurality of parking frame candidates are detected through the same processing, and when the interval between parking frame candidates is substantially equal to a second predetermined distance determined in advance, in other words, the interval between white lines of a parking frame, the parking frame candidates are detected as a parking frame.
  • a road surface paint other than a parking frame is detected by an image recognition program that executes processing as follows. First, edges are extracted from an input captured image through the line extraction filter or the like. Edges can be detected by searching for pixels that have an edge intensity larger than a certain value determined in advance and for which the interval between the edges is equal to a distance determined in advance and corresponding to the width of a white line.
  • the on-board processing apparatus 120 When landmark positioning is performed by using the camera 102A, the on-board processing apparatus 120 preferably detects any other vehicle or person by, for example, known template matching and excludes the detected vehicle or person from a measurement result. In addition, a moving object detected as described below may be excluded from a measurement result. Specifically, the on-board processing apparatus 120 calculates the positional relation between an object in a captured image and the camera 102A by using internal and external parameters. Subsequently, the on-board processing apparatus 120 calculates the relative speeds of the vehicle 1 and the object by tracing the object in captured images continuously acquired by the camera 102A.
  • the on-board processing apparatus 120 calculates the speed of the vehicle 1 by using outputs from the vehicle speed sensor 108 and the rudder angle sensor 109, and when the calculated speed is not equal to the speed relative to the object, the on-board processing apparatus 120 determines that the object is a moving object, and excludes information related to the moving object from a measurement result. Similarly, in landmark positioning using the sonar 102B or the radar 102C, a moving object may be determined and excluded from a measurement result.
  • Figure 2 is a diagram illustrating an exemplary point group map 124A stored in the storage unit 124.
  • Figure 2 illustrates an example in which two pieces of car park data are stored as the point group map 124A.
  • Each piece of car park data in the point group map 124A includes the coordinate value of each apex of a parking area, the coordinate value of each point representing a landmark on a two-dimensional plane, and the coordinate value of each point representing a traveled path at storage.
  • the coordinate value of each point is recorded in combination with corresponding latitude, longitude, and type.
  • the type records, for example, the kind of a landmark performed by the point.
  • the car park data in the point group map 124A is not limited to a coordinate value on a two-dimensional plane but may include the coordinate value of three-dimensional space coordinates including a z coordinate as the altitude.
  • Each coordinate value of car park data in the point group map 124A indicates a coordinate value in a coordinate system unique to the car park data.
  • a coordinate system indicating each coordinate value of car park data is referred to as a "car-park coordinate system”.
  • a latitude and a longitude recorded in association with each coordinate value indicate an absolute position on the Earth.
  • a coordinate system indicating a latitude and a longitude is referred to as an "absolute coordinate system”.
  • the car-park coordinate system is set with reference to a predetermined place in a car park.
  • the origin of the car-park coordinate system is set to be the position of the vehicle 1 when recording of the car park data is started
  • the Y axis of the car-park coordinate system is set to be the traveling direction of the vehicle 1
  • the X axis of the car-park coordinate system is set to be the right direction of the vehicle 1.
  • a place on a travel path through the vehicle 1 traveled in the past is set as a reference position (first reference position), and the relative position of each point with respect to the first reference position is recorded in association with position information including a latitude and a longitude expressed in the absolute coordinate system.
  • the coordinate values of the parking area are recorded as the coordinate values of four apexes of the rectangular area.
  • the parking area is not limited to a rectangular shape but may have a polygonal shape or an elliptical shape other than a rectangular shape.
  • a set of coordinate values of a parking area is recorded in each of the two pieces of car park data, but a plurality of sets of coordinate values of a parking area may be recorded in each piece of car park data. In this manner, in a car park in which a plurality of parking areas exist, the position of each parking area can be expressed in the parking data.
  • the on-board processing apparatus 120 performs position estimation of the vehicle 1 in the car-park coordinate system by using the point group map 124A and the local surrounding information 122B as described later in detail.
  • the outlier list 122C stores information of the local surrounding information 122B not to be processed in the position estimation of the vehicle 1.
  • the outlier list 122C is updated by the on-board processing apparatus 120 as appropriate as described later.
  • the local surrounding information 122B stored in the RAM 122 will be described below.
  • the local surrounding information 122B is used when the position estimation of the vehicle 1 in the car-park coordinate system is performed.
  • the local surrounding information 122B stores the coordinate value of each point included in a landmark detected by the on-board processing apparatus 120 in a position estimation mode to be described later.
  • the coordinate value is set in accordance with a unique coordinate system that is set with reference to the position and posture of the vehicle 1 when recording of the local surrounding information 122B is started.
  • a coordinate system expressing each coordinate value of the local surrounding information 122B is referred to as a "local coordinate system".
  • the origin of the local coordinate system is set to be the position of the vehicle 1 at the start of recording of the local surrounding information 122B
  • the Y axis of the local coordinate system is set to be the traveling direction of the vehicle 1
  • the X axis of the local coordinate system is set to be the right direction of the vehicle 1.
  • a place that is on a travel path traveled by the vehicle 1 and different from that in the point group map 124A described above is set as a reference position (second reference position), and the relative position of each point with respect to the second reference position is indicated.
  • Figure 3 is a diagram for description of operation modes of the on-board processing apparatus 120 and transition conditions thereof.
  • the on-board processing apparatus 120 has the five operation modes illustrated in Figure 3 , namely, a normal travel mode, a map storage mode, the position estimation mode, an automated parking mode, and a give-up mode. These operation modes are switched in accordance with a situation by the mode switching unit 121I.
  • the on-board processing apparatus 120 starts in the normal travel mode.
  • the vehicle 1 In the normal travel mode, the vehicle 1 is driven by the user.
  • the on-board processing apparatus 120 does not perform generation of the point group map 124A and estimation of the own-vehicle position in the point group map 124A but only performs monitoring of a mode transition condition.
  • the mode transition condition is monitored based on the position of the vehicle 1 (hereinafter referred to as a "navigation self-position") acquired from the car navigation system 107 and the point group map 124A.
  • the latitude and longitude of a car park (registered car park) that is registered by the user in advance and in which the vehicle 1 is to be parked are compared with the latitude and longitude of the navigation self-position.
  • mode transition from the normal travel mode is performed.
  • transition to the map storage mode is performed when no car park data near a travel place where the vehicle 1 is currently traveling exists in the point group map 124A, or transition to the position estimation mode is performed when such car park data exists.
  • Whether car park data near the travel place exists in the point group map 124A may be determined based on latitude-longitude information corresponding to the coordinate value of a landmark included in each piece of car park data in the point group map 124A. Specifically, the latitude and longitude of the navigation self-position are compared with the latitude-longitude information of a landmark in each piece of car park data in the point group map 124A, the distance between the travel place and the landmark is calculated from the latitude-longitude difference. As a result, it is determined that car park data near the travel place exists when there is a landmark, the calculated distance of which is within a certain range determined in advance, for example, 10 m, or it is determined that no car park data near the travel place exists when there is no such landmark.
  • the vehicle 1 is driven by the user.
  • the on-board processing apparatus 120 performs landmark positioning based on detection information of surroundings of the vehicle 1, which is acquired from the external sensor group 102 included in the vehicle 1, and collects point group data from which car park data is to be produced, in other words, information of a plurality of points representing a white line, an obstacle, a parking position, a travel path, and the like existing in the registered car park.
  • the on-board processing apparatus 120 produces car park data by using the collected point group data and stores the car park data as part of the point group map 124A in the storage unit 124.
  • the on-board processing apparatus 120 compares the latitude and longitude of the registered car park with the latitude and longitude of the navigation self-position. As a result, when the vehicle 1 is out of a certain range of, for example, 100 m from the registered car park, transition to the normal travel mode is performed. When the vehicle 1 has entered an area for which car park data exists during execution of the map storage mode, transition to the position estimation mode is performed. In this case, whether the vehicle 1 has entered the area for which car park data exists can be determined by a method same as the method for the existence of car park data near the travel place in the normal travel mode.
  • the latitude and longitude of the navigation self-position are compared with the latitude-longitude information of a landmark in each piece of car park data in the point group map 124A, and the distance between the travel place and the landmark is calculated from the latitude-longitude difference.
  • the vehicle 1 has entered the area for which car park data exists when there is a landmark the calculated distance of which is within a certain range determined in advance, for example, 10 m, or it is determined that the vehicle 1 has not entered when there is no such landmark.
  • the vehicle 1 is driven by the user.
  • the on-board processing apparatus 120 collates detection information of surroundings of the vehicle 1, which is acquired from the external sensor group 102 included in the vehicle 1, and information of the point group map 124A, and estimates the position of the vehicle 1 in the car-park coordinate system.
  • the on-board processing apparatus 120 detects a white line and an obstacle in the registered car park, which exist in surroundings of the vehicle 1, based on the detection information from the external sensor group 102 and collates the positions thereof with the coordinate value of each landmark in car park data expressed by the point group map 124A, thereby estimating the current position of the vehicle 1 in the car-park coordinate system.
  • the on-board processing apparatus 120 compares the latitude and longitude of the registered car park with the latitude and longitude of the navigation self-position. As a result, when the vehicle 1 is out of a certain range of, for example, 100 m from the registered car park, transition to the normal travel mode is performed. When the vehicle 1 has entered an area for which no car park data exists during execution of the position estimation mode, transition to the map storage mode is performed. In this case, whether the vehicle 1 has entered the area for which no car park data exists can be determined by a method opposite to a condition on transition from the map storage mode to the position estimation mode.
  • the latitude and longitude of the navigation self-position are compared with the latitude-longitude information of a landmark in each piece of car park data in the point group map 124A, and the distance between the travel place and the landmark is calculated from the latitude-longitude difference.
  • the vehicle 1 has entered the area for which no car park data exists when there is no landmark, the calculated distance of which is out of a certain range determined in advance, for example, 10 m, or it is determined that the vehicle 1 has not entered when there is such a landmark.
  • transition to the automated parking mode is performed.
  • transition to the give-up mode is performed.
  • the on-board processing apparatus 120 moves the vehicle 1 to a parking position specified by the user and parks the vehicle 1 at the parking position based on the position of the vehicle 1 in the car-park coordinate system, which is obtained in the position estimation mode.
  • the on-board processing apparatus 120 estimates the position of the vehicle 1 in the car-park coordinate system.
  • the on-board processing apparatus 120 detects a white line and an obstacle in the registered car park, which exist in surroundings of the vehicle 1, based on detection information from the external sensor group 102 and collates the positions thereof with the point group map 124A, thereby estimating the current position of the vehicle 1 in the car-park coordinate system.
  • the on-board processing apparatus 120 presents, to the user, that automated parking is impossible through the display apparatus 111.
  • the on-board processing apparatus 120 does not perform storage of the point group map 124A, estimation of the position of the vehicle 1 in the car-park coordinate system, nor automated parking control.
  • transition to the normal travel mode is performed.
  • the on-board processing apparatus 120 executes each operation mode as described above.
  • the map storage mode, the position estimation mode, and the automated parking mode will be sequentially described in detail.
  • the on-board processing apparatus 120 when the vehicle 1 has entered a predetermined range centered at the registered car park and no car park data near the travel place is recorded in the point group map 124A, the on-board processing apparatus 120 operates in the map storage mode. While operating in the map storage mode, the on-board processing apparatus 120 displays, on a screen of the display apparatus 111, that the on-board processing apparatus 120 is operating in the map storage mode, and collects the point group data 122A by performing landmark positioning.
  • the point group data verification unit 121B starts operating when the user stops the vehicle 1 after moving the vehicle 1 to a parking position and then actuates the parking brake.
  • the point group data verification unit 121B verifies whether the point group data 122A collected in the map storage mode holds as the point group map 124A, in other words, whether the collected point group data 122A can be used for the position estimation of the vehicle 1 in the position estimation mode and the automated parking mode.
  • the collected point group data 122A cannot be used for the position estimation, automated parking fails because the position estimation cannot be performed by using the point group map 124A produced by using the collected point group data 122A, and thus the verification is performed to prevent this failure.
  • the point group data verification unit 121B sets a verification interval of a predetermined interval on the travel locus of the vehicle 1 and verifies, with reference to two verification factors of local formability and autonomous navigability, whether the point group data 122A acquired at each verification interval can be used for the position estimation.
  • the local formability as the former verification factor is determined based on the state of distribution of a plurality of points in the point group data 122A. Specifically, it is determined that the local formability is satisfied when the density of the point group data 122A acquired in the verification interval is equal to or higher than a predetermined value and there is no identical distribution pattern.
  • the autonomous navigability as the latter verification factor is determined based on an error in estimation of the movement amount of the vehicle 1 and an allowable accuracy for control of the vehicle 1.
  • an error in the position estimation of the vehicle 1 in a well-known dead reckoning technology using the vehicle speed sensor 108 and the rudder angle sensor 109 is acquired from the sensor error database 150, and an allowable position accuracy for control of the vehicle 1, which is determined in advance for a path width and a parking area size in a car park is acquired from the environment database 151. Then, a travel distance that the vehicle 1 can travel by autonomous navigation is calculated based on these acquired pieces of information, thereby determining whether the autonomous navigability is satisfied.
  • the collected point group data 122A can be used for the position estimation even when the above-described two verification factors are both not satisfied. For example, when the error in the position estimation by autonomous navigation is within a sufficiently small allowable accuracy in a verification interval for which the local formability does not hold, it is determined that the point group data 122A acquired in the verification interval can be used for the position estimation and holds as the point group map 124A. In this case, when the local formability does not hold and the error in the position estimation by autonomous navigation is larger than the allowable accuracy, it is determined that the point group data 122A cannot be used for the position estimation.
  • the on-board processing apparatus 120 displays a predetermined message or image on the display apparatus 111 to query the user for whether the point group data 122A acquired this time is to be recorded in the point group map 124A.
  • the response button 110A is pressed down by the user in response to the query, the on-board processing apparatus 120 produces car park data included in the point group map 124A by using the acquired point group data 122A through the point group map generation unit 121C, records the produced car park data in the storage unit 124, and ends the map storage mode.
  • the on-board processing apparatus 120 displays, on the display apparatus 111, that the point group data 122A acquired this time does not sufficiently have information necessary for the point group map 124A. Then, similarly to the case in which it is determined that the point group data 122A can be used for the position estimation, when the response button 110A is pressed down by the user, the on-board processing apparatus 120 produces car park data included in the point group map 124A by using the acquired point group data 122A, records the produced car park data in the storage unit 124, and ends the map storage mode. In this case, a message that prompts the user to travel the same path again and acquire the necessary point group data 122A may be displayed. Alternatively, the map storage mode may be ended without producing car park data. In this case, the point group map 124A is not updated.
  • the operation of the on-board processing apparatus 120 in the map storage mode is divided into the two pieces of processing of extracting the point group data 122A included in a landmark and recording the point group map 124A based on the extracted point group data 122A.
  • the processing of extracting the point group data 122A by the on-board processing apparatus 120 will be described below.
  • the on-board processing apparatus 120 When transition is made to the map storage mode, the on-board processing apparatus 120 reserves a temporary record area in the RAM 122. Then, the on-board processing apparatus 120 repeats processing as follows until the map storage mode ends or transition is made to another mode. Specifically, the on-board processing apparatus 120 extracts the point group data 122A related to a plurality of points representing a landmark based on detection information acquired from the external sensor group 102 by landmark positioning, for example, a captured image of the camera 102A and observation information of the sonar 102B and the radar 102C.
  • the on-board processing apparatus 120 calculates, based on outputs from the vehicle speed sensor 108 and the rudder angle sensor 109, a movement amount by which and a movement direction in which the vehicle 1 has moved until the current image capturing since the previous image capturing by the camera 102A. Then, the on-board processing apparatus 120 calculates the coordinate value of each point included in the extracted point group data 122A based on the positional relation with the vehicle 1 and the movement amount and movement direction of the vehicle 1, and records the calculated coordinate value in the RAM 122. In addition, the on-board processing apparatus 120 calculates the latitude and longitude of each point based on a latitude and a longitude output from the car navigation system 107 and records the calculated latitude and longitude. The on-board processing apparatus 120 repeats this processing in the map storage mode.
  • the coordinate value of each point in the point group data 122A is recorded as a coordinate value in a coordinate system set with reference to the position of the vehicle 1 when recording of the point group data 122A is started.
  • this coordinate system is referred to as a "recording coordinate system".
  • the origin of the recording coordinate system is set to be the position of the vehicle 1 when recording of the point group data 122A is started
  • the Y axis of the recording coordinate system is set to be the traveling direction (posture) of the vehicle 1
  • the X axis of the recording coordinate system is set to be the right direction of the vehicle 1.
  • the coordinate value of a point representing the same landmark is not necessarily constant for the point group data 122A recorded in an identical car park because a different recording coordinate system is set when the position or posture of the vehicle 1 at the start of the recording is different.
  • the on-board processing apparatus 120 calculates the coordinate value of a parking area based on the current position of the vehicle 1 and records the calculated coordinate value in the RAM 122.
  • the coordinate value of the parking area is recorded as, for example, the coordinate values of the four corners of a rectangular shape to which the vehicle 1 is approximated.
  • the coordinate value of the parking area may be determined with taken into account the difference between the vehicle 1 and the size of the parking area.
  • the on-board processing apparatus 120 performs the processing of recording the point group map 124A based on the point group data 122A as described below.
  • the on-board processing apparatus 120 determines whether the difference between the current position of the vehicle 1 at parking completion, in other words, the latitude and longitude of a parking position and the latitude and longitude of any parking area in car park data already recorded in the point group map 124A is in a predetermined range. As a result, when there is no car park data including a parking area, the latitude-longitude difference of which from the parking position is in the predetermined range, the on-board processing apparatus 120 records, in the point group map 124A as new car park data, the point group data 122A stored in the RAM 122.
  • the on-board processing apparatus 120 determines whether to merge, as data of the same car park with the car park data, information of the point group data 122A stored in the RAM 122. For this determination, the on-board processing apparatus 120 first performs coordinate transform of the point group data 122A so that the position of the parking area included in the car park data matches with a parking position in the point group data 122A recorded in the RAM 122. Subsequently, the on-board processing apparatus 120 calculates a point group matching rate indicating the degree of matching between the point group data 122A after the coordinate transform and the point group map 124A.
  • the on-board processing apparatus 120 when having determined that both pieces of data are not to be integrated, the on-board processing apparatus 120 records, in the point group map 124A as new car park data, the point group data 122A stored in the RAM 122. When having determined that both pieces of data are to be integrated, the on-board processing apparatus 120 adds the information of the point group data 122A stored in the RAM 122 to the existing car park data in the point group map 124A.
  • Figure 4 is a flowchart illustrating processing executed by the on-board processing apparatus 120 in the map storage mode.
  • the executor of each step described below is the calculation unit 121 of the on-board processing apparatus 120.
  • the calculation unit 121 functions as the point group data acquisition unit 121A, the point group data verification unit 121B, the point group map generation unit 121C, the absolute position acquisition unit 121D, the movement amount estimation unit 121E, and the display control unit 121G.
  • step S501 the calculation unit 121 determines whether the map storage mode is activated. When having determined that the map storage mode is activated, the calculation unit 121 proceeds to S502. When the map storage mode is not activated, the calculation unit 121 stays at step S501.
  • the calculation unit 121 performs initializing processing.
  • a new storage area is reserved in the RAM 122.
  • the point group data 122A and the position of the vehicle 1, which are extracted by landmark positioning are recorded as coordinate values in the above-described recording coordinate system in the subsequent processing.
  • the calculation unit 121 performs, through the point group data acquisition unit 121A, the above-described landmark positioning by using information acquired from the external sensor group 102. Specifically, observation values of a plurality of points representing a landmark in a car park existing in surroundings of the vehicle 1 are acquired by using a captured image of the camera 102A or observation information of the sonar 102B or the radar 102C and recorded in the RAM 122 as the point group data 122A.
  • the point group data 122A acquired at this stage is a coordinate value indicating a relative position with respect to the current position of the vehicle 1, but is not a coordinate value in the recording coordinate system.
  • the calculation unit 121 performs GPS positioning of calculating the current position of the vehicle 1.
  • latitude-longitude information based on received signals of the GPS receiver 107A, which is output from the car navigation system 107 is acquired by the absolute position acquisition unit 121D.
  • the position of each point in the point group data 122A in the absolute coordinate system is acquired by associating the current position of the vehicle 1, which is indicated by the acquired latitude-longitude information, with the observation value.
  • the calculation unit 121 estimates, through the movement amount estimation unit 121E, the movement amount of the vehicle 1 until the current landmark positioning since the previous landmark positioning and updates the current position of the vehicle 1 in the recording coordinate system, which is recorded in the RAM 122.
  • the movement amount of the vehicle 1 may be estimated by a plurality of methods. For example, as described above, the movement amount of the vehicle 1 may be estimated based on change in the position of an object existing on a road surface in a captured image of the camera 102A. When a highly accurate GPS receiver 107A having a small error is mounted on the car navigation system 107, an output from the GPS receiver 107A may be used.
  • the movement amount of the vehicle 1 may be estimated based on the vehicle speed output from the vehicle speed sensor 108 and the rudder angle output from the rudder angle sensor 109.
  • the estimated movement amount includes, for example, the three parameters of movement amounts ⁇ x and ⁇ y on the two-dimensional plane and ⁇ indicating change in the direction of the vehicle on the plane.
  • the estimated movement amount includes, for example, the three parameters of movement amounts ⁇ x, ⁇ y, and ⁇ z in the three-dimensional space and ⁇ , ⁇ , and ⁇ indicating change in the direction of the vehicle in the space.
  • the parameter ⁇ corresponds to the rotational amount about the x axis
  • the parameter ⁇ corresponds to the rotational amount about the y axis
  • the parameter ⁇ corresponds to the rotational amount about the z axis.
  • the movement amount of the vehicle 1 is estimated by using an optional method, and the current position of the vehicle 1 in the recording coordinate system is updated and then temporarily stored in the RAM 122, and the calculation unit 121 subsequently proceeds to step S505.
  • the calculation unit 121 temporarily records the point group data 122A of the recording coordinate system in the RAM 122.
  • the coordinate value of each point in the point group data 122A acquired by landmark positioning at step S503 is transformed into a coordinate value in the recording coordinate system based on the current position of the vehicle 1 updated at step S504.
  • the point group data 122A in the RAM 122 is updated with the coordinate value after the transform as the point group data 122A in the recording coordinate system.
  • the coordinate value of each point in the point group data 122A recorded in the RAM 122 is rewritten with reference to the position and posture of the vehicle 1 when recording of the point group data 122A is started.
  • the calculation unit 121 determines whether parking of the vehicle 1 is completed. Until the parking brake of the vehicle 1 is actuated, the calculation unit 121 determines that parking is not completed, and returns to step S503 to continue the landmark positioning and the recording of the point group data 122A. When the parking brake of the vehicle 1 is actuated, the calculation unit 121 determines that parking is completed, and proceeds to step S507.
  • the calculation unit 121 determines, through the point group data verification unit 121B, map formability of the point group data 122A recorded in the RAM 122 at step S505.
  • verification related to the formability of the point group map 124A is performs for each point in the point group data 122A by a method as described above.
  • the calculation unit 121 displays, through the display control unit 121G, a result of the determination on the display apparatus 111 and queries the user for whether to record the point group data 122A in the point group map 124A.
  • Information indicating the result of the map formability determination performed at step S507 is recorded in combination with the coordinate value of each point when the point group data 122A is recorded in the point group map data 124A later at step S514 or S515.
  • the calculation unit 121 determines whether storage of the point group map 124A is approved by the user. When a predetermined storage approval operation is performed by the user through the response button 110A, the calculation unit 121 determines that storage of the point group map 124A is approved, and proceeds to step S509. When the storage approval operation is not performed by the user, the calculation unit 121 determines that storage of the point group map 124A is not approved, and ends the flowchart of Figure 4 . In this case, the point group data 122A recorded in the RAM 122 is discarded, and the point group map 124A is not recorded.
  • the calculation unit 121 performs GPS positioning to calculate the parking position of the vehicle 1. Specifically, the latitude-longitude information based on received signals of the GPS receiver 107A, which is output from the car navigation system 107 is acquired by the absolute position acquisition unit 121D to acquire the latitude and longitude of the parking position. Then, the coordinate value of a parking area corresponding to the parking position in the recording coordinate system is recorded based on the acquired latitude and longitude.
  • the coordinate value of the parking position in the recording coordinate system is calculated from the difference between the latitude and longitude of the current position of the vehicle 1, which are acquired last at step S503, and the latitude and longitude of the parking position, which are acquired at step S509, and the coordinate values of four apexes of the parking area corresponding to the parking position are calculated based on the calculated coordinate value and recorded in the RAM 122.
  • the calculation unit 121 determines whether car park data including a parking area, the latitude-longitude difference of which from the parking position acquired at step S509 is in a predetermined range is already recorded in the point group map 124A. In this process, the calculation unit 121 determines whether there are a latitude and a longitude, the difference of which from the latitude and longitude of the parking position is in the predetermined range among latitudes and longitudes corresponding to the coordinate values of parking areas included in each piece of car park data already recorded in the point group map 124A. As a result, when a parking area, the latitude-longitude difference of which from the parking position is in the predetermined range exists in the point group map 124A, the calculation unit 121 proceeds to step S511.
  • step S515. it may be determined that car park data corresponding to the parking position is already recorded in the point group map 124A only when the latitude-longitude difference from the parking position is in the predetermined range for all four apexes of the parking area.
  • car park data in the point group map 124A including a parking area for which it is determined at step S510 that the latitude-longitude difference from the parking position is in the predetermined range is referred to as "target car park data”.
  • the calculation unit 121 performs, through the point group map generation unit 121C, coordinate transform of the point group data 122A recorded in the RAM 122 at step S505 from the recording coordinate system to the car-park coordinate system of the target car park data.
  • a formula of coordinate transform from the recording coordinate system to the car-park coordinate system is derived so that the latitude and longitude of the parking area in the target car park data match with the latitude and longitude of the parking position acquired at step S509.
  • the coordinate transform formula is used to transform, into coordinate values in the car-park coordinate system of the target car park data, the coordinate values of points respectively representing a landmark and the current position of the vehicle 1 in the point group data 122A stored in the recording coordinate system in the RAM 122.
  • the calculation unit 121 calculates a point group matching rate IB of the point group data 122A subjected to the coordinate transform at step S511 and the target car park data.
  • the calculation unit 121 determines whether the point group matching rate IB calculated at step 512 is higher than a predetermined threshold value. As a result, when having determined that the point group matching rate IB is higher than the threshold value, the calculation unit 121 proceeds to step S514. When having determined that the point group matching rate IB is equal to or lower than the threshold value, the calculation unit 121 proceeds to step S515.
  • the calculation unit 121 performs, through the point group map generation unit 121C, processing of merging the point group data 122A subjected to the coordinate transform at step S511 with the target car park data.
  • the merge processing is performed to generate a new point group map 124A by adding the point group data 122A subjected to the coordinate transform at step S511 to the target car park data in the point group map 124A stored in the storage unit 124.
  • step S515 which is executed when the negative determination is performed at step S510 or S513, the calculation unit 121 records, through the point group map generation unit 121C, in the point group map 124A as new car park data, the point group data 122A subjected to the coordinate transform at step S511 and the latitude and longitude of the parking position and the coordinate value of the parking area corresponding to the parking position, which are acquired at step S509.
  • the flowchart of Figure 4 is ended.
  • the on-board processing apparatus 120 transitions to the position estimation mode.
  • Figure 5 is a flowchart illustrating processing executed by the on-board processing apparatus 120 in the position estimation mode.
  • the executor of each step described below is the calculation unit 121 of the on-board processing apparatus 120.
  • the calculation unit 121 functions as the point group data acquisition unit 121A, the absolute position acquisition unit 121D, the movement amount estimation unit 121E, the position estimation unit 121F, and the display control unit 121G.
  • the calculation unit 121 determines whether the position estimation mode is activated. When having determined that the position estimation mode is activated, the calculation unit 121 proceeds to S602. When the position estimation mode is not activated, the calculation unit 121 stays at step S601.
  • the calculation unit 121 performs GPS positioning to calculate the current position of the vehicle 1. Specifically, the latitude-longitude information based on received signals of the GPS receiver 107A, which is output from the car navigation system 107 is acquired by the absolute position acquisition unit 121D to acquire the latitude and longitude corresponding to the current position of the vehicle 1.
  • the calculation unit 121 determines the existence of the map formability of the point group map 124A in surroundings of the current position of the vehicle 1.
  • the existence of car park data including, in path data, a point having a latitude and a longitude in a certain range from the current position of the vehicle 1 and having map formability is determined.
  • the existence of the map formability can be determined based on the above-described information recorded at step S507 in Figure 4 when the point group data 122A acquired in the map storage mode is recorded in the point group map 124A.
  • the calculation unit 121 determines that the map formability exists for the point group map 124A in surroundings of the current position of the vehicle 1, and proceeds to step S604.
  • the calculation unit 121 determines that no map formability exists for the point group map 124A in surroundings of the current position of the vehicle 1, and returns to step S602 to continue the GPS positioning.
  • the calculation unit 121 performs initialization processing.
  • the calculation unit 121 performs initialization of the local surrounding information 122B stored in the RAM 122 and initialization of the current position of the vehicle 1 stored in the RAM 122.
  • any existing recorded information is deleted, and a new coordinate system is set as the above-described local coordinate system.
  • the local coordinate system is set based on the position and posture of the vehicle 1 when step S604 is executed.
  • the origin of the local coordinate system is set to be the position of the vehicle 1 when step S604 is executed, and the X and Y axes of the local coordinate system are set based on the direction of the vehicle 1 when step S604 is executed.
  • the origin (0, 0) is set to be the current position of the vehicle 1.
  • step S605 the calculation unit 121 performs self-position estimation through the position estimation unit 121F.
  • the current position of the vehicle 1 in the car-park coordinate system is estimated in accordance with the procedure of a flowchart illustrated in Figure 6 to be described later.
  • the calculation unit 121 determines whether the self-position, in other words, the current position of the vehicle 1 in the car-park coordinate system is estimated at step S605. In this process, whether the self-position is correctly estimated is determined based on a result of self-position diagnosis performed at step S627 in the flowchart illustrated in Figure 6 to be described later. As a result, when it is determined by the self-position diagnosis that the reliability of the self-position is high, the calculation unit 121 determines that the self-position is correctly estimated, and proceeds to step S607. When it is determined that the reliability of the self-position is low, the calculation unit 121 determines that the self-position is not correctly estimated, and returns to step S605 to continue the self-position estimation.
  • the calculation unit 121 displays, through the display control unit 121G, a result of the self-position estimation at step S605 on the display apparatus 111.
  • the calculation unit 121 causes the display apparatus 111 to display, based on the result of the self-position estimation, for example, a map image indicating the positional relation between an object in surroundings of the vehicle 1 and the vehicle 1, which is expressed in the point group map 124A, so that the user can recognize the estimation result of the self-position in a car park.
  • the calculation unit 121 may generate, by using a captured image acquired from the camera 102A, a panoramic image illustrating the status of the vehicle 1 when viewed from above, and cause the display apparatus 111 to display the panoramic image, thereby indicating the positional relation between an object in surroundings of the vehicle 1 and the vehicle 1, which is expressed in the point group map 124A on the panoramic image.
  • the calculation unit 121 determines whether the self-position displayed at step S607 is within a predetermined distance, for example, 10 m or less from any parking position in car park data in the point group map 124A. Until the self-position is within 10 m from the parking position, the calculation unit 121 returns to step S605 to continue the self-position estimation. When the self-position is within 10 m, the calculation unit 121 proceeds to step S609. Specifically, when the difference between the coordinate value of the parking position in the car-park coordinate system expressed in the point group map 124A and the position of the vehicle 1 estimated at step S605 is shorter than the predetermined distance, the calculation unit 121 performs the positive determination at step S608 and proceeds to the next step S609.
  • a predetermined distance for example, 10 m or less from any parking position in car park data in the point group map 124A.
  • the calculation unit 121 displays, through the display control unit 121G, on the display apparatus 111 that automated parking is possible. This queries the user for whether to perform automated parking of the vehicle 1.
  • step S610 the calculation unit 121 determines whether the automated parking button 110B is pressed by the user. When the automated parking button 110B is not pressed by the user, the calculation unit 121 returns to step S609 to continue the display that automated parking is possible. When the automated parking button 110B is pressed by the user, the calculation unit 121 proceeds to step S611.
  • the calculation unit 121 switches, through the mode switching unit 121I, the operation mode of the on-board processing apparatus 120 from the position estimation mode to the automated parking mode. Then, the calculation unit 121 executes operation in the automated parking mode in accordance with the procedure of a flowchart illustrated in Figure 9 to be described later. Having executed the automated parking mode at step S611, the calculation unit 121 ends the flowchart of Figure 5 .
  • Figure 6 is a flowchart illustrating self-position estimation processing executed by the on-board processing apparatus 120 at step S605 in Figure 5 .
  • the calculation unit 121 performs processing same as that of steps S503 to 505 in Figure 4 , respectively, and acquires the local surrounding information 122B. Specifically, at step S621, the calculation unit 121 performs landmark positioning in a procedure same as that of step S503 to acquire observation values of a plurality of points representing a landmark in a car park existing in surroundings of the vehicle 1, and records the observation values in the RAM 122 as the point group data 122A. At step S622, the calculation unit 121 estimates the movement amount of the vehicle 1 in a procedure similar to that of step S504 and updates the current position of the vehicle 1 in the local coordinate system.
  • the calculation unit 121 transforms, based on the current position of the vehicle 1 updated at step S622, the observation value of each point in the point group data 122A acquired by landmark positioning at step S621 into a coordinate value in the local coordinate system in a procedure similar to that of step S505, and temporarily records the transformed observation value in the RAM 122 as the local surrounding information 122B. Accordingly, the calculation unit 121 newly generates the local surrounding information 122B in accordance with the current position of the vehicle 1 and updates the contents of the local surrounding information 122B in the RAM 122.
  • the calculation unit 121 selects, at step S624, the local surrounding information 122B updated at step S623.
  • the calculation unit 121 selects, in accordance with the procedure of a flowchart illustrated in Figure 7 to be described later, a point to be used later in matching processing from among points, the coordinate value of each of which is recorded as the local surrounding information 122B.
  • the points in the local surrounding information 122B may include a point for which matching cannot be performed because of difference in the shape of distribution from the point group map 124A due to a cumulative error in estimation of the movement amount of the vehicle 1.
  • a range of points that each have a small error and for which matching can be performed are adaptively selected from the local surrounding information 122B.
  • the calculation unit 121 performs matching processing illustrated in detail in Figure 8 at step S625.
  • a relational expression indicating the correspondence relation between the local coordinate system and the car-park coordinate system in other words, a formula of coordinate transform from the local coordinate system to the car-park coordinate system is obtained.
  • the car-park coordinate system is a coordinate system used in the point group map 124A and set by using a place on the past travel path of the vehicle 1 as the reference position (first reference position).
  • the local coordinate system is a coordinate system used in the local surrounding information 122B and set by using a place on the past travel path of the vehicle 1, which is different from that of the car-park coordinate system, as the reference position (second reference position).
  • the calculation unit 121 calculates a relational expression indicating the relation between the first reference position in the point group map 124A (car-park coordinate system) and the second reference position in the local surrounding information 122B (local coordinate system).
  • the calculation unit 121 calculates the coordinates of the vehicle 1 in the car-park coordinate system, in other words, an estimated self-position by using the coordinates of the vehicle 1 in the local coordinate system updated at step S622 and the coordinate transform formula obtained at step S625. Having calculated the estimated self-position in this manner, the calculation unit 121 subsequently proceeds to step S627.
  • the calculation unit 121 executes self-position diagnosis on the estimated self-position calculated at step S626.
  • the reliability of the self-position estimated at step S626 is determined by using, for example, three indexes as follows.
  • the first index is an error in the current position in the local coordinate system.
  • the movement amount of the vehicle 1 estimated by the well-known dead reckoning technology by using outputs from the vehicle speed sensor 108 and the rudder angle sensor 109 is compared with the movement amount of the self-position estimated at step S626 in a predetermined duration. As a result, when the difference therebetween is larger than a threshold value determined in advance, it is determined that the reliability of the self-position is low.
  • the second index is an error amount between corresponding points, which is calculated at matching. Specifically, when the error amount between corresponding points is larger than a threshold value determined in advance, it is determined that the reliability of the self-position is low.
  • the third index is the existence of a similar solution at matching. Specifically, when a similar solution for a solution of the self-position obtained at matching is searched by a method such as translation by the width of a parking frame and the error amount between corresponding points is substantially equal for both solutions, it is determined that the reliability of the self-position is low.
  • step S627 for example, when it is determined based on all of the above-described three indexes that the reliability is high, it is determined that the self-position is correctly estimated. When it is determined based on any one or more of the indexes that the reliability is low, it is determined that the self-position is not correctly estimated.
  • the self-position estimation is completed and the process proceeds from step S605 to the next step S606 in Figure 5 . Not all the three indexes necessarily need to be used.
  • the self-position diagnosis may be executed by using another index.
  • Figure 7 is a flowchart illustrating processing of selecting the local surrounding information 122B executed by the on-board processing apparatus 120 at step S624 in Figure 6 .
  • the calculation unit 121 calculates a locus on which the vehicle 1 has traveled in the position estimation mode.
  • the calculation unit 121 calculates the locus of the vehicle 1 so far since the start of the position estimation mode by using the result of the estimation of the movement amount of the vehicle 1, which is performed at step S622 in Figure 6 .
  • the locus of the vehicle 1 can be calculated by interpolating a plurality of coordinate points indicating the position of the vehicle 1 in the local coordinate system, which is calculated from the movement amount of the vehicle 1 estimated at step S622.
  • the calculation unit 121 calculates the effective range of matching for each point in the local surrounding information 122B updated at step S623 in Figure 6 .
  • the calculation unit 121 calculates, as an effective range in which matching is possible, a range in which a shape error is small as compared to the point group map 124A.
  • the effective range may be determined based on the length and shape of the locus calculated at step S680. Specifically, an estimation error is more likely to occur to the coordinate value of each point obtained with the local surrounding information 122B as the travel distance of the vehicle 1 is longer and the rotation amount of the vehicle 1 is larger. However, it is difficult to perform matching when the number of points at which the coordinate value is obtained in the local surrounding information 122B is too small.
  • points in a range extending backward from the current position of the vehicle 1 by a shortest distance D [m] determined in advance are acquired as points in the effective range.
  • the amount of change in the angle of the tangent line of the locus is accumulated, and points around the locus in a range up to a position where this accumulated value changes to or beyond an angle threshold value 0 [deg] determined in advance are acquired as points in the effective range.
  • the range of X [m] ⁇ Y [m] determined in advance and centered at the locus is set as the effective range of the local surrounding information 122B. Accordingly, the effective range of the local surrounding information 122B has a shape along the obtained locus.
  • step S682 the calculation unit 121 acquires each point in the effective range obtained at step S681 as a point indicating the local surrounding information 122B. Accordingly, the local surrounding information 122B is selected. Having selected the local surrounding information 122B in this manner, the calculation unit 121 proceeds from step S624 to the next step S625 in Figure 6 .
  • Figure 8 is a flowchart illustrating matching processing executed by the on-board processing apparatus 120 at step S625 in Figure 6 .
  • the calculation unit 121 applies the outlier list 122C stored in the RAM 122 to the local surrounding information 122B.
  • any point listed in the outlier list 122C among a plurality of points included in the local surrounding information 122B is temporarily excluded as a processing target.
  • the range of application of the outlier list 122C at step S641 includes steps S642 to S653 described below. Specifically, after the outlier list 122C is updated at step S654 to be described later, any point previously included in the outlier list 122C is included as a processing target. However, steps S641 to S643 cannot be executed at initial execution of the flowchart illustrated in Figure 8 , and thus execution starts at step S650. Having applied the outlier list 122C at step S641, the calculation unit 121 subsequently proceeds to step S642.
  • the calculation unit 121 transforms each point in the latest local surrounding information 122B, in other words, the coordinate value of each point representing a landmark detected by landmark positioning performed at step S621 in Figure 6 into a coordinate value in the car-park coordinate system in car park data for which it is determined at step S603 in Figure 5 that the map formability exists.
  • This transform is achieved by using the current position of the vehicle 1 in the local coordinate system, which is updated at step S622 in Figure 6 and the previously calculated formula of coordinate transform from the local coordinate system to the car-park coordinate system.
  • the calculation unit 121 calculates a momentary matching degree IC of the local surrounding information 122B subjected to the coordinate transform at step S642 and the car park data used for the coordinate transform.
  • step S644 the calculation unit 121 determines whether the momentary matching degree IC calculated at step S643 is higher than the predetermined threshold value. As a result, when having determined that the momentary matching degree IC is higher than the threshold value, the calculation unit 121 proceeds to step S650. When having determined that the momentary matching degree IC is equal to or lower than the threshold value, the calculation unit 121 proceeds to step S645.
  • the calculation unit 121 detects a periodic characteristic, for example, a plurality of arranged parking frames from the local surrounding information 122B. Since a point group included in the point group map 124A is obtained by extracting an image edge or the like as described above, a parking frame line can be detected from points arranged at an interval corresponding to the thickness of a white line.
  • the calculation unit 121 determines whether a periodic characteristic is detected at step S645. When having determined that a periodic characteristic is detected, the calculation unit 121 proceeds to step S647. When having determined that no periodic characteristic is detected, the calculation unit 121 proceeds to step S650.
  • the calculation unit 121 calculates the period of the periodic characteristic detected at step S645, for example, the width of a parking frame.
  • the width of a parking frame is the interval between white lines included in the parking frame. The calculation unit 121 subsequently proceeds to step S647.
  • the calculation unit 121 calculates the momentary matching degree IC of the local surrounding information 122B and the car park data when the coordinate transform formula calculated at the previous step S653 is changed as a reference in a plurality of manners.
  • the coordinate transform formula is changed in a plurality of manners by using the period calculated at step S647 so that each point in the local surrounding information 122B subjected to the coordinate transform is shifted by integral multiples of the periodic characteristic detected at step S645.
  • a whole matching degree IW is calculated for each of the plurality of coordinate transform formulae after the change.
  • the whole matching degree IW is calculated by, for example, Expression (3) below.
  • IW DWin / DWall
  • DWin represents the number of points, the distance of each of which from any point in the car park data in the point group map 124A is equal to or shorter than a predetermined value among points in the local surrounding information 122B subjected to the coordinate transform by using the above-described coordinate transform formula.
  • "DWall” represents the number of points detected at step S621. Having calculated the whole matching degree IW at step S648, the calculation unit 121 subsequently proceeds to step S649.
  • the calculation unit 121 stores, in the RAM 122, a coordinate transform formula that provides the maximum whole matching degree IW among the plurality of whole matching degrees IW calculated at step S648, and proceeds to step S650.
  • An iterative closest point (ICP) algorithm as a known point group matching technology may be used in association processing at step S650, error minimization processing at step S651, and convergence determination processing at step S652 described below.
  • initial value setting at step S650 is unique to the present embodiment and thus will be described in detail, whereas the other processing will be only schematically described.
  • step S650 which is executed when the positive determination is performed at step S644, when the negative determination is performed at step S646, when the execution of step S649 ends, or when the negative determination is performed at step S652, the calculation unit 121 associates each point included in the car park data in the point group map 124A with each point included in the local surrounding information 122B.
  • thee points are associated by using a different coordinate transform formula in accordance with processing executed beforehand. Specifically, when the process is executed following step S644 or S649, coordinate transform from the local surrounding information 122B to the car park data is performed by using a coordinate transform formula recorded in the RAM 122.
  • step S653 the coordinate transform formula calculated at the previously executed step S653 is used.
  • step S650 is executed following step S649, the coordinate transform formula stored at step S649 is used.
  • the calculation unit 121 subsequently proceeds to step S651.
  • the calculation unit 121 minimizes an error between points associated at step S650.
  • the coordinate transform formula is changed so that the sum of a distance index between each pair of points associated at step S650 is minimum.
  • the sum of the absolute value of the distance between each pair of associated points may be employed as the sum of the distance index between the points.
  • the calculation unit 121 determines whether the error has converged. As a result, when having determined that the error has converged, the calculation unit 121 proceeds to step S653. When having determined that the error has not converged, the calculation unit 121 returns to step S650 to continue the association processing and the error minimization processing.
  • the calculation unit 121 stores the coordinate transform formula lastly changed at step S651 in the RAM 122, and proceeds to step S654.
  • the calculation unit 121 updates the outlier list 122C as follows. First, the existing outlier list 122C stored in the RAM 122 is cleared. Subsequently, the coordinate value of each point in the local surrounding information 122B is transformed into a coordinate value in the car-park coordinate system by using the coordinate transform formula recorded at step 653, and the distance, in other words, the Euclidean distance between each point included in the local surrounding information 122B and a point included in the point group map 124A and corresponding to the point is calculated. When the calculated distance is longer than a distance determined in advance, the point in the local surrounding information 122B is added to the outlier list 122C.
  • the addition to the outlier list 122C may be performed by using another condition such that the point is positioned at a spatially end part.
  • the spatially end part is a point, the distance of which to other points is long, such as a point acquired when recording is started.
  • the calculation unit 121 ends the flowchart of Figure 8 and proceeds from step S625 to the next step S626 in Figure 6 .
  • Figure 9 is a flowchart illustrating processing of the automated parking mode executed by the on-board processing apparatus 120 at step S611 in Figure 5 .
  • the executor of each step described below is the calculation unit 121 of the on-board processing apparatus 120.
  • the calculation unit 121 functions as the position estimation unit 121F and the vehicle control unit 121H.
  • the on-board processing apparatus 120 transitions from the position estimation mode to the automated parking mode only while the automated parking button 110B is pressed down by the user when the position estimation of the vehicle 1 is successful in the position estimation mode and the latitude and longitude, in other words, the absolute position of the vehicle 1 is in a predetermined range from a parking position. Accordingly, automated control of the vehicle 1 is performed upon approval by the user.
  • step S661 the calculation unit 121 performs self-position estimation through the position estimation unit 121F to estimate the position of the vehicle 1 in the car-park coordinate system.
  • the contents of processing at the present step are same as those of step S605 in Figure 5 , which are described above with reference to Figure 6 , and thus description thereof is omitted here.
  • the calculation unit 121 generates, by a known path generation method, a travel path from the position estimated at step S661 to a parking position stored in the point group map 124A.
  • the calculation unit 121 subsequently proceeds to step S663.
  • the calculation unit 121 controls, through the vehicle control unit 121H, the travel state of the vehicle 1 via the vehicle control apparatus 130.
  • the on-board processing apparatus 120 controls each of the steering apparatus 131, the drive apparatus 132, and the braking apparatus 133 by outputting a predetermined vehicle control command to the vehicle control apparatus 130 based on the position of the vehicle 1 estimated at step S661, thereby moving the vehicle 1 to the parking position along the path generated at step S662.
  • an operation command is preferably output to the drive apparatus 132 only while the automated parking button 110B is continuously pressed by the user.
  • the braking apparatus 133 is preferably operated to stop the vehicle 1 when a person, a moving vehicle, or the like is extracted from a captured image of the camera 102A.
  • step S664 similarly to step S661, the calculation unit 121 performs self-position estimation through the position estimation unit 121F to estimate the position of the vehicle 1 in the car-park coordinate system.
  • the calculation unit 121 determines whether parking of the vehicle 1 is completed, in other words, whether the vehicle 1 has reached the parking position. As a result, when having determined that parking is not completed, the calculation unit 121 returns to step S663 to continue control of the vehicle 1. When having determined that parking is completed, the calculation unit 121 ends the flowchart of Figure 9 .
  • Figure 10 (a) is a plan view illustrating an exemplary car park 901.
  • the car park 901 includes a plurality of parking areas provided in surroundings of a building 902. Each parking area is surrounded by a parking frame line such as a white line painted on a road surface.
  • a parking frame line such as a white line painted on a road surface.
  • One gateway of the car park 901 is provided at a lower-left part on the drawing.
  • the following describes exemplary operation of the on-board processing apparatus 120 in the map storage mode when the vehicle 1 is to be parked in a parking area 903 hatched in Figure 10 (a) in the car park 901.
  • the on-board processing apparatus 120 detects a parking frame line in the car park 901 as a landmark.
  • the position of the vehicle 1 is represented by a triangle as illustrated in Figure 10 (a) , and an acute angle of the triangle indicates the traveling direction of the vehicle 1.
  • the on-board processing apparatus 120 starts landmark positioning (steps S502 and S503) and records the coordinate values of points included in a parking frame line in the RAM 122 as the point group data 122A (steps S504 and S505). Then, the processing at steps S503 to S506 in Figure 4 is repeated until parking of the vehicle 1 in the parking area 903 is completed.
  • Figure 10 (b) is a diagram visualizing each point of a landmark represented by the point group data 122A stored in the RAM 122.
  • a solid line illustrates a point group of a landmark stored in the RAM 122
  • a dashed line illustrates a landmark not stored in the RAM 122. Since the range in which sensing is possible by the external sensor group 102 of the vehicle 1 is limited, only any parking frame line near the entrance of the car park 901 is recorded in the point group data 122A when the vehicle 1 is near the entrance of the car park 901 as illustrated in Figure 10 (b) .
  • the on-board processing apparatus 120 can record point groups of landmarks in the entire car park 901 to the point group data 122A as the user moves the vehicle 1 to the back of the car park 901 after the recording to the point group data 122A is started.
  • the on-board processing apparatus 120 acquires the latitude and longitude of the vehicle 1 from the car navigation system 107 and records the coordinate values of four corners of the parking area 903 (step S509).
  • the point group data 122A stored in the RAM 122 is recorded in the storage unit 124 as new data included in the point group map 124A, in other words, new car park data (step S515).
  • the car park data illustrated in Figure 11 (a) is exemplary data obtained when the vehicle 1 travels on the right side from the entrance of the car park 901 illustrated in Figure 10 (a) to the parking area 903.
  • data is not acquired for parking frame lines illustrated with dotted lines in Figure 11 (a) nor recorded as car park data in the point group map 124A.
  • the point group data 122A illustrated in Figure 11 (b) is exemplary data obtained when the vehicle 1 travels on the left side from the entrance of the car park 901 to the parking area 903. In this case, since the vehicle 1 has traveled on the left side as compared to the case of Figure 11 (a) , no point group data of parking frame lines illustrated with dotted lines in Figure 11 (b) is acquired. In the point group data 122A illustrated in Figure 11 (b) , since the vehicle 1 does not squarely face the car park 901 when transition is made to the map storage mode, the car park 901 is recorded in an entirely tilted manner as compared to the case of Figure 11 (a) .
  • the on-board processing apparatus 120 determines that car park data, the difference of which from the current latitude and longitude of the vehicle 1 is in the predetermined range is recorded in the point group map 124A (YES at step S510). Then, coordinate transform of the point group data 122A is performed with reference to a parking position in Figures 11 (a) and (b) , in other words, the parking area 903 (step S511).
  • the point group matching rate IB is calculated (step S512), and when it is determined that the point group matching rate IB is higher than a predetermined threshold value (YES at step S513), the point group data 122A illustrated in Figure 11 (b) is integrated with the car park data illustrated in Figure 11 (a) (step S514).
  • the point group map 124A Through the integration, data of parking frame lines on the left side in Figure 11 (a) , which is not recorded in the data of the drawing is newly recorded in the point group map 124A, and the density of points representing already recorded parking frame lines on the right side and the upper side in drawing increases in the point group map 124A.
  • Figure 12 (a) is a diagram illustrating the current position of the vehicle 1 having approached the parking area 903 in the car park 901.
  • the vehicle 1 faces toward the right side in drawing.
  • Figure 12 (b) is a diagram illustrating exemplary data obtained by transforming, from the local coordinate system to the car-park coordinate system, the local surrounding information 122B acquired in the position estimation mode as the vehicle 1 moves from the entrance of the car park 901 to the position illustrated in Figure 12 (a) .
  • parking frame lines and the locus of the vehicle 1 indicated by the local surrounding information 122B are illustrated with dashed lines and dotted lines, respectively.
  • the parking frame lines and the locus illustrated with dashed lines indicate points of the parking frame lines and the locus in the local surrounding information 122B out of the effective range, in other words, points listed in the outlier list 122C and excluded as a matching processing target.
  • the parking frame lines and the locus illustrated with dotted lines indicate points of the parking frame lines and the locus in the local surrounding information 122B in the effective range.
  • car park data corresponding to the entire car park 901 illustrated in Figure 10 (a) is stored in the point group map 124A in advance.
  • Figure 13 (a) is a diagram illustrating the current position of the vehicle 1 traveling toward the parking area 903 in the car park 901. In Figure 13 (a) , the vehicle 1 faces upward in the drawing.
  • Figure 13 (b) is a diagram illustrating exemplary data obtained by transforming, from the local coordinate system to the car-park coordinate system, the coordinate value of each point in the local surrounding information 122B obtained through landmark positioning of parking frame lines in a front-side area of the vehicle 1, in other words, a part surrounded by a dashed line circle in Figure 13 (a) when the vehicle 1 is at a position illustrated in Figure 13 (a) .
  • the data illustrated in Figure 13 (b) is data of point groups detected from a latest captured image and processed at step S642 in Figure 8 in the local surrounding information 122B. In Figure 13 (b) , this data is illustrated as dashed lines, not as points.
  • Figure 13 (b) also illustrates the position of the vehicle 1 for comparison with Figure 13 (a) .
  • data of parking frame lines continuously exists on the left side of the vehicle 1, but on the right side of the vehicle 1, data of parking frame lines exists only on the front side of the vehicle 1.
  • Figure 13 (c) is a diagram illustrating exemplary comparison between the point group map 124A and the local surrounding information 122B illustrated in Figure 13 (b) when a result of estimation of the position of the vehicle 1 in the car-park coordinate system includes an error.
  • the previous position estimation result is shifted by one parking frame width approximately, and thus deviation from the point group map 124A occurs to the local surrounding information 122B existing on the right side of the vehicle 1.
  • the momentary matching degree IC is calculated at step S643 in Figure 8 , the momentary matching degree IC for the right side of the vehicle 1 is lower due to the above-described deviation.
  • parking frames are detected as a periodic characteristic (YES at steps S645 and S646), and the parking frame width is calculated from the point group map 124A (step S647). Then, the whole matching degree IW is calculated by shifting the local surrounding information 122B subjected to coordinate transform by integral multiples of the parking frame width (step S648).
  • Figures 14 (a), (b), and (c) are diagrams illustrating an exemplary relation between the point group map 124A and the local surrounding information 122B when the local surrounding information 122B illustrated in Figure 13 (c) is shifted by integral multiples of the parking frame width.
  • the local surrounding information 122B subjected to coordinate transform is shifted upward in the drawing by +1, 0, and -1 multiples of the parking frame width, respectively.
  • the on-board processing apparatus 120 estimates a coordinate transform formula of the car-park coordinate system and the local coordinate system based on the point group map 124A and the local surrounding information 122B, and estimates the position of the vehicle 1 in the car-park coordinate system.
  • the point group map 124A is information stored in the storage unit 124 in advance, and the local surrounding information 122B is generated based on outputs from the external sensor group 102, in other words, the camera 102A, the sonar 102B, the radar 102C, the vehicle speed sensor 108, and the rudder angle sensor 109.
  • the on-board processing apparatus 120 can acquire information of point groups in a coordinate system different from the coordinate system of recorded point groups and estimate the position of the vehicle 1 in the recorded coordinate system based on the correspondence relation between the different coordinate systems. Since the on-board processing apparatus 120 estimates the coordinate transform formula of the car-park coordinate system and the local coordinate system based on the point group map 124A and the local surrounding information 122B, the coordinate transform formula is unlikely to be affected by noise included in part of data of the local surrounding information 122B. Specifically, the position estimation of the vehicle 1 by the on-board processing apparatus 120 is robust over disturbance.
  • the position estimation unit 121F searches the point group map 124A for a point corresponding to each point included in the local surrounding information 122B (step S650 in Figure 8 ), and estimates the coordinate transform formula of the car-park coordinate system and the local coordinate system so that the distance between the corresponding points becomes minimum (step S651 in Figure 8 ).
  • the position estimation unit 121F performs search and estimation by excluding point data in the local surrounding information 122B of a point that corresponds to the point and the distance of which from a point included in the point group map 124A or the local surrounding information 122B is longer than a threshold value determined in advance, in other words, by applying the outlier list 122C (steps S641 and S653 in Figure 8 ).
  • a threshold value determined in advance
  • Points included in the point group map 124A and the local surrounding information 122B are expressed as coordinates in a two-dimensional space.
  • the position estimation unit 121F performs search and estimation by excluding a point, the distance of which from a corresponding point is longer than a threshold value determined in advance and that is positioned at a spatially end part in the local surrounding information 122B.
  • a point group stored in car park data in the point group map 124A relates to a landmark closer to a parking position than a place where the map storage mode is started.
  • the point group map 124A includes a periodic characteristic. After estimating the coordinate transform formula of the car-park coordinate system and the local coordinate system, the position estimation unit 121F corrects the coordinate transform formula of the car-park coordinate system and the local coordinate system based on the distance corresponding to one period of the periodic characteristic so that the distance between corresponding points is short (steps S646 to S648 in Figure 8 ).
  • the point group map 124A includes a periodic characteristic, matching tends to be achieved with shift by integral multiples of the distance corresponding to the period thereof. Once matching is made with such shift, it is difficult to achieve matching to a correct position due to the property of repetitive processing.
  • This problem is solved by shifting the coordinate transform formula by integral multiples of the period after a solution of the repetitive processing has converged.
  • consideration is made on probability that a local solution shifted from a global solution by several periods of the periodic characteristic is reached through repetitive calculation, and the coordinate transform formula is shifted by an amount in accordance with the periodic characteristic, thereby obtaining the global solution or a local solution closer to the global solution.
  • the position estimation unit 121F corrects the coordinate transform formula when the momentary matching degree IC as an index indicating the degree of matching between the local surrounding information 122B in the car-park coordinate system and the point group map 124A in the car-park coordinate system, which is produced based on the previously estimated position of the vehicle 1 in the car-park coordinate system, information of the latest point group data 122A acquired by the point group data acquisition unit 121A, and the latest position information of the vehicle 1 acquired by the movement amount estimation unit 121E, is lower than a threshold value determined in advance (steps S642 to S644 in Figure 8 ).
  • the processing at steps S645 to S649 in Figure 8 can be executed only when determined to be needed based on the necessity of the processing, instead of being always executed.
  • the on-board processing apparatus 120 includes the point group map generation unit 121C configured to produce, based on the information acquired by the point group data acquisition unit 121A and the position information of the vehicle 1 acquired by the movement amount estimation unit 121E, the point group data 122A including a plurality of coordinate values of points representing part of an object except for a moving object in a third coordinate system (the recording coordinate system) and store the point group data 122A in the storage unit 124 as the point group map 124A.
  • the on-board processing apparatus 120 can produce the point group map 124A when the vehicle 1 on which the on-board processing apparatus 120 is mounted travels.
  • the processing of producing the point group map 124A and the processing of position estimation of the vehicle 1 are common in landmark positioning and can use a common program module.
  • the third coordinate system (recording coordinate system) is set based on the position and posture of the vehicle 1 when production of the point group data 122A is started.
  • the point group map generation unit 121C estimates the relation among the different coordinate systems with reference to the parking position of the vehicle 1 and integrates the plurality of pieces of the point group data 122A (steps S511 and S514 in Figure 4 ).
  • the plurality of pieces of the point group data 122A can be integrated. This is based on a fact that the position and posture of the vehicle 1 when acquisition of the point group data 122A is started are different but the vehicle 1 is parked at the same parking position.
  • the on-board processing apparatus 120 includes the vehicle control apparatus 130 configured to drive the vehicle 1 based on an output from the position estimation unit 121F and move the vehicle 1 to a parking position specified in advance in the car-park coordinate system, and the absolute position acquisition unit 121D configured to acquire the latitude and longitude of the vehicle 1 from the car navigation system 107 configured to receive information (latitude and longitude) related to the position of the vehicle 1.
  • a point included in the point group map 124A is a point indicating part of a constituent object in the car park.
  • the storage unit 124 also stores the latitude and longitude of the car park as the point group map 124A.
  • the on-board processing apparatus 120 includes the vehicle control unit 121H configured to move the vehicle 1 to a parking position by using the vehicle control apparatus 130.
  • the on-board processing apparatus 120 can perform automated parking of the vehicle 1 to a parking position included in the point group map 124A.
  • the on-board processing apparatus 120 includes the car navigation system 107, and the external sensor group 102 configured to sense a point group in surroundings of the vehicle 1, in other words, the sensor camera 102A, the sonar 102B, and the radar 102C, and generates, through the point group map generation unit 121C, the point group map 124A in a data format as illustrated in Figure 2 in which the point group and the latitude-longitude information are associated with each other.
  • the existence of a point group in surroundings and the relation with a road map can be determined by referring to the latitude-longitude information, and transition can be automatically made to the map storage mode and the position estimation mode.
  • the user does not need to perform an operation for mode transition, and transition can be smoothly made to map storage and automated parking during travel of the vehicle 1.

Landscapes

  • Engineering & Computer Science (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Navigation (AREA)
  • Traffic Control Systems (AREA)
  • Control Of Driving Devices And Active Controlling Of Vehicle (AREA)
EP19812343.2A 2018-05-30 2019-03-05 Information processing device Active EP3805702B1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018104102A JP7137359B2 (ja) 2018-05-30 2018-05-30 情報処理装置
PCT/JP2019/008631 WO2019230098A1 (ja) 2018-05-30 2019-03-05 情報処理装置

Publications (3)

Publication Number Publication Date
EP3805702A1 EP3805702A1 (en) 2021-04-14
EP3805702A4 EP3805702A4 (en) 2022-03-16
EP3805702B1 true EP3805702B1 (en) 2024-04-24

Family

ID=68697952

Family Applications (1)

Application Number Title Priority Date Filing Date
EP19812343.2A Active EP3805702B1 (en) 2018-05-30 2019-03-05 Information processing device

Country Status (5)

Country Link
US (1) US11560160B2 (enExample)
EP (1) EP3805702B1 (enExample)
JP (1) JP7137359B2 (enExample)
CN (1) CN112204348B (enExample)
WO (1) WO2019230098A1 (enExample)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019107237A1 (ja) * 2017-11-30 2019-06-06 日本電気株式会社 位置推定装置、位置推定方法とプログラムを格納する記録媒体、並びに、位置推定システム
JP2020017199A (ja) * 2018-07-27 2020-01-30 パイオニア株式会社 情報処理装置
US11372097B2 (en) * 2018-10-09 2022-06-28 Metawave Corporation Method and apparatus for phase unwrapping radar detections using optical flow
JP7479108B2 (ja) * 2020-01-31 2024-05-08 パナソニックオートモーティブシステムズ株式会社 情報処理装置、情報処理方法
WO2022049615A1 (ja) * 2020-09-01 2022-03-10 株式会社ネイン 情報処理システム、情報処理方法およびコンピュータプログラム
JP7712070B2 (ja) * 2020-09-28 2025-07-23 若辰 司 経路案内装置、経路案内システム、及びプログラム
CN112735130B (zh) * 2020-12-25 2022-05-10 阿波罗智联(北京)科技有限公司 交通数据的处理方法、装置、电子设备和介质
JP7610904B2 (ja) * 2021-01-15 2025-01-09 パナソニックオートモーティブシステムズ株式会社 駐車支援方法および駐車支援装置
JP7409330B2 (ja) * 2021-01-28 2024-01-09 トヨタ自動車株式会社 自己位置推定精度検証方法、自己位置推定システム
JP2022142826A (ja) * 2021-03-17 2022-10-03 本田技研工業株式会社 自車位置推定装置
JP7561683B2 (ja) * 2021-05-13 2024-10-04 日立Astemo株式会社 地図生成・自己位置推定装置
JP7072311B1 (ja) * 2021-07-21 2022-05-20 株式会社センシンロボティクス 移動体の移動経路生成方法及びプログラム、管理サーバ、管理システム
JP2023019930A (ja) * 2021-07-30 2023-02-09 キヤノン株式会社 情報処理装置、移動体、情報処理装置の制御方法およびプログラム
JP2024173590A (ja) * 2023-05-31 2024-12-12 トヨタ自動車株式会社 制御装置
CN116486639B (zh) * 2023-06-14 2023-09-29 眉山环天智慧科技有限公司 一种基于遥感和北斗卫星数据分析的车辆监管方法

Family Cites Families (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3595680B2 (ja) * 1998-05-19 2004-12-02 クラリオン株式会社 ナビゲーションシステム及び方法並びにナビゲーション用ソフトウェアを記録した記録媒体
JP4277717B2 (ja) * 2004-03-17 2009-06-10 株式会社日立製作所 車両位置推定装置およびこれを用いた運転支援装置
JP2006072431A (ja) * 2004-08-31 2006-03-16 Matsushita Electric Ind Co Ltd 自動駐車システム
JP4483589B2 (ja) * 2005-01-12 2010-06-16 日産自動車株式会社 車両用情報提供装置
JP5328252B2 (ja) 2008-07-30 2013-10-30 アルパイン株式会社 ナビゲーションシステムの位置検出装置および位置検出方法
JP5634046B2 (ja) * 2009-09-25 2014-12-03 クラリオン株式会社 センサコントローラ、ナビゲーション装置、センサ制御方法
JP2012141662A (ja) 2010-12-28 2012-07-26 Toyota Motor Corp ロボットの自己位置推定方法
GB201116959D0 (en) * 2011-09-30 2011-11-16 Bae Systems Plc Vehicle localisation with 2d laser scanner and 3d prior scans
JP5926558B2 (ja) * 2011-12-29 2016-05-25 クラリオン株式会社 車両用情報システムおよびサーバ装置
GB2501466A (en) * 2012-04-02 2013-10-30 Univ Oxford Localising transportable apparatus
DE102013015348A1 (de) * 2013-09-17 2014-04-10 Daimler Ag Verfahren und Vorrichtung zum Betrieb eines Fahrzeugs
JP2015077862A (ja) 2013-10-16 2015-04-23 トヨタ自動車株式会社 駐車支援装置
JP6314932B2 (ja) 2014-09-26 2018-04-25 キヤノンマーケティングジャパン株式会社 錠剤供給システム、情報処理装置、錠剤供給システムの制御方法、情報処理装置の制御方法及びプログラム
DE102014224077A1 (de) * 2014-11-26 2016-06-02 Robert Bosch Gmbh Verfahren und Vorrichtung zum assistierten Führen eines Fahrzeugs
DE102015201209A1 (de) * 2015-01-26 2016-07-28 Robert Bosch Gmbh Valet Parking-Verfahren und Valet-Parking System
CA2987373C (en) * 2015-05-28 2018-12-04 Nissan Motor Co., Ltd. Position estimation device and position estimation method
JP6589410B2 (ja) * 2015-06-24 2019-10-16 株式会社豊田中央研究所 地図生成装置及びプログラム
JP6649191B2 (ja) 2016-06-29 2020-02-19 クラリオン株式会社 車載処理装置
CA3032068C (en) 2016-07-26 2020-01-14 Nissan Motor Co., Ltd. Self-position estimation method and self-position estimation device
JP6958117B2 (ja) * 2017-08-29 2021-11-02 株式会社アイシン 駐車支援装置
US10719719B2 (en) * 2018-03-02 2020-07-21 Here Global B.V. Method, apparatus, and computer program product for determining if probe data points have been map-matched

Also Published As

Publication number Publication date
WO2019230098A1 (ja) 2019-12-05
CN112204348A (zh) 2021-01-08
US11560160B2 (en) 2023-01-24
JP7137359B2 (ja) 2022-09-14
EP3805702A4 (en) 2022-03-16
CN112204348B (zh) 2024-08-27
EP3805702A1 (en) 2021-04-14
US20210206391A1 (en) 2021-07-08
JP2019207214A (ja) 2019-12-05

Similar Documents

Publication Publication Date Title
EP3805702B1 (en) Information processing device
CN110945578B (zh) 车载处理装置
CN109313031B (zh) 车载处理装置
JP7393128B2 (ja) 車載処理装置、移動支援システム
JP6757261B2 (ja) 車載処理装置
JP7132037B2 (ja) 車載処理装置
CN115803781A (zh) 用于生成与对象相关联的鸟瞰图边界框的方法和系统
EP3795952A1 (en) Estimation device, estimation method, and computer program product
JP7227763B2 (ja) 車載処理装置、位置推定方法、出庫システム
JP2023039626A (ja) 車載処理装置、車両制御装置、及び自己位置推定方法

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20201224

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20220215

RIC1 Information provided on ipc code assigned before grant

Ipc: G06V 20/58 20220101ALI20220209BHEP

Ipc: B60W 30/06 20060101ALI20220209BHEP

Ipc: G01C 21/00 20060101ALI20220209BHEP

Ipc: G01C 21/36 20060101ALI20220209BHEP

Ipc: G01C 21/30 20060101AFI20220209BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20230102

RIC1 Information provided on ipc code assigned before grant

Ipc: G01C 21/36 20060101ALI20230821BHEP

Ipc: G01C 21/00 20060101ALI20230821BHEP

Ipc: B60W 30/06 20060101ALI20230821BHEP

Ipc: G06V 20/56 20220101ALI20230821BHEP

Ipc: G06V 20/58 20220101ALI20230821BHEP

Ipc: G01C 21/30 20060101AFI20230821BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20231120

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602019051003

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20240424

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1680035

Country of ref document: AT

Kind code of ref document: T

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240824

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240725

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240826

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240826

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240724

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240824

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240725

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240724

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602019051003

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20250127

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20250218

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20250218

Year of fee payment: 7

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20250221

Year of fee payment: 7

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20240424

REG Reference to a national code

Ref country code: CH

Ref legal event code: H13

Free format text: ST27 STATUS EVENT CODE: U-0-0-H10-H13 (AS PROVIDED BY THE NATIONAL OFFICE)

Effective date: 20251023

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20250305

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20250331