WO2017057053A1

WO2017057053A1 - Information processing device, information processing method

Info

Publication number: WO2017057053A1
Application number: PCT/JP2016/077426
Authority: WO
Inventors: 貝野　彰彦; 辰吾鶴見; 江島　公志; 嵩明加藤; 福地　正樹; 卓青木; 琢人元山; 章中村
Original assignee: ソニー株式会社
Priority date: 2015-09-30
Filing date: 2016-09-16
Publication date: 2017-04-06

Abstract

The present technique relates to an information processing device and an information processing method which makes it easier for a driver to park. This information processing device is provided with a projection control unit which controls multiple projection devices that project a prescribed pattern, and a path creation unit which creates a path when a mobile body is moving to a target position. The projection control unit controls that projection by a projection device positioned on the path created by the path creation unit. The information processing device is provided with an imaging unit which images the prescribed pattern projected by the projection device, an estimation unit which estimates the local position using the pattern imaged by the imaging unit, and a control signal generation unit which, on the basis of a path supplied from another device and the estimated local position, generates a control signal for controlling each part to enable moving along the path. This technique can be adapted to systems for managing a parking lot.

Description

Information processing apparatus and information processing method

This technology relates to an information processing apparatus and an information processing method. Specifically, the present invention relates to an information processing apparatus and an information processing method that provide support when guiding a moving object to a destination.

Conventionally, a parking support device that supports a parking operation by calculating a parking route of a vehicle has been proposed. For example, the parking support system described in Patent Document 1 below sets an index that serves as a mark in an environment that is to be parked, and performs parking support by guiding the vehicle based on the distance and direction from the index.

In the parking assistance system described in Patent Document 1, there is a possibility that the position of the vehicle being guided may be displaced due to environmental conditions such as the presence of pebbles, the occurrence of crosswinds, and aging of the road surface shape. Some indicators may be missed. In that case, since it becomes impossible to detect the position of the vehicle accurately, there is a possibility that parking assistance cannot be performed appropriately.

Therefore, in Patent Document 2, the surrounding state of the host vehicle is detected to detect the marker set in the map information, and the current position of the host vehicle is determined based on the relative positional relationship between the detected marker and the host vehicle. It has been proposed to estimate.

JP 2008-087676 A Japanese Patent Laying-Open No. 2015-083430

In Patent Document 2, a fixed marker is set in advance, and processing is performed by detecting the marker. However, there is a possibility that a marker installed in advance is hidden by a parked vehicle and cannot be detected. Also in Patent Document 2, there is a possibility that a shift due to environmental conditions may occur, and it becomes impossible to appropriately perform parking support because the position of the vehicle cannot be accurately detected.

The present technology has been made in view of such a situation, and is capable of accurately detecting the position of a vehicle and appropriately guiding to a predetermined place.

A first information processing apparatus according to an aspect of the present technology includes a projection control unit that controls a plurality of projection apparatuses that project a predetermined pattern, and a path creation unit that creates a path when the moving body moves to a destination. The projection control unit controls the projection of the projection device located on the path created by the path creation unit.

The projection control unit sets a pattern projected by the projection apparatus installed at a predetermined position as a marker pattern, and sets a pattern projected by the projection apparatus installed at a position other than the predetermined position as a random pattern. Can be.

The predetermined position may be at least one of a corner or the vicinity of the destination.

The route creation unit creates a route to the destination to which the moving body can move using at least one information of the width, length, height, and minimum turning radius of the moving body. be able to.

The destination may be a parking space, and the route creation unit may create the route up to a position where the moving body starts an approach for parking in the parking space.

A holding unit that holds a global map indicating the position of the feature point in the predetermined area in the three-dimensional space, and a local map that indicates the position of the feature point in the image captured from the moving body in the three-dimensional space. A global map update unit that updates the global map based on the local map may be further provided.

The route creation unit can create a route with reference to the global map held in the holding unit.

The global map indicating the path created by the path creating unit, the marker information of the marker pattern, and the position of the feature point in a predetermined area can be supplied to the moving body.

A first information processing method according to an aspect of the present technology includes a step of controlling a plurality of projection devices that project a predetermined pattern, and creating a path when the moving body moves to a destination, and controls the projection Is performed by controlling the projection of the projection device located on the created path.

A second information processing apparatus according to an aspect of the present technology includes an imaging unit that captures a predetermined pattern projected by a projection device, and an estimation unit that estimates a self-position using the pattern captured by the imaging unit And a path supplied from another device, and a control signal generation unit that generates a control signal for controlling each unit to move on the path based on the estimated self-position.

It is possible to further include a local map generation unit that generates a local map indicating the position in the three-dimensional space of the feature points in the image captured by the imaging unit.

A holding unit that holds a global map indicating a position in a three-dimensional space of feature points in a predetermined area supplied from the other device; and whether or not a position corresponding to the local map has changed in the global map And a determination unit that determines whether the local map is supplied to the other device when the determination unit determines that there has been a change.

The route can be corrected with the local map.

When the marker is detected from the predetermined pattern, the self-position estimated by the estimation unit can be corrected.

The control signal may be a signal that notifies the user of the route.

A second information processing method according to an aspect of the present technology captures a predetermined pattern projected by a projection device, estimates a self-position using the captured pattern, and supplies a route supplied from another device And generating a control signal for controlling each part to move on the route based on the estimated self-position.

In the first information processing apparatus and information processing method according to one aspect of the present technology, a plurality of projection apparatuses that project a predetermined pattern are controlled, and a route when the moving body moves to the destination is created. Further, the projection is controlled by controlling the projection of the projection device located on the created path.

The second information processing apparatus and the information processing apparatus according to an aspect of the present technology capture an image of a predetermined pattern projected by the projection device, and use the pattern captured by the imaging unit to perform self-position estimation. Based on a route supplied from another device and the estimated self-position, a control signal for controlling each part to move on the route is generated.

According to one aspect of the present technology, it is possible to accurately detect the position of the vehicle and appropriately guide to a predetermined place.

It should be noted that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a figure for demonstrating the structure of a parking lot. It is a figure for demonstrating a path | route. It is a figure for demonstrating the structure of a parking assistance system. It is a figure for demonstrating ID allocated to the projection apparatus. It is a figure for demonstrating the projection range of a projection apparatus. It is a flowchart for demonstrating operation | movement of a parking assistance system. It is a figure which shows the structure of a parking assistance apparatus. It is a flowchart for demonstrating operation | movement of a parking assistance apparatus. It is a figure which shows the structure of a parking assistance vehicle side apparatus. It is a figure which shows the structure of a self-position estimation process part. It is a figure for demonstrating the position of the imaging device of a vehicle side. It is a flowchart for demonstrating operation | movement of the parking assistance vehicle side apparatus. It is a figure for demonstrating a recording medium.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. Parking assistance 2. Configuration of parking support system Operation of parking support system 4. 4. Configuration of parking assist device Operation of parking assist device 6. 6. Configuration of parking assist vehicle side device 7. Operation of parking assist vehicle side device About recording media

<About parking assistance>
The present technology can be applied to a system that guides a vehicle to an empty space and assists in parking in a parking lot. For example, there is a parking lot 10 as shown in FIG. 1, and as shown in FIG. 2, the present invention is applied to a system in which a vehicle 20 entering the parking lot 10 is guided to an empty space by a route indicated by an arrow. it can.

As shown in FIG. 1, an entrance gate 11 is installed in the parking lot 10, and a parking space is indicated by a white line (black line in the figure). Here, the description will be continued assuming that the entrance gate 11 is installed and the vehicle 20 is temporarily stopped at the entrance gate 11, but there is a parking lot where the entrance gate 11 is not installed and a place where the vehicle 20 is temporarily stopped. The technology described below can be applied to a parking lot that is not provided.

When there is a vehicle 20 that has entered, a parking space that is considered appropriate is identified for the vehicle 20 in consideration of the environment at that time, and a route to the parking space is generated. Here, the description will be continued assuming that the route indicated by the arrow in FIG. 2 has been generated. The vehicle 20 travels based on the generated route, moves to the parking space, and parks.

A description will be given of a parking support system that generates a route such as this and supports the movement to the parking space and the process of arriving at the parking space and stopping the vehicle 20 in the parking space.

Here, the description will be continued by taking the vehicle 20 as an example, but the present technology can be applied not to the vehicle 20 but to a moving body (moving object).

<Configuration of parking support system>
FIG. 3 is a diagram illustrating a configuration of an embodiment of the parking assistance system. The parking assistance system shown in FIG. 3 includes a parking assistance device 51, an imaging device 52, a projection device 53, a network 61, a database 71, and a parking assistance vehicle side device 81.

The parking support device 51 is a device that manages the parking lot 10 shown in FIG. Although details will be described later, the parking assistance device 51 creates a route, provides the created route to the vehicle 20, and holds and updates a global map (details will be described later).

The imaging device 52 is installed near the entrance gate 11 (FIG. 1) and images the vehicle 20 that has entered the parking lot 10. The parking assist device 51 acquires information about the vehicle such as the vehicle width, the vehicle length, the vehicle height, and the minimum turning radius of the vehicle 20 imaged by the imaging device 52 (hereinafter referred to as vehicle information as appropriate). When acquiring, the database 71 connected to the network 61 can be referred to.

The network 61 is a network composed of a WAN (Wide Area Network), a LAN (Local Area Network), or the like, and the parking support device 51 and the database 71 exchange information (data) with each other via the network 61. It can be configured.

The database 71 is a database that manages information related to the vehicle 20 as described above. Here, an example in which the database 71 is connected to the network 61 is shown, but a configuration provided in the parking assistance device 51 may be adopted.

In addition, here, the description is continued on the assumption that information on the vehicle 20 is acquired with reference to the database 71, but information on the vehicle 20 may be acquired by other methods. For example, information such as the vehicle width and the vehicle height may be acquired by analyzing an image captured by the imaging device 52.

Projection device 53 is also connected to parking support device 51. The projection device 53 projects a predetermined pattern onto an object in the parking lot 10 such as a floor or a parked vehicle based on an instruction from the parking support device 51. A plurality of projection devices 53 are installed in the parking lot 10. Further, when the parking lot 10 is an indoor parking lot, the projection device 53 is installed on the ceiling and projects a predetermined pattern on the floor or the vehicle.

The vehicle 20 includes an imaging device (camera) that images the pattern projected by the projection device 53. The parking assistance vehicle side device 81 analyzes the pattern imaged by the imaging device provided in the vehicle 20, and estimates the self position and creates a local map (details will be described later).

In the parking support system having such a configuration, a plurality of projection devices 53 are installed in the parking lot 10 as described above. Here, an ID is assigned to each of the projection devices 53 installed in the parking lot 10, and each projection device 53 can be identified by the ID. Here, as shown in FIG. 4, the description will be continued assuming that 21 projection devices 53 are installed in the parking lot 10 and are assigned IDs.

21 projectors 53 are assigned ID1 to ID21, respectively. Hereinafter, for example, the projection device 53 assigned ID1 is described as the projection device 53 ID1, and the projection device 53 assigned ID2 is described as the projection device 53 ID2. The other projection devices 53 are also described in a similar manner to distinguish the individual projection devices 53 in the parking lot 10.

For example, the ID1 projection device 53 is installed in the vicinity of the entrance gate 11, and the ID2 projection device 53 is installed on the right side of the ID1 projection device 53. Note that an IP address or the like can also be used as information for identifying each projection device 53.

As shown in FIG. 4, for the sake of explanation, a number is assigned to the parking space for one vehicle. In the example shown in FIG. 4, numbers 1 to 48 are assigned. That is, 48 parking spaces are provided in the parking lot 10. In the following description, for example, it is described as a parking space 1, and the parking space 1 is a parking space located near the entrance gate 11 in the parking lot 10 of FIG. 4. Other parking spaces are listed as well.

Here, refer to FIG. 2 and FIG. Assume that the route indicated by the arrow in FIG. 2 is determined as the route to the parking space of the vehicle 20. In this case, a parking space 37 is first set as a parking space, and a route like an arrow is set as a route from the entrance gate 11 to the parking space 37.

On the path of the arrow, there are an ID1 projection device 53, an ID2 projection device 53, an ID3 projection device 53, an ID7 projection device 53, an ID11 projection device 53, and an ID12 projection device 53 in the vicinity of the parking space 37. Is installed.

In such a case, the parking assistance device 51 gives instructions for projecting predetermined patterns to the projection devices 53 of ID1, ID2, ID3, ID7, ID11, and ID12. As shown in FIG. 5, the projection device 53 is installed on the ceiling 91 and projects a predetermined pattern onto the floor 92. In FIG. 5, the projection device 53 of ID1 and the projection device 53 of ID2 are shown. The projection device 53 may be installed so that the projection ranges of the adjacent ID1 projection device 53 and ID2 projection device 53 overlap, or the projection device 53 does not overlap. May be installed.

The projected pattern may be visible light, or may be other than visible light, such as infrared rays.

The pattern projected from the projection device 53 is a random pattern, but the pattern projected from the projection device 53 installed at a specific position is a marker pattern. The projection device 53 that projects the marker pattern can be a projection device 53 installed at a corner.

As will be described later, the projected pattern is picked up by the image pickup device of the vehicle 20, and the parking assist vehicle-side device 81 detects feature points from the picked-up image and performs analysis such as matching, thereby Estimate location or create local map. By making the pattern projected from the projection device 53 a random pattern, the parking assist vehicle-side device 81 can easily estimate the self position.

If the pattern projected from the projection device 53 is a pattern having a predetermined shape such as a lattice shape, and the same lattice shape pattern is projected from the ID1 projection device 53 and the ID2 projection device 53, parking is performed. It is difficult to determine whether the pattern captured by the assisting vehicle side device 81 is a pattern projected by the ID1 projection device 53 or a pattern projected by the ID2 projection device 53. An error may occur in the creation.

By making the pattern projected from the projection device 53 into a random pattern, the pattern from each projection device 53 can be made different, and the parking assist vehicle-side device 81 as described above can estimate its own position, etc. It is possible to suppress the occurrence of an error.

Further, a random pattern and a marker pattern are provided on the pattern projected from the projection device 53. The marker pattern is used for correction of self-position estimation. For example, the marker pattern is a pattern projected by the projection device 53 installed at a corner.

For example, referring again to FIG. 4, the projection devices 53 of ID1, ID3, ID5, ID9, ID11, ID13, ID17, ID19, and ID21 are installed at the corners, and these projection devices 53 display the marker pattern. Project. Projectors 53 other than these projectors 53 project a random pattern.

When the parking assist vehicle-side device 81 images the marker pattern, the parking assist vehicle-side device 81 corrects the estimated self-position from the information obtained from the marker pattern. Further, the projection device 53 installed near (in the vicinity of) the parking space to be guided may project a marker pattern. For example, when the parking space 37 is a guided parking space, the ID12 projection device 53 may project a marker pattern instead of a random pattern. When configured in this way, the projection device 53 of ID12 is configured to project by switching between a random pattern and a marker pattern.

As described above, the projection device 53 can be configured to project only one of the random pattern and the marker pattern, or can be configured to switch and project the random pattern or the marker pattern as necessary. You can also.

As shown in FIG. 5, the projection device 53 is installed on the ceiling 91 and is installed so as to project a predetermined pattern onto the floor 92. By projecting from the ceiling 91 to the floor 92, the projected pattern is prevented from being blocked on the road surface on which the vehicle 20 travels without being affected by the shielding object. Is possible.

Also, in the parking lot 10, there are places where vehicles are parked in the parking space and places where parking is not done. In addition, there are cases where the parked vehicle is properly parked within the parking space. That is, the environment in the parking lot 10 is constantly changing. The projection from the projection device 53 is also performed on a parked vehicle, a baggage placed, and the like (for example, a shopping cart, an object loaded and unloaded on the vehicle, an object such as a person). Therefore, the parking assist vehicle-side device 81 can capture and analyze a pattern projected on an object such as a parked vehicle.

The parking assist vehicle-side device 81 can create a local map that also considers objects in the parking lot 10 and estimate its own position, and estimates the route to be taken in consideration of the objects in the parking lot 10. can do. For example, it is possible to estimate a route that travels avoiding an object that is not always an object in the parking lot 10, such as a vehicle that protrudes to a traveling area, not a parking space, or a luggage or a person.

If a predetermined pattern is drawn on a floor or a pillar instead of projecting a pattern with the projection device 53, and the self-position is estimated by imaging the pattern, the predetermined pattern on the floor or the pillar is There is a possibility that an image such as a parked vehicle, object, or person is blocked and cannot be imaged. Also, if the pattern is drawn, it may disappear due to aging. For this reason, when the self-position is estimated using the drawn pattern, there is a high possibility that it cannot be estimated correctly.

However, by projecting the pattern with the projection device 53, it is possible to prevent the pattern from disappearing due to deterioration over time. Further, by projecting the pattern from the ceiling with the projection device 53, in other words, projecting from above, it is only possible to prevent the pattern from being blocked by an object such as a parked vehicle, object, or person. In addition, it is possible to detect objects such as vehicles, objects, and people, and it is possible to estimate the self-position and the route with higher accuracy.

By the way, in order to estimate the self-position, generally, a method using GPS (Global Positioning System) communication is used. However, in GPS communication, communication becomes unstable indoors, and it is difficult to accurately estimate the position. In addition, although self-position estimation is performed using an inertial measurement device (IMU: Inertial Measurement Unit) such as a gyro sensor or an acceleration sensor, it is difficult to estimate the position with high accuracy.

However, according to the present technology, since the self-position is estimated using the pattern projected by the projection device 53, the position can be accurately estimated even indoors, unlike the case of using GPS communication. It becomes. Moreover, since it becomes possible to perform the process as described later in the parking assistance vehicle side device 81, for example, the process of correcting the position estimated by the marker pattern, the position can be estimated with high accuracy. Become.

Although details will be described later, the parking assist vehicle-side device 81 captures the projected pattern by the image capturing device of the vehicle 20, detects feature points from the captured pattern, and performs matching analysis, etc. Estimate self-position. Suppose that a self-position is estimated by imaging a structure in the parking lot 10, not a pattern projected from the projection device 53, detecting feature points from the image of the structure, and performing analysis such as matching. It can also be done.

Although the parking lot 10 has a structure such as a floor or a pillar as a structure, it is a structure with poor texture, and it is difficult to estimate the self-position by acquiring and analyzing an image of such a structure. It is difficult to estimate accurately and maintain stability.

However, according to the present technology, by projecting a pattern with the projection device 53, even a structure with poor texture can be made into a structure with texture, thereby improving the accuracy of self-position estimation. Thus, it is possible to perform estimation while maintaining stability.

<Operation of parking support system>
The operation of the parking support system shown in FIG. 3 will be described with reference to the flowchart of FIG. Detailed operations of the parking assist device 51 and the parking assist vehicle side device 81 will be described later, and here, an outline of processing in the system will be described.

In step S11, the parking assistance device 51 acquires the vehicle information of the vehicle 20 that has entered. In step S12, the parking assistance device 51 generates a route. The route is also generated using the vehicle information of the vehicle 20 acquired in step S11. For example, in consideration of the vehicle width of the vehicle, a parking space that can be parked is searched, and a route that can be passed through the vehicle width of the vehicle to the searched parking space is searched.

In step S13, control of projection of a predetermined pattern in the projection device 53 is started. The projection device 53 on the generated path is instructed to start projection.

In step S14, the generated route is provided to the parking assist vehicle-side device 81. Information provided to the parking assist vehicle-side device 81 is information on a generated route (hereinafter referred to as route information), a global map, and the like.

In step S <b> 21, the parking assistance vehicle side device 81 acquires route information provided from the parking assistance device 51. And the parking assistance vehicle side apparatus 81 controls each part of the vehicle 20, or starts estimation of a self-position so that it can drive | work based on the provided route information in step S22.

In step S23, the parking assistance vehicle side device 81 creates a local map. The local map is a map related to the environment around the parking assist vehicle-side device 81 (vehicle 20). For example, it is information relating to the environment in which the parking assist vehicle-side device 81 (vehicle 20) is traveling, such as what position and at what warehousing angle the parked vehicle is stopped. The local map is a map that is used when the self-position is estimated or when the route is corrected.

In step S24, the parking assistance vehicle side device 81 provides the created local map to the parking assistance device 51 as necessary.

In step S15, the parking assistance device 51 receives the local map provided from the parking assistance vehicle side device 81. And in step S16, the parking assistance apparatus 51 updates the global map which self manages, as needed.

<Configuration of parking assistance device>
The structure of the parking assistance apparatus 51 is demonstrated. FIG. 7 is a diagram illustrating an internal configuration of the parking assist device 51. The parking assist device 51 includes an entrance vehicle detection unit 101, a vehicle information acquisition unit 102, a projection control unit 103, a pattern holding unit 104, a route creation unit 105, a global map holding unit 106, a global map update unit 107, and a communication unit 108. It is configured to include.

The entrance vehicle detection unit 101 detects the vehicle 20 that has entered the parking lot 10. Here, the description is continued on the assumption that the image captured by the imaging device 52 is supplied to the entrance vehicle detection unit 101 and the vehicle 20 entering the parking lot 10 is detected based on the supplied image.

The vehicle information acquisition unit 102 acquires information on the vehicle 20 that has entered the parking lot 10, such as information on the vehicle width, vehicle height, minimum turning radius, and the like (hereinafter referred to as vehicle information).

Projection control unit 103 controls projection device 53. The projection control unit 103 controls the start of projection by the projection device 53 based on a control signal from the entrance vehicle detection unit 101, and projects by the projection device 53 based on a control signal based on information such as a route from the route creation unit 105. To control.

The pattern holding unit 104 holds, for example, the ID of the projection device 53 and the pattern (image) to be projected in association with each other. The projection control unit 103 reads the pattern associated with the ID of the projection device 53 located on the path created by the route creation unit 105 from the pattern holding unit 104 and supplies the pattern to the projection device 53 located on the path. To do.

The route creation unit 105 uses the vehicle information acquired by the vehicle information acquisition unit 102 to determine a parking space suitable for the vehicle 20 that has entered the vehicle to park and a route from the entrance gate 11 to the parking space. The determination is made with reference to the global map held in the global map holding unit 106.

The global map holding unit 106 holds a global map. The global map update unit 107 updates the global map stored in the global map storage unit 106 with reference to the local map supplied from the parking assistance vehicle side device 81 via the communication unit 108 as necessary.

The communication unit 108 communicates with the parking assistance vehicle side device 81. The communication unit 108 performs communication when the route information created by the route creation unit 105, the global map held by the global map holding unit 106, and the like are provided to the parking assistance vehicle side device 81. In addition, the communication unit 108 performs communication when supplying a local map or the like supplied from the parking assistance vehicle side device 81 to the global map update unit 107 or the route creation unit 105.

Here, the global map held in the global map holding unit 106 will be described.

The global map is a map showing the position in a three-dimensional space of a stationary object within a predetermined wide area. For example, the global map includes information indicating the position and feature amount of a feature point of a stationary object in a predetermined region on a three-dimensional spatial coordinate system. The spatial coordinate system is represented by, for example, latitude, longitude, and height from the ground.

The global map has a base global map (referred to as the base global map), and the global map is updated as appropriate according to the environment. Here, the global map is a map of the parking lot 10, and the base global map is a planar map as shown in FIG. That is, the basic global map is a map of the parking lot 10 when the vehicle is empty.

The base global map can be, for example, a map created using a design drawing at the time of designing the parking lot 10.

The base global map is updated by the global map update unit 107. The update is performed when a change occurs in the environment (situation) in the parking lot 10 such as an update reflecting a newly parked vehicle or an update reflecting a vehicle leaving the parking lot 10.

When the update is performed, the base global map is updated to a map as shown in FIG. 2, for example. The map shown in FIG. 2 is a state in which a plurality of vehicles are parked in the parking lot 10. The map shown in FIG. 2 is a map in which a vehicle is parked in the parking space 1, the parking space 3, the parking space 13, the parking space 25, the parking space 30, and the parking space 33, referring to FIG. is there.

The parking assist vehicle side device 81 creates a local map while traveling along the designated route in the parking lot 10. The local map is a map indicating the position of a stationary object around the vehicle 20 in the three-dimensional space, and is generated by the parking assistance vehicle side device 81. For example, the local map includes information indicating the position and the feature amount on the three-dimensional spatial coordinate system of the feature point of the stationary object around each moving object, as in the global map.

The local map is a map that reflects the surrounding environment where the vehicle is traveling. For example, when traveling around the parking space 1 and the parking space 3, a map is created that the vehicle is parked in the parking space 1 and the parking space 3.

In addition, when traveling around the parking space 13, a map is created in which vehicles parked in the parking space 13 are parked outside the parking space. By creating such a local map, when the parking assist vehicle side device 81 passes the left side of the vehicle parked in the parking space 13, the parking assist vehicle side device 81 does not protrude and collide with the parked vehicle. A route that passes through the parking space 6 is created, and each part in the vehicle 20 is controlled so that traveling based on the created route is performed.

Thus, the local map created by the parking assist vehicle-side device 81 is a map reflecting the environment (situation) in the parking lot 10. Such a local map is supplied from the parking assistance vehicle side device 81 to the global map updating unit 107 of the parking assistance device 51 at a predetermined timing. The global map update unit 107 refers to the supplied local map and updates a portion where a change has occurred in the global map.

In other words, when there is a newly parked vehicle, the global map update unit 107 determines whether the vehicle is parked in the parking space where the vehicle is parked (what position with respect to the parking space). In addition, an update reflecting the angle at which the vehicle is parked is performed.

In addition, when the parked vehicle leaves the parking lot 10, in other words, when the vacant parking space is available, the global map updating unit 107 vacates the vacant parking space. Update to return to the previous state.

As described above, since the local map is created by the parking assist vehicle side device 81 of the vehicle 20 that is actually traveling in the parking lot 10, for example, a vehicle parked in the parking space 13 Information indicating that the vehicle parked outside the parking space 13 or parked in the parking space 3 is parked in the parking space 3 but parked diagonally can be used. .

グローバル By updating the global map based on such a local map, the global map can be made a near real-time map according to the environment in the parking lot 10. Therefore, with reference to such a global map, it is possible to appropriately set a parking space in which the entering vehicle 20 should park and a route to the parking space.

<Operation of parking assist device>
Next, operation | movement of the parking assistance apparatus 51 is demonstrated with reference to the flowchart shown in FIG.

In step S101, the entrance vehicle detection unit 101 determines whether or not the vehicle 20 that has entered the parking lot 10 has been detected. For example, when the vehicle 20 is captured in the image captured by the imaging device 52 (FIG. 3) installed near the entrance gate 11 in the parking lot 10, the entrance vehicle detection unit 101 Detect that you have entered.

Alternatively, by detecting a vehicle 20 entering the entrance gate 11 with a sensor (not shown) installed near the entrance gate 11 in the parking lot 10, and obtaining information from such a sensor, The entrance vehicle detection unit 101 may detect that the vehicle 20 has entered the parking lot 10.

Alternatively, the imaging device 52 (FIG. 3) installed near the entrance gate 11 in the parking lot 10 is configured to take an image of the vehicle 20 when the vehicle 20 enters, and the entrance vehicle detection unit 101. May detect that the vehicle 20 has entered the parking lot 10 when an image is provided from the imaging device 52.

In step S101, the process in step S101 is repeated and the standby state is maintained until it is determined that the vehicle 20 that has entered the parking lot 10 is detected. On the other hand, if it is determined in step S101 that the vehicle 20 entering the parking lot 10 has been detected, the process proceeds to step S102.

In step S102, the vehicle information acquisition unit 102 acquires vehicle information. The vehicle information acquisition part 102 acquires the image imaged with the imaging device 52, and acquires the vehicle information of the vehicle 20 reflected in the image. The vehicle information includes vehicle width, vehicle height, minimum turning radius, and the like, and is information used for searching for a parking space to be guided and determining a route to the parking space.

The vehicle information is acquired with reference to the database 71 as described with reference to FIG.

Alternatively, information may be provided from the vehicle 20 side. As the information to be provided, vehicle information itself may be provided, or information that can identify a vehicle (vehicle type) may be provided. When the information is such that the vehicle can be specified, the database 71 can be referred to using the provided information to obtain the vehicle information.

In addition, the database 71 may be provided as a part of the parking assistance device 51. In such a case, the vehicle information acquisition unit 102 acquires vehicle information with reference to the provided database 71. .

Note that vehicle information may be provided from the vehicle 20 side via the communication unit 108. For example, when the driver operates a predetermined button or when a request for provision of vehicle information is issued from the parking assist device 51 and responds to the request, the vehicle information is transmitted from the vehicle 20 side, and the transmission is performed. The vehicle information acquired may be acquired by the vehicle information acquisition unit 102.

Further, vehicle information such as the vehicle width and the vehicle length may be acquired by analyzing the image captured by the imaging device 52.

In step S103, the route creation unit 105 creates a route. The route creation unit 105 determines a parking space where the entering vehicle 20 is parked using information such as the vehicle width, the vehicle length, the vehicle height, and the minimum turning radius of the vehicle 20. For example, in the case of the vehicle 20 having a large vehicle width, a parking space in which a width capable of parking the vehicle 20 having such a vehicle width can be secured is searched with reference to the global map.

Since the status of the parked vehicle is written in the global map, for example, the parking space 2 (FIG. 4) is vacant, but because the vehicle is in the parking space 1 and the parking space 3, the vehicle width is large. It can be determined that the vehicle 20 cannot park.

In addition, since vehicles parked in the parking space 30 (FIG. 4) are parked out of the parking space, vehicles with a large minimum turning radius or vehicles with a long vehicle length are parked in the parking space 30. Since there is a possibility that it is difficult to park in the parking space in the vicinity of the parking space 30 because of the existing vehicle, it can be determined that it is better to search for a parking space at a position away from the parking space 30.

In addition, because the minimum turning radius is large, the parking space is vacant, but the parking space where the space required for turning back, etc. cannot be secured before the parking space is not searched, and turning back is sufficient. A search such as searching for a parking space in a place where a space required for the vehicle can be secured is also performed.

Also, when a vehicle 20 with a high vehicle height enters, a search such as searching for a parking space from the rooftop or outdoor parking lot is also performed.

Such a search can be performed by referring to such a global map because it is written in which parking space the vehicle is parked and in what state the vehicle is parked. Such a global map is updated by the global map update unit 107 so that the latest state is maintained as much as possible.

The route creation unit 105 refers to the global map and determines a parking space where the entering vehicle 20 parks in consideration of vehicle information.

In addition, the determination method of the parking space mentioned above is an example, and does not show limitation. For example, the parking space may be determined in consideration of the following conditions in addition to the following structural conditions of the vehicle 20.

For example, an image in which a predetermined mark such as a mark indicating that an elderly person is driving, a mark indicating that a disabled person is driving, or a mark indicating that a beginner is driving is captured by the imaging device 52 May be searched for a parking space suitable for the driver. For example, in the parking lot 10, there is a parking lot provided with a parking space where a disabled person can preferentially park. In such a parking lot 10, a mark indicating that the disabled person is driving. May be determined as the parking space of the vehicle 20 that has entered the parking space where the disabled person can preferentially park.

In this way, a parking space suitable for the vehicle 20 entering the vehicle is determined. When the parking space is determined, a route from the entrance gate 11 to the parking space is searched.

The route is also set by referring to the global map. For example, in the parking space 13 and the parking space 30 (FIG. 4), there are vehicles parked outside the parking space, so that the road width that can be traveled is narrow. In such a case, for example, when it is determined that it is difficult for a vehicle with a wide vehicle width to travel, when the entering vehicle 20 is a vehicle with a wide vehicle width, the parking space 13 or the parking space 30 The route to reach the determined parking space is searched and set.

Further, when the vehicle 20 arrives at the set parking space, the vehicle 20 starts moving to park in the parking space, and the parking start position is also set. The parking start position is, for example, a position when parking is started, and is a distance or inclination with respect to the parking space to be parked. If an approach to the parking space is started from the parking start position, the vehicle is placed in the center of the parking space. It is a position where parking is possible with the center of 20 coming.

Thus, the route creation unit 105 searches and sets a parking space suitable for the vehicle 20 that has entered, a route to the parking space, and a parking start position. Here, the set parking space, route, and parking start position are described as route information. The route information may include information other than these pieces of information. Further, the route information may be information such that only the route information is included, and it is sufficient that at least one information of the parking space, the route, and the parking start position is included.

In step S104, information such as route information and markers is provided to the vehicle 20 via the communication unit. The route information is information generated by the route creation unit 105. The marker information is information related to the marker of the marker pattern, and is information in which information on the marker itself is associated with a position on the global map. In addition, a global map held by the global map holding unit 106 is also provided to the vehicle 20.

The vehicle 20 images the marker pattern, determines whether or not it matches the supplied marker, and identifies the projection device 53 that projects the marker pattern by confirming that it matches, in other words, If so, the position on the global map can be specified. By identifying the position in the global map, comparing the identified position with the estimated self position, you can know the amount of displacement of the self position, and correct surrounding objects and the self position of the vehicle etc. It can be performed.

The information about the marker and the global map are supplied to the vehicle 20 so that such processing can be performed on the vehicle 20 side.

When the route as shown in FIG. 2 is set, information about the marker provided by the ID3 projection device 53 and the marker pattern projected by the ID11 projection device 53 are provided as the marker-related information. Information.

Further, the pattern projected from the projection device 53 located in the vicinity of the determined parking space (the parking space 37 in FIGS. 2 and 4) is also a marker pattern, and the parking assist vehicle indicates that the parking space is the destination parking space. The side device 81 may be able to recognize.

Provision of route information and the like from the parking assistance device 51 to the parking assistance vehicle side device 81 is performed by the communication unit 108. For example, an entrance gate 11 is provided, and a mechanism for temporarily stopping the vehicle 20 at the entrance gate 11 is provided. When the vehicle 20 is stopped at the entrance gate 11, route information and the like are provided from the communication unit 108 to the vehicle 20.

In addition to the entrance gate 11, an information providing place may be provided, and route information may be provided at the information providing place. The information providing place may be a predetermined place in the parking lot 10, for example, an area between the entrance gate 11 and the position where the ID1 projection device 53 is installed.

The information provision location may be different for each vehicle 20 that has entered. For example, it is assumed that a plurality of vehicles 20 may enter the parking lot 10 continuously. Assume a case where a plurality of vehicles 20, for example, three vehicles 20 (referred to as vehicles 20-1, 20-2, and 20-3) enter.

In such a case, the information providing locations of the vehicle 20-1, the vehicle 20-2, and the vehicle 20-3 may be changed. For example, for the vehicle 20-1, when the vicinity of the ID3 projection device 53 is set as the information providing place, and the vehicle 20-1 travels (temporarily stops) near the ID3 projection device 53, the route information, etc. To be provided to 20-1.

For the vehicle 20-2, the vicinity of the ID2 projection device 53 is used as the information providing place, and when the vehicle 20-2 travels (temporarily stops) near the ID2 projection device 53, the route information and the like are displayed. -To be provided in 2. Further, for the vehicle 20-3, when the vicinity of the ID1 projection device 53 is used as the information providing place, and the vehicle 20-3 travels (temporarily stops) near the ID1 projection device 53, the route information and the like are obtained. -To be provided in 3.

If information is provided to the vehicle 20-1, the vehicle 20-2, and the vehicle 20-3 at the same information provision location, the same route information may be included in the vehicle 20-1 and the vehicle 20- 2. There is a possibility of being provided to the vehicle 20-3. That is, there is a possibility that information is erroneously provided.

As described above, by setting different information providing locations for each vehicle 20, even if a plurality of vehicles 20 have entered continuously, each vehicle 20 can have different information providing locations. On the other hand, provision of information to each vehicle 20 can be performed. Therefore, it is possible to reduce the possibility that the same route information is erroneously provided to the plurality of vehicles 20.

Also, an ID for identifying the vehicle 20 that has entered may be assigned to each vehicle 20. A combination of the above-described information providing location with a mechanism that varies for each vehicle 20 and a mechanism that allows the vehicle 20 to determine whether or not the information is information about itself based on the ID may be provided. For example, the parking support device 51 allocates an ID to the vehicle 20 that has entered the entrance gate 11 or a place corresponding to the entrance gate 11, and the assigned ID is assigned to the vehicle 20 (the parking support vehicle side device 81). To provide.

The parking support device 51 controls the communication unit 108 so as to provide the information at the information providing place by associating the allocated ID with the provided information when providing the route information or the like to the vehicle 20. When the parking support vehicle-side device 81 of the vehicle 20 acquires information from the parking support device 51, it determines whether or not the ID managed by the device and the ID associated with the acquired information match. If it matches, it is stored as information addressed to itself, and if it does not match, the information is not addressed to itself and is discarded.

Thus, it becomes possible to provide information to the vehicle 20 more appropriately by using the information providing location and the ID.

It is also possible to provide information using only ID. The information providing place may be set to one place, and information may be provided at that place, and the vehicle 20 side may determine whether the information is addressed to itself based on the ID.

In step S105, the projection control unit 103 starts controlling the projection of the projection device 53.

The projection control unit 103 controls the projection device 53 installed in the parking lot 10. Here, the projection control unit 103 controls each projection by the projection devices 53 (FIG. 4) ID1 to ID21. When receiving a projection start instruction control signal from the entrance vehicle detection unit 101, the projection control unit 103 starts projection control of the projection device 53.

That is, in this case, the projection by the projection device 53 is not always performed (the projection is always performed during the time when the parking lot 10 is used), and the vehicle 20 has entered the parking lot 10. Sometimes, the projection is started, and the projection is terminated when a predetermined condition is satisfied such that the vehicle 20 has passed, the vehicle has stopped in the parking space, or a predetermined time has elapsed since the projection was started.

As described above, the projection is not always performed, but is performed as necessary, so that the power consumed by the projection device 53 can be reduced, and the power consumed by the parking assistance system can be reduced. It becomes possible. Note that in a situation where it is not necessary to reduce power consumption, the projection by the projection device 53 can be configured so that it is always performed. Technology can be applied.

Projection is controlled by the projection control unit 103 in the projection device 53 located on the path. For example, when the route as shown in FIG. 2 is set, the projection of the projection device 53 of ID1, ID2, ID3, ID7, ID11, and ID12 is controlled.

Specifically, the projection devices 53 of ID1, ID2, ID3, ID7, ID11, and ID12 start projecting a predetermined pattern, and the other projection devices 53 are in a state in which a standby state in which projection is not performed is maintained. Is done. Of the projection devices 53 of ID1, ID2, ID3, ID7, ID11, and ID12, the projection devices 53 of ID1, ID3, ID11, and ID12 are controlled to project a marker pattern, and the projection devices of ID2, ID7 53 is controlled to project a random pattern.

The projection control unit 103 identifies the projection device 53 that starts projection from the route information supplied from the route creation unit 105, and selects a pattern (random pattern or marker pattern) associated with the ID of the identified projection device 53. Read from the pattern holding unit 104. The pattern holding unit 104 holds the ID, pattern, and marker information of the projection device 53 in association with each other.

The projection control unit 103 reads the pattern held by the pattern holding unit 104 and supplies the pattern to the projection device 53 identified by the corresponding ID. The projection device 53 projects a predetermined pattern under the control of the projection control unit 103.

It should be noted that the projection control may be started at other timings, such as when the vehicle 20 that has entered the vehicle is detected, not after the route information is generated. For example, when the entrance vehicle detection unit 101 detects an entering vehicle 20, the entrance vehicle detection unit 101 instructs the projection control unit 103 to start projection. The projection control unit 103 starts projection of the projection device 53 installed near the entrance gate 11.

After that, when a route is generated by the route creation unit 105, projection of the projection device 53 located on the route may be started.

In step S106, the global map update unit 107 determines whether a local map has been acquired. The local map is supplied from the parking assistance vehicle side device 81 at a predetermined timing. The predetermined timing may be any timing. For example, when the parking assist vehicle-side device 81 side determines that there is a difference between the held global map and the local map, a marker pattern is projected. For example, when the vehicle travels under the projector 53, when the vehicle arrives at the parking space, or when parking is completed.

In step S106, the process of step S106 is repeated until it is determined that the local map has been acquired. On the other hand, if it is determined in step S106 that a local map has been acquired, the process proceeds to step S107.

The global map update unit 107 updates the global map held in the global map holding unit 106 using the acquired local map. As described above, the local map is a map created by the parking assist vehicle side device 81 reflecting the environment around the vehicle 20 when the vehicle 20 actually travels in the parking lot 10.

The global map update unit 107 refers to such a local map, determines whether or not there is a change in the area of the global map corresponding to the acquired local map. Update the global map to reflect.

For example, as a change, if the global map is a parking space without a parked vehicle, but the local map is a parking space with a parked vehicle, it is determined that there is a change that the vehicle is parked in an empty parking space. As a result, the global map is updated to reflect the changes.

In this way, the global map is updated in response to environmental (situation) changes in the parking lot 10.

In this way, when the global map is updated, the updated global map may be provided to the vehicle 20 traveling in the parking lot 10 at the time of the update.

The support for parking the vehicle 20 is performed by repeatedly performing the processing described with reference to the flowchart of FIG.

<Configuration of parking assist vehicle side device>
Next, the parking assistance vehicle side device 81 will be described. First, the configuration of the parking assistance vehicle side device 81 will be described with reference to the block diagram of FIG.

The parking assist vehicle side device 81 includes a self-position estimation processing unit 201, an information reception instruction unit 202, a communication unit 203, a global map update determination unit 204, an information holding unit 205, a local route generation unit 206, a vehicle control signal generation unit 207, And a notification generation unit 208.

The self-position estimation processing unit 201 has a configuration as described later with reference to FIG. 10, and performs self-position estimation and creation of a local map.

The information reception instruction unit 202 issues an instruction to receive information supplied from the parking support device 51 to the communication unit 203. Information received by the communication unit 203 includes the above-described route information, global map, information on markers, and the like.

The information reception instruction unit 202 issues a reception instruction to the communication unit 203 when reception of information supplied from the parking support device 51 is instructed by an input from a user (driver), for example. Further, as described above, when the entrance gate 11 or the information providing place is designated, the information reception instruction unit 202 determines whether or not it has arrived at such a position, and at such a position. When it is determined that it has arrived, a reception instruction is issued to the communication unit 203.

When the information reception instruction unit 202 gives an instruction, the position where the vehicle 20 is located can be used as the start position of the self-position estimation so that the subsequent self-position estimation is performed. The first marker that determines a specific position (part of the global map) can be received by a start signal that is issued when the user performs an operation, and the position where the first marker is detected Can be configured so that the subsequent self-position estimation is performed as the self-position estimation start position.

The communication unit 203 communicates with the parking assistance device 51. For example, the communication unit 203 receives a global map, route information, and the like supplied from the parking assistance device 51 and supplies the local map to the parking assistance device 51.

The global map update determination unit 204 determines whether or not to update the global map held in the information holding unit 205, and updates the global map as necessary. The global map update determination unit 204 determines that there is a difference between the local map generated by the self-position estimation processing unit 201 and the position corresponding to the local map of the global map stored in the information storage unit 205. If it is determined that the global map is to be updated.

If the global map update determination unit 204 determines to update the global map, the global map update determination unit 204 updates a part of the global map stored in the information storage unit 205 that is different from the local map so as to match the local map.

When the global map is updated, the global map update determination unit 204 transmits the local map to the parking assistance device 51 via the communication unit 203.

When the global map update determination unit 204 determines that the global map needs to be updated, the local map is transmitted to the parking support device 51 without updating the global map held in the information holding unit 205. Only the processing to be performed may be executed. That is, when the global map held on the parking support device 51 side needs to be updated in order to match the environment (situation) in the parking lot 10, the parking support device 51 is notified that the update is necessary. The global map update determination unit 204 can be configured to supply a local map indicating a location to be updated.

The information holding unit 205 holds the global map, route information, and marker information received by the communication unit 203.

The local route generation unit 206 modifies the route specified by the route information held in the information holding unit 205 with reference to the local map. For example, it is assumed that the route information designates passing the parking space 13 (FIGS. 2 and 4) and the vehicle is designated to travel on the center of the road. Then, as shown in FIG. 2, it is assumed that a local map is created in which a vehicle parked in the parking space 13 protrudes from the parking space 13 and protrudes to the traveling area.

In such a case, when the local route generation unit 206 travels beside the parking space 13, the local route generation unit 206 does not travel in the center of the road, but travels around a vehicle parked in the parking space 13, for example, parking. Change to the route from the space 6 side.

Also, when the local route generation unit 206 arrives at the parking start position, it also generates a route from the parking start position to parking in the parking space. When parking in a parking space, depending on the environment (situation) around the parking space, especially vehicles parked on both sides and vehicles parked in front of the vehicle, There is a high possibility that the route on which 20 should travel is different. Therefore, the local route generation unit 206 generates a route that takes into account the environment around the parking space.

The vehicle control signal generation unit 207 controls a control system that controls the speed and the traveling direction of each unit in the vehicle 20, such as an engine, a brake, and a handle, based on the local route generated by the local route generation unit 206. Generate a signal to control. Each unit in the vehicle 20 is controlled based on the control signal generated by the vehicle control signal generation unit 207.

Note that the vehicle 20 is not controlled based only on the control signal generated by the vehicle control signal generation unit 207, but may be a control signal for assisting the driving of the driver.

The notification generation unit 208 performs control for notifying the user (driver) of the local route generated by the local route generation unit 206. For example, a local route is displayed on the display of the car navigation system, or a message is displayed when approaching a corner. In addition to the display, notification may be made by voice. It is also possible to configure such that a warning is given when the route is deviated, or a notification for returning to the correct route is given.

FIG. 10 is a diagram illustrating an internal configuration of the self-position estimation processing unit 201.

The self-position estimation processing unit 201 can partially apply a technique called SLAM (Simultaneous Localization and Mapping). The self-position estimation processing unit 201 is configured to include an estimation unit 301, a position information generation unit 302, an object detection unit 303, and a local map generation unit 304.

The estimation unit 301 calculates the movement amount, position, posture, and speed of the vehicle based on the relative positions of the feature points in the left image and the right image captured by the

imaging devices

401L and 401R (FIG. 11) and the moving object. presume. As shown in FIG. 11, the imaging device 401L and the imaging device 401R are installed in front of the vehicle 20, the imaging device 401L images the left front of the vehicle 20, and the imaging device 401R It is installed in the position which images each.

The estimation unit 301 includes image correction units 321L and 311R, a feature point detection unit 322, a parallax matching unit 323, a distance estimation unit 324, a feature amount calculation unit 325, a map information storage unit 326, a motion matching unit 327, and a movement amount estimation unit 328. , An object dictionary storage unit 329, an object recognition unit 330, a position / orientation information storage unit 331, a position / orientation estimation unit 332, and a speed estimation unit 333.

The image correction unit 321L and the image correction unit 321R correct the left image supplied from the imaging device 401L and the right image supplied from the imaging device 401R, respectively, so that the images are directed in the same direction. The image correction unit 321L supplies the corrected left image to the feature point detection unit 322 and the motion matching unit 327. The image correction unit 321R supplies the corrected right image to the parallax matching unit 323.

The feature point detection unit 322 detects a feature point of the left image. The feature point detection unit 322 supplies two-dimensional position information indicating the position of each detected feature point on the two-dimensional image coordinate system to the parallax matching unit 323 and the feature amount calculation unit 325. The image coordinate system is represented by, for example, an x coordinate and ay coordinate in the image.

The parallax matching unit 323 detects the feature point of the right image corresponding to the feature point detected in the left image. Thereby, the parallax which is the difference between the position on the left image of each feature point and the position on the right image is obtained. The parallax matching unit 323 supplies two-dimensional position information indicating the positions of the feature points on the image coordinate system in the left image and the right image to the distance estimation unit 324.

The distance estimation unit 324 estimates the distance to each feature point based on the parallax between the left image and the right image of each feature point, and further calculates the position of each feature point on the three-dimensional spatial coordinate system To do. The distance estimation unit 324 supplies three-dimensional position information indicating the position of each feature point on the spatial coordinate system to the feature amount calculation unit 325.

The feature amount calculation unit 325 calculates the feature amount of each feature point of the left image. The feature amount calculation unit 325 causes the map information storage unit 326 to store feature point information including the three-dimensional position information of each feature point and the feature amount.

The map information storage unit 326 stores a global map supplied from the parking assist device 51 in addition to the feature point information used for the local map.

The motion matching unit 327 acquires, from the map information storage unit 326, the three-dimensional position information of each feature point detected in the previous frame. Next, the motion matching unit 327 detects a feature point corresponding to each feature point detected in the previous frame in the left image of the current frame. Then, the motion matching unit 327 supplies the movement amount estimation unit 328 with the three-dimensional position information in the previous frame of each feature point and the two-dimensional position information indicating the position on the image coordinate system in the current frame. .

Note that, here, the description is continued assuming that the previous frame is compared with the current frame, but the frame before N frames (a plurality of frames) and the current frame may be compared. In the following description, the previous frame is also described, but the frame may be N frames before, and the present technology is not limited to the comparison with the previous frame.

Based on the three-dimensional position information of the previous frame of each feature point and the two-dimensional position information of the current frame, the movement amount estimation unit 328 determines the vehicle between frames (more precisely, the imaging device 401L). Estimate the amount of movement of the position and orientation. The movement amount estimation unit 328 supplies movement amount information indicating the estimated movement amount of the position and posture of the host vehicle to the object detection unit 303, the position / orientation estimation unit 332, and the speed estimation unit 333.

The object recognition unit 330 recognizes an object in the left image based on the object dictionary stored in the object dictionary storage unit 329. Based on the recognition result of the object, the object recognition unit 330 sets initial values (hereinafter, referred to as an initial position and an initial posture) of the position and posture of the own vehicle (more precisely, the imaging device 401L) in the spatial coordinate system. To do. The object recognition unit 330 causes the position / orientation information storage unit 331 to store initial position / orientation information indicating the set initial position and initial posture.

The position / orientation estimation unit 332 is based on the initial position / orientation information stored in the position / orientation information storage unit 221, the position / orientation information of the previous frame, and the estimation result of the movement amount of the own vehicle. Estimate position and orientation. Further, the position / orientation estimation unit 332 corrects the estimated position and orientation of the host vehicle based on the global map stored in the map information storage unit 326 as necessary.

The position / orientation estimation unit 332 supplies position / orientation information indicating the estimated position and orientation of the host vehicle to the dangerous area determination unit 112, the risk prediction unit 113, the position information generation unit 302, and the speed estimation unit 333, The information is stored in the position / orientation information storage unit 221.

The speed estimation unit 333 estimates the speed of the host vehicle by dividing the estimated movement amount of the host vehicle by the elapsed time. The speed estimation unit 333 supplies speed information indicating the estimated speed to the danger prediction unit 113 and the position information generation unit 302.

The location information generation unit 302 generates location information including the location and speed of the vehicle when notified from the danger region determination unit 112 that the vehicle is in the danger region. The position information generation unit 302 supplies the generated position information to the transmission unit 104.

Based on the movement amount information and the feature point information of the previous frame and the current frame stored in the map information storage unit 326, the object detection unit 303 detects the stationary object and the moving body around the host vehicle. To detect. The object detection unit 303 notifies the local map generation unit 304 of detection results of stationary objects and moving objects around the host vehicle.

The local map generation unit 304 generates a local map based on the detection results of stationary objects and moving objects around the host vehicle and the feature point information of the current frame stored in the map information storage unit 326. The generated local map is transmitted from the communication unit 203 (FIG. 9) to the parking assist device 51 as necessary.

<Operation of parking assist vehicle side device>
Next, operation | movement of the parking assistance vehicle side apparatus 81 is demonstrated with reference to the flowchart of FIG.

In step S201, the

imaging devices

401R and 401L provided in the vehicle 20 start imaging. An image picked up by the image pickup device 401 is a random pattern or a marker pattern projected by the projection device 53.

When imaging is started by the imaging apparatus 401, the self-position estimation processing unit 201 starts self-position estimation in step S202. The self-position estimation processing unit 201 generates a local map and estimates a position where the self is in the parking lot 10. Specifically, it is estimated as follows.

The feature point detection unit 322 detects a feature point of the left image. For the feature point detection method, for example, an arbitrary method such as a Harris corner can be used. The feature point detection unit 322 supplies two-dimensional position information indicating the position of each detected feature point on the image coordinate system to the parallax matching unit 323.

The parallax matching unit 323 detects the feature point of the right image corresponding to the feature point detected in the left image. The parallax matching unit 323 supplies two-dimensional position information indicating the positions of the feature points on the image coordinate system in the left image and the right image to the distance estimation unit 324.

The feature amount calculation unit 325 calculates the feature amount of each feature point of the left image. As the feature amount, for example, an arbitrary feature amount such as SURF (Speeded Up Robust 例えば Features) can be used. The feature amount calculation unit 325 causes the map information storage unit 326 to store feature point information including the three-dimensional position information of each feature point and the feature amount.

The movement amount estimation unit 328 estimates the movement amount of the host vehicle (more precisely, the imaging device 401L) between the previous frame and the current frame. For example, the movement amount estimation unit 328 calculates the movement amount dX that minimizes the value of the cost function f in the following equation (1).

f = Σ || Z _t -proj (dX, M _t-1 ) || ² (1)

The moving amount dX indicates the moving amount of the position and posture of the own vehicle (more precisely, the imaging device 401L) from the previous frame to the current frame. For example, the movement amount dX indicates the movement amount of the position in the three-axis direction (three degrees of freedom) and the posture around each axis (three degrees of freedom) in the spatial coordinate system.

M _t−1 and Z _t indicate the positions of the frame immediately before the corresponding feature point and the current frame. More specifically, M _t-1 indicates the position of the feature point on the spatial coordinate system of the previous frame, and Z _t indicates the position of the feature point on the image coordinate system of the current frame. Yes.

Further, proj (dX, M _t-1 ) is the image coordinates of the left image of the current frame, using the movement amount dX as the position M _t-1 of the feature point in the previous frame on the spatial coordinate system. The projected position on the system is shown. That is, proj (dX, M _t-1 ) estimates the position of the feature point on the left image of the current frame based on the position M _t-1 of the feature point in the previous frame and the movement amount dX. Is.

The movement amount estimation unit 328 obtains a movement amount dX that minimizes the sum of squares of Z _t -proj (dX, M _t-1 ) of each feature point shown in Expression (1) by, for example, the least square method. That is, the movement amount estimation unit 328 calculates the feature point of the left image of the current frame on the image coordinate system based on the position M _t-1 of the feature point on the spatial coordinate system and the movement amount dX of the previous frame. The amount of movement dX that minimizes the error when the position of is estimated is obtained. The movement amount estimation unit 328 supplies movement amount information indicating the obtained movement amount dX to the object detection unit 303, the position / orientation estimation unit 332, and the speed estimation unit 333.

The position / orientation estimation unit 332 acquires position / orientation information in the previous frame from the position / orientation information storage unit 331. Then, the position / orientation estimation unit 332 adds the movement amount dX estimated by the movement amount estimation unit 328 to the position and orientation of the own vehicle in the previous frame, thereby obtaining the current position and orientation of the own vehicle. presume.

The position / orientation estimation unit 332 acquires initial position / orientation information from the position / orientation information storage unit 331 when estimating the position and orientation of the vehicle in the first frame. Then, the position / orientation estimation unit 332 estimates the position and orientation of the host vehicle by adding the movement amount dX estimated by the movement amount estimation unit 328 to the initial position and initial posture of the host vehicle.

Further, the position / orientation estimation unit 332 corrects the estimated position and orientation of the own vehicle based on the global map stored in the map information storage unit 326 as necessary.

The position / orientation estimation unit 332 supplies position / orientation information indicating the estimated position and orientation of the host vehicle to the position information generation unit 302 and the speed estimation unit 333 and causes the position / orientation information storage unit 331 to store the position / orientation information.

Speed estimation unit 333 estimates the speed of the vehicle. Specifically, the speed estimation unit 333 estimates the speed of the host vehicle by dividing the movement amount dX estimated by the movement amount estimation unit 328 by the elapsed time. The speed estimation unit 333 supplies speed information indicating the estimated speed to the position information generation unit 302.

The object detection unit 303 detects a surrounding object, for example, a parked vehicle, a structure such as a pillar, or the like. Specifically, the object detection unit 303 acquires the feature point information of the previous frame and the current frame from the map information storage unit 326. Next, the object detection unit 303 performs matching between the feature point of the previous frame and the feature point of the current frame, and detects the movement of each feature point between frames.

Next, the object detection unit 303 performs a feature point that moves corresponding to the movement of the own vehicle based on the movement amount dX estimated by the movement amount estimation unit 328 and a movement that corresponds to the movement of the own vehicle. Distinguish from no feature points. Next, the object detection unit 303 detects a stationary object around the own vehicle based on the feature point that moves corresponding to the movement of the own vehicle. Further, the object detection unit 303 detects a moving body around the own vehicle based on feature points that do not move corresponding to the movement of the own vehicle. Then, the object detection unit 303 notifies the local map generation unit 204 of the detection results of stationary objects and moving objects around the own vehicle.

The local map generation unit 304 acquires feature point information of the current frame from the map information storage unit 326. Next, the local map generation unit 304 deletes information on the feature points of the surrounding moving objects detected by the object detection unit 303 from the acquired feature point information. Then, the local map generation unit 304 generates a local map based on the remaining feature point information.

In this way, self-location is estimated and a local map is generated.

In step S203, the communication unit 203 communicates with the parking support device 51, receives route information, a global map, and information on the marker, supplies the information to the information holding unit 205, and holds the information. As described above, this communication is performed at the entrance gate 11, the information providing place, etc., and information is received.

In step S204, the local route generation unit 206 generates a local route. The local route can be such that the route specified in the route information is a finely tuned route in the local map. For example, as described above, in a place where the vehicle is parked out of the parking space on the designated route, the route is finely adjusted to avoid the vehicle. Further, there is a possibility that the route is deviated, and in such a case, the route is corrected.

Also, when arriving at the parking start position, a route from the parking start position to parking in the parking space is generated. When parking, more detailed driving is required than when traveling, such as turning back, and more detailed routes are generated as routes, and routes that take into account the environment (situation) around the parking space are generated. To do.

The guidance to the parking space may be performed by using the marker pattern of the marker pattern being imaged as the pattern projected from the projection device 53 near the parking start position.

The marker of the marker pattern being imaged changes even if it is the same marker as the vehicle 20 moves. From the change, it is determined whether or not the vehicle is traveling accurately on the designated route. When it is determined that the vehicle is not traveling, the control for correcting the vehicle is repeatedly performed to guide the parking space. May be performed.

In step S205, the vehicle control signal generation unit 207 generates a control signal for controlling the speed and direction of the vehicle 20 based on the local route generated by the local route generation unit 26.

In step S206, the notification generation unit 208 notifies the user (driver) of the route. For example, the route is displayed to the user by using the navigation system to display the route on the display of the navigation system or by notifying a corner or the like by voice.

Note that only one of the generation of the control signal for controlling the vehicle 20 and the notification of the route to the user may be performed. For example, only the generation of a control signal for controlling the vehicle 20 may be performed, and the vehicle 20 may be controlled based on the local route to reduce the burden on the driver. Further, for example, only the route notification to the user may be performed, and the user (driver) may use the notification to drive or park the designated parking space.

In step S207, the global map update determination unit 204 determines whether or not the global map stored in the information storage unit 205 needs to be updated. The global map update determination unit 204 determines that there is a difference between the local map generated by the self-position estimation processing unit 201 and the position corresponding to the local map of the global map stored in the information storage unit 205. If it is determined that the global map is to be updated.

In step S207, when the global map update determination unit 204 determines to update the global map, the process proceeds to step S208, where a difference occurs from the local map of the global map held in the information holding unit 205. Information (global map update information) is generated and provided to the parking assistance device 51.

In addition, a process in which a part of the global map stored in the information storage unit 205 that is different from the local map is updated to match the local map may be performed.

As the global map update information, the local map itself may be supplied to the parking assistance device 51 via the communication unit 203. As described above, when the parking assistance device 51 receives the supply of the local map, the parking assistance device 51 updates the global map held by itself.

On the other hand, if it is determined in step S207 that the global map is not updated, or if the global map is updated in step S208 (when the local map is provided to the parking assist device 51 side), step S209. The process proceeds.

In step S209, it is determined whether or not a marker has been detected. The self-position estimation processing unit 201 determines whether or not the marker held in the information holding unit 205 has been detected from the image captured by the imaging device 401. If it is determined in step S209 that a marker has been detected, the process proceeds to step S210.

In step S210, self-position correction is performed. The projection device 53 installed at a predetermined position on the global map projects a marker pattern, and the marker is detected when the marker pattern projected by the projection device 53 is imaged. When the marker pattern projected at such a specific position is imaged and a marker is detected, the self-position estimation processing unit 201 corrects the position estimated by itself.

On the other hand, if it is determined in step S209 that no marker is detected, the process in step S210 is skipped, and the process proceeds to step S211. In step S211, it is determined whether or not the destination has been reached.

In step S211, it is determined whether or not the destination has been reached. The parking start position may be set as the destination, or until the parking space is parked. Further, the projection device 53 installed at the position set as the destination projects the marker pattern, and it is determined whether or not the marker according to the marker pattern is detected. This determination can be made.

In step S211, processing is returned to step S204 until it is determined that the destination has been reached, and the subsequent processing is repeated. That is, the execution of the process of generating a local route and moving to the destination based on the local route is maintained.

On the other hand, if it is determined in step S211 that the destination has been reached, the processing of the parking assistance vehicle side device 81 is terminated.

Thus, since the parking assistance vehicle side device 81 travels on the route set by the parking assistance device 51 by creating a local map or a local route, assistance when the driver parks in the parking lot 10. It is possible to reduce the burden on the driver.

In addition, although the case where assistance is performed from entering the parking lot 10 until parking in the parking space is described here as an example, assistance until leaving the parking space can be performed in the same way, The present technology can be applied even when leaving. That is, a route from the parking space to the exit gate is generated by the parking assist device 51, and the vehicle 20 travels based on the route, thereby assisting until the vehicle exits.

In the above-described embodiment, the case where assistance is performed when parking in the parking lot 10 has been described as an example. However, the present technology can be applied to the parking lot 10 both indoors and outdoors. In the case of indoors, the projector 53 is installed on the ceiling, and in the case of outdoors, a pole or the like is installed, and the projector 53 is installed on the pole.

Moreover, in the above-described embodiment, the case where the support at the time of parking at the parking lot 10 is described as an example, but it is not limited to the support in the parking lot, but the support at other places Even if it exists, this technique is applicable. For example, the present technology can be applied to support for a vehicle traveling on a highway.

In the above-described embodiment, the case where the vehicle 20 is supported has been described as an example. However, the present technology is not applied only to the vehicle 20, and other objects (moving objects) It is also possible to apply the present technology to For example, the present technology can be applied to a vehicle (robot) traveling in a factory, an airplane (including a drone), and the like.

In the above-described embodiment, the case where the position, posture, speed, and the like of the vehicle 20 are estimated by a stereo camera system using two imaging devices 401 in the vehicle 20 has been described as an example. Alternatively, the position, posture, speed, and the like of the moving body may be estimated using three or more imaging devices.

Also, it is possible to estimate the position, posture, speed, etc. of the moving body based on an image taken from the vehicle (moving body) by a method other than the above-described method.

In addition, the present technology can be applied not only to a case where the moving body is a vehicle that is moved by a prime mover, but also to a case that a vehicle is driven by rails or overhead lines, a vehicle that is moved by human power, or the like. Furthermore, the present technology can be applied regardless of differences in vehicle driving methods (for example, automatic driving, manual driving, remote control, etc.).

In addition, the present technology can be applied to, for example, a case where many moving bodies come and go and a blind spot from the moving body is likely to occur. For example, when a person wearing a head-mounted display uses an AR (augmented reality) or VR (virtual reality) application, such as collision, rear-end collision, contact, etc. between people on a road or event venue where many people come and go. It can be applied to avoid accidents and guide to a predetermined position.

<About recording media>
The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing various programs by installing a computer incorporated in dedicated hardware.

FIG. 13 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processing by a program. In the computer, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are connected to each other via a bus 1004. An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

The input unit 1006 includes a keyboard, a mouse, a microphone, and the like. The output unit 1007 includes a display, a speaker, and the like. The storage unit 1008 includes a hard disk, a nonvolatile memory, and the like. The communication unit 1009 includes a network interface. The drive 1010 drives a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

In the computer configured as described above, the CPU 1001 loads, for example, the program stored in the storage unit 1008 to the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the program. Is performed.

The program executed by the computer (CPU 1001) can be provided by being recorded on the removable medium 1011 as a package medium, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

In the computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by attaching the removable medium 1011 to the drive 1010. Further, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. In addition, the program can be installed in the ROM 1002 or the storage unit 1008 in advance.

The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

In addition, in this specification, the system represents the entire apparatus composed of a plurality of apparatuses.

Note that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

In addition, this technique can also take the following structures.
(1)
A projection control unit that controls a plurality of projection devices that project a predetermined pattern;
A route creation unit that creates a route when the moving body moves to the destination, and
The projection control unit controls projection of the projection device located on the path created by the path creation unit.
(2)
The projection control unit sets a pattern projected by the projection apparatus installed at a predetermined position as a marker pattern, and sets a pattern projected by the projection apparatus installed at a position other than the predetermined position as a random pattern. The information processing apparatus according to (1).
(3)
The information processing apparatus according to (2), wherein the predetermined position is at least one of a corner and a vicinity of the destination.
(4)
The route creation unit creates a route to the destination to which the moving body can move using at least one of the width, length, height, and minimum rotation radius of the moving body. The information processing apparatus according to any one of (3) to (3).
(5)
The destination is a parking space;
The information processing apparatus according to any one of (1) to (4), wherein the route creation unit creates the route up to a position where the moving body starts an approach for parking in the parking space.
(6)
A holding unit for holding a global map indicating the position of a feature point in a three-dimensional space within a predetermined area;
The system further includes a global map update unit that acquires a local map indicating a position in a three-dimensional space of a feature point in an image photographed from the moving body, and updates the global map based on the local map. The information processing apparatus according to any one of (5) to (5).
(7)
The information processing apparatus according to (6), wherein the route creation unit creates a route with reference to the global map held in the holding unit.
(8)
The route created by the route creation unit, marker information of the marker pattern, and a global map indicating the position of a feature point in a predetermined region in a three-dimensional space are supplied to the moving body. Information processing device.
(9)
Controlling a plurality of projection devices that project a predetermined pattern;
Including a step of creating a route when the moving object moves to the destination,
The projection control is performed by controlling the projection of the projection device located on the created path.
(10)
An imaging unit for imaging a predetermined pattern projected by the projection device;
An estimation unit that estimates a self-position using the pattern imaged by the imaging unit;
An information processing apparatus comprising: a route supplied from another device; and a control signal generation unit that generates a control signal for controlling each unit to move on the route based on the estimated self-position.
(11)
The information processing apparatus according to (10), further including a local map generation unit that generates a local map indicating a position in a three-dimensional space of the feature point in the image captured by the imaging unit.
(12)
A holding unit for holding a global map indicating the position of a feature point in a three-dimensional space in a predetermined region supplied from the other device;
A determination unit that determines whether or not the position corresponding to the local map has changed in the global map; and
The information processing apparatus according to (11), wherein when the determination unit determines that there is a change, the local map is supplied to the other apparatus.
(13)
The information processing apparatus according to (11), wherein the route is corrected with the local map.
(14)
The information processing apparatus according to any one of (10) to (13), wherein when the marker is detected from the predetermined pattern, the self-position estimated by the estimation unit is corrected.
(15)
The information processing apparatus according to any one of (10) to (14), wherein the control signal is a signal for notifying a user of the route.
(16)
Capture a predetermined pattern projected by the projector,
Use the captured pattern to estimate the self-position,
An information processing method including a step of generating a control signal for controlling each unit to move on the route based on a route supplied from another device and the estimated self-position.

10 parking lot, 20 vehicles, 51 parking support device, 52 imaging device, 53 projection device, 61 network, 71 database, 81 parking support vehicle side device, 101 entrance vehicle detection unit, 102 vehicle information acquisition unit, 103 projection control unit, 104 pattern holding unit, 105 route creation unit, 106 global map holding unit, 107 global map update unit, 108 communication unit, 201 self-position estimation processing unit, 202 information reception instruction unit, 203 communication unit, 204 global map update determination unit, 205 information holding unit, 206 local route generation unit, 207 vehicle control signal generation unit, 208 notification generation unit

Claims

A projection control unit that controls a plurality of projection devices that project a predetermined pattern;
A route creation unit that creates a route when the moving body moves to the destination, and
The projection control unit controls projection of the projection device located on the path created by the path creation unit.
The projection control unit sets a pattern projected by the projection apparatus installed at a predetermined position as a marker pattern, and sets a pattern projected by the projection apparatus installed at a position other than the predetermined position as a random pattern. The information processing apparatus according to claim 1.
The information processing apparatus according to claim 2, wherein the predetermined position is at least one of a corner and the vicinity of the destination.
The route creation unit creates a route to the destination to which the moving body can move using at least one information of the width, length, height, and minimum turning radius of the moving body. The information processing apparatus described in 1.
The destination is a parking space;
The information processing apparatus according to claim 1, wherein the route creation unit creates the route up to a position where the moving body starts an approach for parking in the parking space.
A holding unit for holding a global map indicating the position of a feature point in a three-dimensional space within a predetermined area;
The global map update part which acquires the local map which shows the position in the three-dimensional space of the feature point in the image image | photographed from the said mobile body, and updates the said global map based on the said local map is further provided. The information processing apparatus described in 1.
The information processing apparatus according to claim 6, wherein the route creation unit creates a route with reference to the global map held in the holding unit.
3. The information according to claim 2, wherein a global map that indicates a path created by the path creation unit, marker information of the marker pattern, and a position of a feature point in a predetermined area in a three-dimensional space is supplied to the moving body. Processing equipment.
Controlling a plurality of projection devices that project a predetermined pattern;
Including a step of creating a route when the moving object moves to the destination,
The projection control is performed by controlling the projection of the projection device located on the created path.
An imaging unit for imaging a predetermined pattern projected by the projection device;
An estimation unit that estimates a self-position using the pattern imaged by the imaging unit;
An information processing apparatus comprising: a route supplied from another device; and a control signal generation unit that generates a control signal for controlling each unit to move on the route based on the estimated self-position.
The information processing apparatus according to claim 10, further comprising a local map generation unit configured to generate a local map indicating a position in a three-dimensional space of a feature point in an image captured by the imaging unit.
A holding unit for holding a global map indicating the position of a feature point in a three-dimensional space in a predetermined region supplied from the other device;
A determination unit that determines whether or not the position corresponding to the local map has changed in the global map; and
The information processing apparatus according to claim 11, wherein when the determination unit determines that there is a change, the local map is supplied to the other apparatus.
The information processing apparatus according to claim 11, wherein the route is corrected by the local map.
The information processing apparatus according to claim 10, wherein when a marker is detected from the predetermined pattern, the self-position estimated by the estimation unit is corrected.
The information processing apparatus according to claim 10, wherein the control signal is a signal for notifying a user of the route.
Capture a predetermined pattern projected by the projector,
Use the captured pattern to estimate the self-position,
An information processing method including a step of generating a control signal for controlling each unit to move on the route based on a route supplied from another device and the estimated self-position.