JP7402695B2 - Information transmission method and client device - Google Patents

Description

The present disclosure relates to an information transmission method and a client device.

Devices and services that utilize three-dimensional data are expected to become widespread in a wide range of fields, such as computer vision for autonomous operation of cars or robots, map information, monitoring, infrastructure inspection, and video distribution. Three-dimensional data is acquired by various methods, such as a distance sensor such as a range finder, a stereo camera, or a combination of multiple monocular cameras.

One of the methods of representing three-dimensional data is a method called point cloud, which represents the shape of a three-dimensional structure using a group of points in a three-dimensional space. A point cloud stores the positions and colors of point clouds. Point clouds are expected to become the mainstream method for expressing three-dimensional data, but point clouds require a very large amount of data. Therefore, when storing or transmitting three-dimensional data, it is essential to compress the amount of data through encoding, just as with two-dimensional moving images (an example is MPEG-4 AVC or HEVC standardized by MPEG). Become.

Further, point cloud compression is partially supported by a public library (Point Cloud Library) that performs point cloud-related processing.

Furthermore, a technique is known that uses three-dimensional map data to search for and display facilities located around a vehicle (for example, see Patent Document 1).

International Publication No. 2014/020663

It is desired to appropriately transmit information for creating such three-dimensional data.

An object of the present disclosure is to provide an information transmission method or a client device that can appropriately transmit information for creating three-dimensional data.

An information transmission method according to an aspect of the present disclosure is an information transmission method in a client device mounted on a mobile body, the information transmission method being a method for transmitting information in a client device mounted on a mobile body, wherein the surrounding situation of the mobile body is obtained by a first sensor mounted on the mobile body. acquiring first sensor information indicating the first sensor information, storing the first sensor information in a storage unit, and obtaining second sensor information having a smaller amount of data than the first sensor information obtained by a second sensor mounted on the moving object; By acquiring the beam directionality in the direction of the position of the access point capable of high-speed communication indicated in the map data, it is determined whether the client device can connect to the access point capable of high-speed communication, and the client device If the device determines that it is connectable to the access point capable of high-speed communication, transmits the first sensor information to the server, and if it is determined that the client device is not connectable to the access point capable of high-speed communication, Sending the second sensor information to the server.

The present disclosure can provide an information transmission method or a client device that can appropriately transmit information for creating three-dimensional data.

FIG. 1 is a diagram showing the structure of encoded three-dimensional data according to the first embodiment. FIG. 2 is a diagram illustrating an example of a prediction structure between SPCs belonging to the lowest layer of GOS according to the first embodiment. FIG. 3 is a diagram illustrating an example of a prediction structure between layers according to the first embodiment. FIG. 4 is a diagram illustrating an example of the GOS encoding order according to the first embodiment. FIG. 5 is a diagram illustrating an example of the GOS encoding order according to the first embodiment. FIG. 6 is a block diagram of a three-dimensional data encoding device according to the first embodiment. FIG. 7 is a flowchart of encoding processing according to the first embodiment. FIG. 8 is a block diagram of a three-dimensional data decoding device according to the first embodiment. FIG. 9 is a flowchart of decoding processing according to the first embodiment. FIG. 10 is a diagram illustrating an example of meta information according to the first embodiment. FIG. 11 is a diagram illustrating a configuration example of the SWLD according to the second embodiment. FIG. 12 is a diagram illustrating an example of the operation of the server and client according to the second embodiment. FIG. 13 is a diagram illustrating an example of the operation of the server and client according to the second embodiment. FIG. 14 is a diagram illustrating an example of the operation of the server and client according to the second embodiment. FIG. 15 is a diagram illustrating an example of the operation of the server and client according to the second embodiment. FIG. 16 is a block diagram of a three-dimensional data encoding device according to the second embodiment. FIG. 17 is a flowchart of encoding processing according to the second embodiment. FIG. 18 is a block diagram of a three-dimensional data decoding device according to the second embodiment. FIG. 19 is a flowchart of decoding processing according to the second embodiment. FIG. 20 is a diagram illustrating a configuration example of a WLD according to the second embodiment. FIG. 21 is a diagram illustrating an example of an octree structure of WLD according to the second embodiment. FIG. 22 is a diagram illustrating a configuration example of the SWLD according to the second embodiment. FIG. 23 is a diagram illustrating an example of an octree structure of SWLD according to the second embodiment. FIG. 24 is a schematic diagram showing how three-dimensional data is transmitted and received between vehicles according to the third embodiment. FIG. 25 is a diagram illustrating an example of three-dimensional data transmitted between vehicles according to the third embodiment. FIG. 26 is a block diagram of a three-dimensional data creation device according to the third embodiment. FIG. 27 is a flowchart of three-dimensional data creation processing according to the third embodiment. FIG. 28 is a block diagram of a three-dimensional data transmitting device according to the third embodiment. FIG. 29 is a flowchart of three-dimensional data transmission processing according to the third embodiment. FIG. 30 is a block diagram of a three-dimensional data creation device according to the third embodiment. FIG. 31 is a flowchart of three-dimensional data creation processing according to the third embodiment. FIG. 32 is a block diagram of a three-dimensional data transmitting device according to the third embodiment. FIG. 33 is a flowchart of three-dimensional data transmission processing according to the third embodiment. FIG. 34 is a block diagram of a three-dimensional information processing apparatus according to Embodiment 4. FIG. 35 is a flowchart of a three-dimensional information processing method according to the fourth embodiment. FIG. 36 is a flowchart of a three-dimensional information processing method according to the fourth embodiment. FIG. 37 is a diagram for explaining three-dimensional data transmission processing according to the fifth embodiment. FIG. 38 is a block diagram of a three-dimensional data creation device according to the fifth embodiment. FIG. 39 is a flowchart of a three-dimensional data creation method according to the fifth embodiment. FIG. 40 is a flowchart of a three-dimensional data creation method according to the fifth embodiment. FIG. 41 is a flowchart of a display method according to the sixth embodiment. FIG. 42 is a diagram illustrating an example of the surrounding environment seen through the windshield according to the sixth embodiment. FIG. 43 is a diagram illustrating a display example of a head-up display according to the sixth embodiment. FIG. 44 is a diagram illustrating a display example of the head-up display after adjustment according to the sixth embodiment. FIG. 45 is a diagram showing the configuration of a system according to Embodiment 7. FIG. 46 is a block diagram of a client device according to Embodiment 7. FIG. 47 is a block diagram of a server according to Embodiment 7. FIG. 48 is a flowchart of three-dimensional data creation processing by the client device according to the seventh embodiment. FIG. 49 is a flowchart of sensor information transmission processing by the client device according to the seventh embodiment. FIG. 50 is a flowchart of three-dimensional data creation processing by the server according to the seventh embodiment. FIG. 51 is a flowchart of three-dimensional map transmission processing by the server according to the seventh embodiment. FIG. 52 is a diagram showing a configuration of a modified example of the system according to the seventh embodiment. FIG. 53 is a diagram showing the configuration of a server and a client device according to the seventh embodiment. FIG. 54 is a diagram showing the configuration of a server and a client device according to Embodiment 8. FIG. 55 is a flowchart of processing by the client device according to the eighth embodiment. FIG. 56 is a diagram showing the configuration of a sensor information collection system according to Embodiment 8.

An information transmission method according to an aspect of the present disclosure is an information transmission method in a client device mounted on a mobile body, the information transmission method being a sensor indicating the surrounding situation of the mobile body obtained by a sensor mounted on the mobile body. acquire information, store the sensor information in a storage unit, determine whether the mobile object exists in an environment where the sensor information can be transmitted to the server, and enable the mobile object to transmit the sensor information to the server. If it is determined that the sensor information exists in a suitable environment, the sensor information is transmitted to the server.

According to this, the information transmission method can appropriately transmit information for creating three-dimensional data.

For example, the information transmission method may further include creating three-dimensional data around the moving object from the sensor information, and estimating the self-position of the moving object using the created three-dimensional data. .

For example, the information transmission method further includes transmitting a three-dimensional map transmission request to the server, receiving the three-dimensional map from the server, and estimating the self-position using the three-dimensional data and the three-dimensional map. The self-position may be estimated using the following.

According to this, the information transmission method can improve the accuracy of self-position estimation.

For example, the sensor information may include at least one of information obtained by a laser sensor, a brightness image, an infrared image, a depth image, sensor position information, and sensor speed information.

For example, the sensor information may include acquisition location information indicating the position of the mobile object or the sensor when the sensor information was acquired by the sensor.

For example, the sensor information may include acquisition time information indicating the time when the sensor information was acquired by the sensor.

For example, the information transmission method may further acquire time information from the server and generate the acquisition time information using the acquired time information.

According to this, it is possible to synchronize acquisition time information of sensor information transmitted from a plurality of client devices.

For example, the information transmission method further includes receiving a sensor information transmission request including specification information specifying a location and time from the server, and storing in the storage unit the sensor information obtained at the location and time indicated by the specification information. If it is determined that the information is stored and the mobile object exists in an environment where the sensor information can be transmitted to the server, the sensor information obtained at the location and time indicated by the specified information is transmitted to the server. You may also send it to

For example, the information transmitting method may further include deleting the sensor information that has already been transmitted to the server from the storage unit.

According to this, the capacity of the storage section can be reduced.

For example, the information transmission method further includes a method in which a difference between a position of the mobile body or the sensor at the time when the sensor information was acquired by the sensor and a current position of the mobile body or the sensor is determined in advance. The sensor information may be deleted from the storage unit when the distance exceeds the specified distance.

According to this, the capacity of the storage section can be reduced.

For example, the information transmission method further includes deleting the sensor information from the storage unit when a difference between a time when the sensor information was acquired by the sensor and a current time exceeds a predetermined time. You may.

According to this, the capacity of the storage section can be reduced.

Further, a client device according to an aspect of the present disclosure is a client device installed in a mobile body, and includes a processor and a memory, and the processor uses the memory to Obtaining sensor information indicating the surrounding situation of the mobile body obtained by a sensor, storing the sensor information in a storage unit, and determining whether the mobile body exists in an environment where the sensor information can be transmitted to a server. However, when it is determined that the mobile object exists in an environment where the sensor information can be transmitted to the server, the sensor information is transmitted to the server.

According to this, the client device can appropriately transmit information for creating three-dimensional data.

Note that these comprehensive or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, and the system, method, integrated circuit, computer program and a recording medium may be used in any combination.

Hereinafter, embodiments will be specifically described with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are examples, and do not limit the present disclosure. Further, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the most significant concept will be described as arbitrary constituent elements.

(Embodiment 1)
First, the data structure of encoded three-dimensional data (hereinafter also referred to as encoded data) according to this embodiment will be explained. FIG. 1 is a diagram showing the structure of encoded three-dimensional data according to this embodiment.

In this embodiment, the three-dimensional space is divided into spaces (SPC) corresponding to pictures in video encoding, and three-dimensional data is encoded in units of spaces. The space is further divided into volumes (VLMs) corresponding to macroblocks in video encoding, and prediction and transformation are performed in units of VLMs. A volume includes a plurality of voxels (VXL), which are the minimum units to which position coordinates are associated. Note that prediction, similar to prediction performed on two-dimensional images, refers to other processing units, generates predicted 3D data similar to the processing unit to be processed, and combines the predicted 3D data with the processing unit to be processed. It is to encode the difference between the processing unit and the processing unit. Furthermore, this prediction includes not only spatial prediction that refers to other prediction units at the same time, but also temporal prediction that refers to prediction units at different times.

For example, when a three-dimensional data encoding device (hereinafter also referred to as an encoding device) encodes a three-dimensional space expressed by point cloud data such as a point cloud, the point cloud is Each point or multiple points included in a voxel are collectively encoded. By subdividing the voxels, the three-dimensional shape of the point cloud can be expressed with high precision, and by increasing the size of the voxels, the three-dimensional shape of the point cloud can be roughly expressed.

Note that although the following explanation will be given using an example in which the three-dimensional data is a point cloud, the three-dimensional data is not limited to a point cloud, and may be three-dimensional data in any format.

Furthermore, hierarchically structured voxels may be used. In this case, in the n-th hierarchy, it may be indicated in order whether or not a sample point exists in the n-1th or lower hierarchy (lower layer of the n-th hierarchy). For example, when decoding only the n-th layer, if a sample point exists in the n-1th layer or lower, decoding can be performed by assuming that the sample point exists at the center of the voxel of the n-th layer.

Further, the encoding device acquires point cloud data using a distance sensor, a stereo camera, a monocular camera, a gyro, an inertial sensor, or the like.

Similar to video encoding, space can be independently decodable intra space (I-SPC), predictive space (P-SPC) that can only be referenced in one direction, and bidirectional reference is possible. It is classified into one of at least three predictive structures including a bidirectional space (B-SPC). Furthermore, the space has two types of time information: decoding time and display time.

Further, as shown in FIG. 1, there is a GOS (Group Of Space), which is a random access unit, as a processing unit including a plurality of spaces. Furthermore, a world (WLD) exists as a processing unit that includes multiple GOS.

The spatial area occupied by the world is associated with an absolute position on the earth using GPS or latitude and longitude information. This location information is stored as meta information. Note that the meta information may be included in the encoded data or may be transmitted separately from the encoded data.

Furthermore, within the GOS, all SPCs may be three-dimensionally adjacent, or there may be SPCs that are not three-dimensionally adjacent to other SPCs.

Note that hereinafter, processing such as encoding, decoding, or referencing of three-dimensional data included in a processing unit such as GOS, SPC, or VLM will also be simply referred to as encoding, decoding, or referencing the processing unit. Further, the three-dimensional data included in the processing unit includes, for example, at least one set of a spatial position such as three-dimensional coordinates and a characteristic value such as color information.

Next, the prediction structure of SPC in GOS will be explained. Multiple SPCs within the same GOS or multiple VLMs within the same SPC occupy different spaces, but have the same time information (decoding time and display time).

Furthermore, the first SPC in the decoding order within the GOS is the I-SPC. Furthermore, there are two types of GOS: closed GOS and open GOS. A closed GOS is a GOS that can decode all SPCs in the GOS when starting decoding from the first I-SPC. In an open GOS, some SPCs whose display time is earlier than the first I-SPC in the GOS refer to a different GOS, and decoding cannot be performed only with that GOS.

Note that in encoded data such as map information, WLD may be decoded in the reverse direction from the encoding order, and if there is dependence between GOS, it is difficult to reproduce in the reverse direction. Therefore, in such a case, closed GOS is basically used.

Further, GOS has a layer structure in the height direction, and encoding or decoding is performed in order from the SPC of the lower layer.

FIG. 2 is a diagram showing an example of a prediction structure between SPCs belonging to the lowest layer of GOS. FIG. 3 is a diagram showing an example of a prediction structure between layers.

There are one or more I-SPCs within a GOS. Objects such as humans, animals, cars, bicycles, traffic lights, and landmark buildings exist in a three-dimensional space, and it is particularly effective to encode small objects as I-SPC. For example, a three-dimensional data decoding device (hereinafter also referred to as a decoding device) decodes only the I-SPC in the GOS when decoding a GOS with a low throughput or at high speed.

Furthermore, the encoding device may switch the encoding interval or appearance frequency of the I-SPC depending on the density of objects in the WLD.

Furthermore, in the configuration shown in FIG. 3, the encoding device or the decoding device encodes or decodes a plurality of layers in order from the lower layer (layer 1). This makes it possible to increase the priority of data near the ground, which has a larger amount of information for autonomous vehicles, for example.

Note that encoded data used in drones and the like may be encoded or decoded in order from the SPC of the upper layer in the height direction within the GOS.

Further, the encoding device or the decoding device may encode or decode a plurality of layers so that the decoding device can roughly grasp the GOS and gradually increase the resolution. For example, the encoding device or the decoding device may encode or decode layers 3, 8, 1, 9, . . . in order.

Next, how to handle static objects and dynamic objects will be explained.

In three-dimensional space, there are static objects or scenes such as buildings or roads (hereinafter collectively referred to as static objects) and dynamic objects such as cars or people (hereinafter referred to as dynamic objects). do. Object detection is performed separately by extracting feature points from point cloud data or camera images such as a stereo camera. Here, an example of a dynamic object encoding method will be described.

The first method is to encode static objects and dynamic objects without distinguishing between them. The second method is to distinguish between static objects and dynamic objects using identification information.

For example, GOS is used as the identification unit. In this case, a GOS including an SPC forming a static object and a GOS including an SPC forming a dynamic object are distinguished by identification information stored within the encoded data or separately from the encoded data.

Alternatively, SPC may be used as the identification unit. In this case, an SPC that includes a VLM that constitutes a static object and an SPC that includes a VLM that constitutes a dynamic object are distinguished by the identification information.

Alternatively, VLM or VXL may be used as the identification unit. In this case, a VLM or VXL that includes a static object and a VLM or VXL that includes a dynamic object are distinguished by the identification information.

Further, the encoding device may encode the dynamic object as one or more VLMs or SPCs, and encode the VLM or SPC including the static object and the SPC including the dynamic object as mutually different GOSs. Furthermore, when the size of the GOS is variable depending on the size of the dynamic object, the encoding device separately stores the size of the GOS as meta information.

Further, the encoding device may encode static objects and dynamic objects independently of each other, and superimpose the dynamic objects on a world made up of static objects. At this time, the dynamic object is composed of one or more SPCs, and each SPC is associated with one or more SPCs that constitute the static object on which the SPC is superimposed. Note that the dynamic object may be expressed by one or more VLMs or VXLs instead of SPC.

Further, the encoding device may encode the static object and the dynamic object as different streams.

Furthermore, the encoding device may generate a GOS that includes one or more SPCs that constitute a dynamic object. Further, the encoding device may set the GOS (GOS_M) including the dynamic object and the GOS of the static object corresponding to the spatial area of GOS_M to be the same size (occupy the same spatial area). Thereby, superimposition processing can be performed in units of GOS.

P-SPC or B-SPC constituting a dynamic object may refer to SPC included in a different encoded GOS. In a case where the position of a dynamic object changes over time and the same dynamic object is encoded as GOSs at different times, references across GOSs are effective from the viewpoint of compression rate.

Furthermore, the first method and the second method may be switched depending on the purpose of the encoded data. For example, when using encoded three-dimensional data as a map, it is desirable to be able to separate dynamic objects, so the encoding device uses the second method. On the other hand, when encoding three-dimensional data of an event such as a concert or sports, the encoding device uses the first method if there is no need to separate dynamic objects.

Further, the decoding time and display time of GOS or SPC can be stored in encoded data or as meta information. Further, the time information of all static objects may be the same. At this time, the actual decoding time and display time may be determined by the decoding device. Alternatively, a different value may be assigned to each GOS or SPC as the decoding time, and the same value may be assigned to all as the display time. Furthermore, like a decoder model in video encoding such as HRD (Hypothetical Reference Decoder) of HEVC, if the decoder has a buffer of a predetermined size and reads the bitstream at a predetermined bit rate according to the decoding time, decoding can be performed without failure. It is also possible to introduce a model that guarantees

Next, the arrangement of GOS within the world will be explained. The coordinates of the three-dimensional space in the world are expressed by three coordinate axes (x-axis, y-axis, and z-axis) that are orthogonal to each other. By providing a predetermined rule for the encoding order of GOS, encoding can be performed so that spatially adjacent GOS are consecutive in the encoded data. For example, in the example shown in FIG. 4, GOS in the xz plane is continuously encoded. The y-axis value is updated after the encoding of all GOS within a certain xz plane is completed. That is, as encoding progresses, the world extends in the y-axis direction. Furthermore, the GOS index numbers are set in the order of encoding.

Here, the three-dimensional space of the world is in one-to-one correspondence with absolute geographic coordinates such as GPS or latitude and longitude. Alternatively, the three-dimensional space may be expressed by a relative position from a preset reference position. The directions of the x-axis, y-axis, and z-axis of the three-dimensional space are expressed as direction vectors determined based on latitude, longitude, etc., and the direction vectors are stored as meta information together with encoded data.

Further, the size of the GOS is fixed, and the encoding device stores the size as meta information. Further, the size of the GOS may be changed depending on whether the location is in an urban area or whether the location is indoors or outdoors. That is, the size of the GOS may be changed depending on the amount or nature of objects that have informational value. Alternatively, the encoding device may adaptively change the size of the GOS or the interval between I-SPCs within the GOS according to the density of objects within the same world. For example, the encoding device decreases the size of the GOS and shortens the interval between I-SPCs within the GOS as the density of objects increases.

In the example of FIG. 5, since the density of objects is high in the third to tenth GOS areas, the GOS is subdivided in order to realize random access at fine granularity. Note that the 7th to 10th GOS exist on the back side of the 3rd to 6th GOS, respectively.

Next, the configuration and operation flow of the three-dimensional data encoding device according to this embodiment will be explained. FIG. 6 is a block diagram of three-dimensional data encoding device 100 according to this embodiment. FIG. 7 is a flowchart showing an example of the operation of the three-dimensional data encoding device 100.

The three-dimensional data encoding device 100 shown in FIG. 6 generates encoded three-dimensional data 112 by encoding three-dimensional data 111. This three-dimensional data encoding device 100 includes an acquisition section 101, an encoding area determining section 102, a dividing section 103, and an encoding section 104.

As shown in FIG. 7, first, the acquisition unit 101 acquires three-dimensional data 111, which is point cloud data (S101).

Next, the encoding region determining unit 102 determines an encoding target region among the spatial regions corresponding to the acquired point cloud data (S102). For example, depending on the position of the user or the vehicle, the encoding area determination unit 102 determines a spatial area around the position as the area to be encoded.

Next, the dividing unit 103 divides the point cloud data included in the region to be encoded into processing units. Here, the processing unit is the above-mentioned GOS, SPC, etc. Further, this region to be encoded corresponds to, for example, the above-mentioned world. Specifically, the dividing unit 103 divides the point cloud data into processing units based on a preset GOS size or the presence or absence or size of a dynamic object (S103). Furthermore, the dividing unit 103 determines the starting position of the first SPC in the encoding order in each GOS.

Next, the encoding unit 104 generates encoded three-dimensional data 112 by sequentially encoding the plurality of SPCs in each GOS (S104).

Note that although an example has been shown in which the region to be encoded is divided into GOS and SPC and then each GOS is encoded, the processing procedure is not limited to the above. For example, a procedure may be used in which the configuration of one GOS is determined, then that GOS is encoded, and then the configuration of the next GOS is determined.

In this way, the three-dimensional data encoding device 100 generates encoded three-dimensional data 112 by encoding the three-dimensional data 111. Specifically, the three-dimensional data encoding device 100 divides the three-dimensional data into first processing units (GOS), each of which is a random access unit and is associated with a three-dimensional coordinate. A processing unit (GOS) is divided into a plurality of second processing units (SPC), and the second processing unit (SPC) is divided into a plurality of third processing units (VLM). Further, the third processing unit (VLM) includes one or more voxels (VXL), which are the minimum units to which position information is associated.

Next, the three-dimensional data encoding device 100 generates encoded three-dimensional data 112 by encoding each of the plurality of first processing units (GOS). Specifically, the three-dimensional data encoding device 100 encodes each of the plurality of second processing units (SPC) in each first processing unit (GOS). Furthermore, the three-dimensional data encoding device 100 encodes each of the plurality of third processing units (VLM) in each second processing unit (SPC).

For example, when the first processing unit (GOS) to be processed is a closed GOS, the three-dimensional data encoding device 100 can encode the second processing unit to be processed included in the first processing unit (GOS) to be processed. (SPC) is encoded with reference to another second processing unit (SPC) included in the first processing unit (GOS) to be processed. That is, the three-dimensional data encoding device 100 does not refer to the second processing unit (SPC) included in a first processing unit (GOS) different from the first processing unit (GOS) to be processed.

On the other hand, if the first processing unit (GOS) to be processed is an open GOS, the second processing unit (SPC) to be processed included in the first processing unit (GOS) to be processed is Another second processing unit (SPC) included in one processing unit (GOS), or a second processing unit (SPC) included in a first processing unit (GOS) different from the first processing unit (GOS) to be processed ).

In addition, the three-dimensional data encoding device 100 selects the type of the second processing unit (SPC) to be processed, a first type (I-SPC) that does not refer to other second processing units (SPC), and another type (I-SPC) that does not refer to other second processing units (SPC). Select one of the second type (P-SPC) that refers to the second processing unit (SPC) and the third type that refers to the other two second processing units (SPC), and process according to the selected type. Encode the second processing unit (SPC) of interest.

Next, the configuration and operation flow of the three-dimensional data decoding device according to this embodiment will be explained. FIG. 8 is a block diagram of the three-dimensional data decoding device 200 according to this embodiment. FIG. 9 is a flowchart showing an example of the operation of the three-dimensional data decoding device 200.

A three-dimensional data decoding device 200 shown in FIG. 8 generates decoded three-dimensional data 212 by decoding encoded three-dimensional data 211. Here, the encoded three-dimensional data 211 is, for example, the encoded three-dimensional data 112 generated by the three-dimensional data encoding device 100. This three-dimensional data decoding device 200 includes an acquisition section 201, a decoding start GOS determining section 202, a decoding SPC determining section 203, and a decoding section 204.

First, the acquisition unit 201 acquires encoded three-dimensional data 211 (S201). Next, the decoding start GOS determining unit 202 determines the GOS to be decoded (S202). Specifically, the decoding start GOS determination unit 202 refers to meta information stored within the encoded three-dimensional data 211 or separately from the encoded three-dimensional data to determine the spatial position, object, or , the GOS including the SPC corresponding to the time is determined as the GOS to be decoded.

Next, the decoded SPC determining unit 203 determines the type of SPC (I, P, B) to be decoded within the GOS (S203). For example, the decoded SPC determining unit 203 determines whether to (1) decode only I-SPC, (2) decode I-SPC and P-SPC, or (3) decode all types. Note that this step may not be performed if the types of SPCs to be decoded have been determined in advance, such as when all SPCs are decoded.

Next, the decoding unit 204 acquires the address position where the first SPC in the decoding order (same as the encoding order) starts in the encoded three-dimensional data 211 in the GOS, and starts from the address position by decoding the first SPC. The encoded data is obtained, and each SPC is sequentially decoded starting from the first SPC (S204). Note that the above address position is stored in meta information or the like.

In this way, the three-dimensional data decoding device 200 decodes the decoded three-dimensional data 212. Specifically, the three-dimensional data decoding device 200 decodes each of the encoded three-dimensional data 211 of a first processing unit (GOS), which is a random access unit and is each associated with a three-dimensional coordinate. As a result, decoded three-dimensional data 212 of the first processing unit (GOS) is generated. More specifically, the three-dimensional data decoding device 200 decodes each of the plurality of second processing units (SPC) in each first processing unit (GOS). Furthermore, the three-dimensional data decoding device 200 decodes each of the plurality of third processing units (VLM) in each second processing unit (SPC).

The meta information for random access will be explained below. This meta information is generated by the three-dimensional data encoding device 100 and included in the encoded three-dimensional data 112 (211).

In conventional random access for two-dimensional moving images, decoding is started from the first frame of the random access unit near the specified time. On the other hand, in the world, random access to space (coordinates, objects, etc.) is assumed in addition to time.

Therefore, in order to realize random access to at least three elements: coordinates, objects, and time, a table is prepared that associates each element with a GOS index number. Furthermore, the index number of the GOS is associated with the address of the I-SPC that is the beginning of the GOS. FIG. 10 is a diagram illustrating an example of a table included in meta information. Note that it is not necessary to use all the tables shown in FIG. 10, and it is sufficient to use at least one table.

As an example, random access starting from coordinates will be described below. When accessing the coordinates (x2, y2, z2), first, the coordinate-GOS table is referred to and it is found that the point whose coordinates are (x2, y2, z2) is included in the second GOS. Next, referring to the GOS address table, it is found that the address of the first I-SPC in the second GOS is addr (2), so the decoding unit 204 acquires data from this address and starts decoding. do.

Note that the address may be an address in a logical format or a physical address of the HDD or memory. Furthermore, information specifying a file segment may be used instead of the address. For example, a file segment is a unit obtained by segmenting one or more GOS.

Furthermore, if an object spans multiple GOSs, the object-GOS table may indicate multiple GOSs to which the object belongs. If the plurality of GOSs are closed GOSs, the encoding device and the decoding device can perform encoding or decoding in parallel. On the other hand, if the plurality of GOSs are open GOSs, compression efficiency can be further improved by mutually referencing the plurality of GOSs.

Examples of objects include humans, animals, cars, bicycles, traffic lights, and landmark buildings. For example, the three-dimensional data encoding device 100 extracts characteristic points of objects from a three-dimensional point cloud when encoding a world, detects objects based on the characteristic points, and transfers the detected objects to random access points. Can be set as .

In this way, the three-dimensional data encoding device 100 generates first information indicating a plurality of first processing units (GOS) and three-dimensional coordinates associated with each of the plurality of first processing units (GOS). generate. Furthermore, the encoded three-dimensional data 112 (211) includes this first information. Further, the first information further indicates at least one of the object, time, and data storage destination, which are associated with each of the plurality of first processing units (GOS).

The three-dimensional data decoding device 200 acquires first information from the encoded three-dimensional data 211, and uses the first information to decode the encoded three-dimensional data of the first processing unit corresponding to the specified three-dimensional coordinates, object, or time. The original data 211 is identified, and the encoded three-dimensional data 211 is decoded.

Examples of other meta information will be described below. In addition to the meta information for random access, the three-dimensional data encoding device 100 may generate and store the following meta information. Furthermore, the three-dimensional data decoding device 200 may use this meta information at the time of decoding.

When three-dimensional data is used as map information, a profile may be defined depending on the purpose, and information indicating the profile may be included in the meta information. For example, profiles for urban areas or suburbs, or for flying objects are defined, and the maximum or minimum size of the world, SPC, or VLM is defined for each profile. For example, in urban areas, more detailed information is required than in suburban areas, so the minimum size of the VLM is set smaller.

The meta information may include a tag value indicating the type of object. This tag value is associated with the VLM, SPC, or GOS that constitutes the object. For example, a tag value of "0" indicates a "person," a tag value of "1" indicates a "car," a tag value of "2" indicates a "traffic light," and so on.A tag value is set for each type of object. Good too. Alternatively, if the type of object is difficult to determine or does not need to be determined, a tag value indicating properties such as size or whether it is a dynamic object or a static object may be used.

Further, the meta information may include information indicating the range of the spatial area occupied by the world.

Further, the meta information may store the size of the SPC or VXL as header information common to a plurality of SPCs, such as the entire encoded data stream or the SPC in GOS.

Further, the meta information may include identification information of a distance sensor or camera used to generate the point cloud, or information indicating the positional accuracy of a point group within the point cloud.

Further, the meta information may include information indicating whether the world is composed of only static objects or includes dynamic objects.

Modifications of this embodiment will be described below.

The encoding device or decoding device may encode or decode two or more different SPCs or GOSs in parallel. GOS to be encoded or decoded in parallel can be determined based on meta information indicating the spatial position of the GOS.

In cases where three-dimensional data is used as a spatial map for moving vehicles or flying objects, or when such a spatial map is generated, the encoding device or decoding device uses information such as GPS, route information, or zoom magnification. The GOS or SPC included in the space specified based on the information may be encoded or decoded.

Further, the decoding device may perform decoding in order from the space closest to the self-location or the travel route. The encoding device or the decoding device may encode or decode a space far from the own position or the driving route with lower priority than a space close to the vehicle. Here, lowering the priority means lowering the processing order, lowering the resolution (processing by thinning out), lowering the image quality (increasing encoding efficiency, for example, increasing the quantization step), etc. .

Furthermore, when decoding encoded data that is hierarchically encoded in space, the decoding device may decode only lower layers.

Further, the decoding device may preferentially decode from the lower hierarchy depending on the zoom magnification or purpose of the map.

In addition, in applications such as self-position estimation or object recognition performed during autonomous driving of a car or robot, the encoding device or decoding device lowers the resolution and encodes areas other than the area within a certain height from the road surface (the area where recognition is performed). It may also be encoded or decoded.

Further, the encoding device may individually encode point clouds expressing the spatial shapes of indoors and outdoors. For example, by separating the GOS that represents indoors (indoor GOS) and the GOS that represents outdoors (outdoor GOS), the decoding device can select the GOS to be decoded depending on the viewpoint position when using encoded data. can.

Furthermore, the encoding device may encode indoor GOS and outdoor GOS that have close coordinates so that they are adjacent to each other in the encoded stream. For example, the encoding device associates both identifiers and stores information indicating the associated identifiers in the encoded stream or in meta information stored separately. Thereby, the decoding device can identify indoor GOS and outdoor GOS whose coordinates are close to each other by referring to the information in the meta information.

Further, the encoding device may switch the size of GOS or SPC between indoor GOS and outdoor GOS. For example, the encoding device sets the GOS size smaller indoors than outdoors. Furthermore, the encoding device may change the accuracy in extracting feature points from the point cloud or the accuracy in object detection between indoor GOS and outdoor GOS.

Furthermore, the encoding device may add information to the encoded data for the decoding device to display dynamic objects in a manner that distinguishes them from static objects. Thereby, the decoding device can display the dynamic object together with a red frame or explanatory text. Note that the decoding device may display only a red frame or explanatory characters instead of the dynamic object. Additionally, the decoding device may display more detailed object types. For example, a red frame may be used for a car, and a yellow frame may be used for a person.

Further, the encoding device or the decoding device encodes dynamic objects and static objects as different SPCs or GOSs depending on the frequency of appearance of dynamic objects or the ratio of static objects to dynamic objects. Alternatively, it may be determined whether to decrypt or not. For example, if the appearance frequency or proportion of dynamic objects exceeds a threshold, SPC or GOS in which dynamic objects and static objects coexist is allowed, and if the appearance frequency or proportion of dynamic objects does not exceed the threshold , an SPC or GOS in which dynamic objects and static objects coexist is not allowed.

When detecting a dynamic object not from a point cloud but from two-dimensional image information from a camera, the encoding device separately acquires information for identifying the detection result (such as a frame or text) and the object position, These pieces of information may be encoded as part of three-dimensional encoded data. In this case, the decoding device displays auxiliary information (frames or characters) indicating the dynamic object in a superimposed manner on the decoding result of the static object.

Further, the encoding device may change the density of VXL or VLM in SPC depending on the complexity of the shape of the static object. For example, the encoding device sets the VXL or VLM more densely as the shape of the static object becomes more complex. Furthermore, the encoding device may determine a quantization step when quantizing spatial position or color information, etc., depending on the density of VXL or VLM. For example, the encoding device sets the quantization step smaller as the VXL or VLM becomes denser.

As described above, the encoding device or decoding device according to the present embodiment encodes or decodes space in space units having coordinate information.

Further, the encoding device and the decoding device perform encoding or decoding in units of volumes within the space. A volume includes voxels, which are the minimum units to which position information is associated.

In addition, the encoding device and the decoding device can associate arbitrary elements using a table that associates each element of spatial information including coordinates, objects, time, etc. with the GOP, or a table that associates each element. Perform encoding or decoding. Further, the decoding device determines the coordinates using the value of the selected element, identifies a volume, voxel, or space from the coordinates, and decodes the space that includes the volume or voxel, or the identified space.

Further, the encoding device determines a volume, voxel, or space that can be selected by an element through feature point extraction or object recognition, and encodes the volume, voxel, or space as a randomly accessible volume, voxel, or space.

A space refers to an I-SPC that can be encoded or decoded by itself, a P-SPC that can be encoded or decoded by referring to any one processed space, and any two processed spaces. It is classified into three types: B-SPC, which is encoded or decoded using

One or more volumes correspond to static or dynamic objects. A space containing static objects and a space containing dynamic objects are encoded or decoded as different GOSs. That is, an SPC containing static objects and an SPC containing dynamic objects are assigned to different GOSs.

Dynamic objects are encoded or decoded object by object and are mapped to one or more spaces containing static objects. That is, a plurality of dynamic objects are individually encoded, and the obtained encoded data of the plurality of dynamic objects is associated with an SPC including a static object.

The encoding device and the decoding device increase the priority of I-SPC in GOS and perform encoding or decoding. For example, the encoding device performs encoding so that I-SPC degradation is reduced (original three-dimensional data is more faithfully reproduced after decoding). Furthermore, the decoding device decodes only I-SPC, for example.

The encoding device may perform encoding by changing the frequency of using I-SPC depending on the density or number (amount) of objects in the world. That is, the encoding device changes the frequency of selecting I-SPCs depending on the number or density of objects included in the three-dimensional data. For example, the encoding device uses I space more frequently as objects in the world are denser.

Further, the encoding device sets a random access point for each GOS, and stores information indicating a spatial region corresponding to the GOS in the header information.

The encoding device uses, for example, a default value as the GOS spatial size. Note that the encoding device may change the size of the GOS depending on the number (amount) or density of objects or dynamic objects. For example, the encoding device reduces the spatial size of the GOS as the density of objects or dynamic objects increases or as the number of objects or dynamic objects increases.

Further, the space or volume includes a group of feature points derived using information obtained by a sensor such as a depth sensor, a gyro, or a camera. The coordinates of the feature point are set to the center position of the voxel. Further, by subdividing voxels, it is possible to improve the accuracy of positional information.

The feature point group is derived using multiple pictures. The plurality of pictures has at least two types of time information: actual time information and time information that is the same for the plurality of pictures associated with the space (for example, encoded time used for rate control, etc.).

Further, encoding or decoding is performed in units of GOS including one or more spaces.

The encoding device and the decoding device refer to the space in the processed GOS and predict the P space or B space in the GOS to be processed.

Alternatively, the encoding device and the decoding device predict the P space or B space in the GOS to be processed using the processed space in the GOS to be processed without referring to different GOSs.

Further, the encoding device and the decoding device transmit or receive encoded streams in units of worlds including one or more GOS.

Further, GOS has a layer structure in at least one direction within the world, and the encoding device and the decoding device perform encoding or decoding from the lower layer. For example, randomly accessible GOS belongs to the lowest layer. A GOS belonging to an upper layer refers to a GOS belonging to the same layer or lower. That is, the GOS is spatially divided in a predetermined direction and includes a plurality of layers each including one or more SPCs. The encoding device and the decoding device encode or decode each SPC by referring to an SPC included in the same layer as the SPC or a layer below the SPC.

Further, the encoding device and the decoding device continuously encode or decode GOS within a world unit including a plurality of GOS. The encoding device and the decoding device write or read information indicating the order (direction) of encoding or decoding as metadata. That is, the encoded data includes information indicating the encoding order of a plurality of GOS.

Further, the encoding device and the decoding device encode or decode two or more different spaces or GOSs in parallel.

Further, the encoding device and the decoding device encode or decode spatial information (coordinates, size, etc.) of space or GOS.

Further, the encoding device and the decoding device encode or decode the space or GOS included in the specific space specified based on external information regarding the own position and/or area size, such as GPS, route information, or magnification. .

An encoding device or a decoding device encodes or decodes a space far from its own position with a lower priority than a space close to it.

The encoding device sets a certain direction of the world according to the magnification or the purpose, and encodes a GOS having a layer structure in that direction. Further, the decoding device preferentially decodes GOS having a layer structure in one direction in a world set according to the magnification or purpose, starting from the lower layer.

The encoding device changes feature point extraction, object recognition accuracy, spatial region size, etc. included in a space between indoors and outdoors. However, the encoding device and the decoding device encode or decode indoor GOS and outdoor GOS that have close coordinates adjacently in the world, and also encode or decode these identifiers in association with each other.

(Embodiment 2)
When using point cloud encoded data in an actual device or service, it is desirable to transmit and receive necessary information depending on the purpose in order to suppress network bandwidth. However, until now, such a function has not existed in the encoding structure of three-dimensional data, and there has also been no encoding method for it.

In this embodiment, a three-dimensional data encoding method and a three-dimensional data encoding apparatus for providing a function of transmitting and receiving only necessary information according to the application in three-dimensional point cloud encoded data, and the A three-dimensional data decoding method and a three-dimensional data decoding device for decoding encoded data will be described.

A voxel (VXL) having a certain amount of feature or more is defined as a feature voxel (FVXL), and a world (WLD) composed of FVXL is defined as a sparse world (SWLD). FIG. 11 is a diagram showing an example of a sparse world and a configuration of the world. The SWLD includes an FGOS which is a GOS made up of FVXL, an FSPC which is an SPC made up of FVXL, and an FVLM which is a VLM made up of FVXL. The data structure and prediction structure of FGOS, FSPC and FVLM may be similar to GOS, SPC and VLM.

The feature amount is a feature amount that expresses the three-dimensional position information of the VXL or the visible light information of the VXL position, and is a feature amount that is often detected particularly at corners and edges of three-dimensional objects. Specifically, this feature amount is a three-dimensional feature amount or a visible light feature amount as shown below, but it may also be any other feature amount that represents the position, brightness, or color information of the VXL. It doesn't matter if it's something.

As the three-dimensional feature amount, a SHOT feature amount (Signature of Histograms of Orientations), a PFH feature amount (Point Feature Histograms), or a PPF feature amount (Point Pair Feature) is used.

The SHOT feature amount is obtained by dividing the area around the VXL, calculating the inner product between the reference point and the normal vector of the divided area, and creating a histogram. This SHOT feature has a high number of dimensions and a high ability to express features.

The PFH feature amount is obtained by selecting a large number of two-point sets near the VXL, calculating normal vectors, etc. from the two points, and creating a histogram. Since this PFH feature is a histogram feature, it has characteristics of being robust against some disturbances and having high feature expressiveness.

The PPF feature amount is a feature amount calculated using a normal vector or the like for each two points of VXL. Since all VXL is used for this PPF feature, it has robustness against occlusion.

In addition, as a feature amount of visible light, SIFT (Scale-Invariant Feature Transform), SURF (Speed Up Robust Features), or HOG (Histogram of Oriented Gradient) using information such as image brightness gradient information is used. dients) etc. can.

The SWLD is generated by calculating the feature amount from each VXL of the WLD and extracting the FVXL. Here, the SWLD may be updated every time the WLD is updated, or it may be updated periodically after a certain period of time has elapsed, regardless of the update timing of the WLD.

SWLD may be generated for each feature amount. For example, separate SWLDs may be generated for each feature amount, such as SWLD1 based on the SHOT feature amount and SWLD2 based on the SIFT feature amount, and the SWLDs may be used differently depending on the purpose. Further, the calculated feature amount of each FVXL may be held in each FVXL as feature amount information.

Next, a method of using the sparse world (SWLD) will be explained. Since SWLD includes only feature voxels (FVXL), the data size is generally smaller than WLD that includes all VXL.

In an application that uses feature amounts to achieve some purpose, by using SWLD information instead of WLD, it is possible to suppress the read time from the hard disk and the bandwidth and transfer time during network transfer. For example, network bandwidth and transfer time can be reduced by storing WLD and SWLD as map information on the server and switching the map information to be sent to WLD or SWLD in response to a client's request. . A specific example will be shown below.

FIGS. 12 and 13 are diagrams showing usage examples of SWLD and WLD. As shown in FIG. 12, when the client 1, which is an in-vehicle device, requires map information for self-position determination, the client 1 sends a request to the server to acquire map data for self-position estimation (S301). The server transmits the SWLD to the client 1 in response to the acquisition request (S302). The client 1 uses the received SWLD to determine its own position (S303). At this time, client 1 acquires VXL information around client 1 using various methods such as a range finder or other distance sensor, a stereo camera, or a combination of multiple monocular cameras, and uses the obtained VXL information and SWLD to Estimate location information. Here, the self-location information includes three-dimensional location information and orientation of the client 1.

As shown in FIG. 13, when the client 2, which is an in-vehicle device, needs map information for map drawing such as a three-dimensional map, the client 2 sends a request to the server to obtain map data for map drawing (S311 ). The server transmits the WLD to the client 2 in response to the acquisition request (S312). The client 2 uses the received WLD to draw a map (S313). At this time, the client 2 creates a rendered image using, for example, an image taken by itself using a visible light camera or the like and the WLD obtained from the server, and draws the created image on the screen of a car navigation system or the like.

As mentioned above, the server sends SWLD to the client for applications that mainly require the feature values of each VXL, such as self-position estimation, and sends WLD when detailed VXL information is required, such as map drawing. Send to client. This makes it possible to efficiently transmit and receive map data.

Note that the client may decide for itself which one is necessary, SWLD or WLD, and request the server to send SWLD or WLD. Furthermore, the server may determine whether to send SWLD or WLD depending on the client or network situation.

Next, a method for switching transmission and reception between sparse world (SWLD) and world (WLD) will be described.

It may also be possible to switch between receiving WLD and SWLD depending on the network band. FIG. 14 is a diagram showing an example of operation in this case. For example, if a low-speed network with limited usable network bandwidth, such as in an LTE (Long Term Evolution) environment, is used, the client accesses the server via the low-speed network (S321) and receives a map from the server. The SWLD is acquired as information (S322). On the other hand, if a high-speed network with sufficient network bandwidth is used, such as in a Wi-Fi (registered trademark) environment, the client accesses the server via the high-speed network (S323) and obtains the WLD from the server. (S324). Thereby, the client can acquire appropriate map information according to the network band of the client.

Specifically, the client receives SWLD via LTE when outdoors, and obtains WLD via Wi-Fi (registered trademark) when entering indoors such as a facility. This allows the client to obtain more detailed indoor map information.

In this way, the client may request WLD or SWLD from the server depending on the band of the network it uses. Alternatively, the client may transmit information indicating the band of the network used by the client to the server, and the server may transmit data (WLD or SWLD) suitable for the client in accordance with the information. Alternatively, the server may determine the network band of the client and transmit data (WLD or SWLD) suitable for the client.

Furthermore, it may be possible to switch whether to receive WLD or SWLD depending on the moving speed. FIG. 15 is a diagram showing an example of operation in this case. For example, if the client is moving at high speed (S331), the client receives SWLD from the server (S332). On the other hand, if the client is moving at low speed (S333), the client receives WLD from the server (S334). This allows the client to obtain map information that matches the speed while suppressing network bandwidth. Specifically, by receiving SWLD with a small amount of data while driving on an expressway, the client can update rough map information at an appropriate speed. On the other hand, the client can obtain more detailed map information by receiving WLD while driving on a general road.

In this way, the client may request WLD or SWLD from the server depending on its own movement speed. Alternatively, the client may transmit information indicating its own moving speed to the server, and the server may transmit data (WLD or SWLD) suitable for the client in accordance with the information. Alternatively, the server may determine the moving speed of the client and send data (WLD or SWLD) appropriate for the client.

Alternatively, the client may first obtain the SWLD from the server, and then obtain the WLD of an important area. For example, when acquiring map data, a client first acquires rough map information using SWLD, then narrows down areas where many features such as buildings, signs, or people appear, and uses WLD for the narrowed down area. Get it later. This makes it possible for the client to acquire detailed information on the necessary area while suppressing the amount of data received from the server.

Further, the server may create a separate SWLD for each object from the WLD, and the client may receive each SWLD depending on the purpose. This allows the network bandwidth to be suppressed. For example, the server recognizes a person or a car in advance from the WLD and creates a person's SWLD and a car's SWLD. The client receives the person's SWLD when it wants to acquire information about people around it, and the car's SWLD when it wants to acquire information about a car. Further, the type of SWLD may be distinguished by information (flag, type, etc.) added to the header or the like.

Next, the configuration and operation flow of the three-dimensional data encoding device (eg, server) according to this embodiment will be explained. FIG. 16 is a block diagram of a three-dimensional data encoding device 400 according to this embodiment. FIG. 17 is a flowchart of three-dimensional data encoding processing by the three-dimensional data encoding apparatus 400.

A three-dimensional data encoding device 400 shown in FIG. 16 generates encoded three-dimensional data 413 and 414, which are encoded streams, by encoding input three-dimensional data 411. Here, the encoded three-dimensional data 413 is encoded three-dimensional data that corresponds to WLD, and the encoded three-dimensional data 414 is encoded three-dimensional data that corresponds to SWLD. This three-dimensional data encoding device 400 includes an acquisition section 401, an encoding area determination section 402, an SWLD extraction section 403, a WLD encoding section 404, and an SWLD encoding section 405.

As shown in FIG. 17, first, the acquisition unit 401 acquires input three-dimensional data 411, which is point group data in a three-dimensional space (S401).

Next, the encoding region determining unit 402 determines a spatial region to be encoded based on the spatial region in which point cloud data exists (S402).

Next, the SWLD extraction unit 403 defines the spatial region to be encoded as a WLD, and calculates a feature amount from each VXL included in the WLD. Then, the SWLD extraction unit 403 extracts VXL whose feature amount is equal to or higher than a predetermined threshold, defines the extracted VXL as FVXL, and generates extracted three-dimensional data 412 by adding the FVXL to SWLD. (S403). In other words, the extracted three-dimensional data 412 whose feature amount is equal to or greater than the threshold is extracted from the input three-dimensional data 411.

Next, the WLD encoding unit 404 generates encoded three-dimensional data 413 corresponding to WLD by encoding input three-dimensional data 411 corresponding to WLD (S404). At this time, the WLD encoding unit 404 adds information to the header of the encoded three-dimensional data 413 to distinguish that the encoded three-dimensional data 413 is a stream including WLD.

Further, the SWLD encoding unit 405 generates encoded three-dimensional data 414 corresponding to SWLD by encoding the extracted three-dimensional data 412 corresponding to SWLD (S405). At this time, the SWLD encoding unit 405 adds information to the header of the encoded three-dimensional data 414 to distinguish that the encoded three-dimensional data 414 is a stream including SWLD.

Note that the processing order of the process of generating encoded three-dimensional data 413 and the process of generating encoded three-dimensional data 414 may be reversed to the above. Further, some or all of these processes may be performed in parallel.

As information added to the headers of the encoded three-dimensional data 413 and 414, for example, a parameter called "world_type" is defined. When world_type=0, it indicates that the stream includes WLD, and when world_type=1, it indicates that the stream includes SWLD. Furthermore, when defining many other types, the assigned numerical value may be increased, such as world_type=2. Furthermore, one of the encoded three-dimensional data 413 and 414 may include a specific flag. For example, a flag indicating that the stream includes SWLD may be added to the encoded three-dimensional data 414. In this case, the decoding device can determine whether the stream includes WLD or SWLD based on the presence or absence of the flag.

Furthermore, the encoding method used by WLD encoding section 404 to encode WLD and the encoding method used by SWLD encoding section 405 to encode SWLD may be different.

For example, in SWLD, data is thinned out, so the correlation with surrounding data may be lower than in WLD. Therefore, in the encoding method used for SWLD, inter prediction among intra prediction and inter prediction may be given priority over the encoding method used for WLD.

Further, the encoding method used for SWLD and the encoding method used for WLD may differ in the representation method of the three-dimensional position. For example, in SWLD, the three-dimensional position of FVXL may be expressed by three-dimensional coordinates, and in WLD, the three-dimensional position may be expressed by an octree, which will be described later, or vice versa.

Further, the SWLD encoding unit 405 performs encoding so that the data size of the SWLD encoded three-dimensional data 414 is smaller than the data size of the WLD encoded three-dimensional data 413. For example, as described above, SWLD may have lower correlation between data than WLD. As a result, the encoding efficiency may decrease, and the data size of the encoded three-dimensional data 414 may become larger than the data size of the WLD encoded three-dimensional data 413. Therefore, if the data size of the obtained encoded three-dimensional data 414 is larger than the data size of the WLD encoded three-dimensional data 413, the SWLD encoding unit 405 performs re-encoding to reduce the data size. The encoded three-dimensional data 414 with reduced values is regenerated.

For example, the SWLD extraction unit 403 regenerates extracted three-dimensional data 412 with a reduced number of extracted feature points, and the SWLD encoding unit 405 encodes the extracted three-dimensional data 412. Alternatively, the degree of quantization in SWLD encoding section 405 may be made coarser. For example, in the octree structure described below, the degree of quantization can be made coarser by rounding off the data at the lowest layer.

Further, if the data size of the SWLD encoded three-dimensional data 414 cannot be made smaller than the data size of the WLD encoded three-dimensional data 413, the SWLD encoding unit 405 does not generate the SWLD encoded three-dimensional data 414. It's okay. Alternatively, WLD encoded three-dimensional data 413 may be copied to SWLD encoded three-dimensional data 414. That is, the WLD encoded three-dimensional data 413 may be used as is as the SWLD encoded three-dimensional data 414.

Next, the configuration and operation flow of the three-dimensional data decoding device (eg, client) according to this embodiment will be explained. FIG. 18 is a block diagram of a three-dimensional data decoding device 500 according to this embodiment. FIG. 19 is a flowchart of three-dimensional data decoding processing by the three-dimensional data decoding device 500.

A three-dimensional data decoding device 500 shown in FIG. 18 generates decoded three-dimensional data 512 or 513 by decoding encoded three-dimensional data 511. Here, the encoded three-dimensional data 511 is, for example, encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding device 400.

This three-dimensional data decoding device 500 includes an acquisition section 501, a header analysis section 502, a WLD decoding section 503, and an SWLD decoding section 504.

As shown in FIG. 19, first, the acquisition unit 501 acquires encoded three-dimensional data 511 (S501). Next, the header analysis unit 502 analyzes the header of the encoded three-dimensional data 511 and determines whether the encoded three-dimensional data 511 is a stream including WLD or a stream including SWLD (S502). For example, the above-mentioned world_type parameter is referred to for determination.

If the encoded three-dimensional data 511 is a stream containing WLD (Yes in S503), the WLD decoding unit 503 generates WLD decoded three-dimensional data 512 by decoding the encoded three-dimensional data 511 (S504). . On the other hand, if the encoded three-dimensional data 511 is a stream including SWLD (No in S503), the SWLD decoding unit 504 generates decoded three-dimensional data 513 of SWLD by decoding the encoded three-dimensional data 511 ( S505).

Further, similarly to the encoding device, the decoding method used by WLD decoding section 503 to decode WLD and the decoding method used by SWLD decoding section 504 to decode SWLD may be different. For example, in the decoding method used for SWLD, inter prediction may be given priority over intra prediction and inter prediction over the decoding method used for WLD.

Further, the decoding method used for SWLD and the decoding method used for WLD may differ in the representation method of the three-dimensional position. For example, in SWLD, the three-dimensional position of FVXL may be expressed by three-dimensional coordinates, and in WLD, the three-dimensional position may be expressed by an octree, which will be described later, or vice versa.

Next, octree representation, which is a method of representing a three-dimensional position, will be explained. The VXL data included in the three-dimensional data is converted into an octree structure and then encoded. FIG. 20 is a diagram showing an example of VXL of WLD. FIG. 21 is a diagram showing an octree structure of the WLD shown in FIG. 20. In the example shown in FIG. 20, there are three VXLs 1 to 3, which are VXLs that include point groups (hereinafter referred to as effective VXLs). As shown in FIG. 21, the 8-branch tree structure is composed of nodes and leaves. Each node has a maximum of eight nodes or leaves. Each leaf has VXL information. Here, among the leaves shown in FIG. 21, leaves 1, 2, and 3 represent VXL1, VXL2, and VXL3 shown in FIG. 20, respectively.

Specifically, each node and leaf corresponds to a three-dimensional position. Node 1 corresponds to the entire block shown in FIG. The block corresponding to node 1 is divided into eight blocks, and among the eight blocks, the block containing valid VXL is set as a node, and the other blocks are set as leaves. The block corresponding to the node is further divided into eight nodes or leaves, and this process is repeated for each layer of the tree structure. Furthermore, all the blocks in the lowest layer are set as leaves.

Further, FIG. 22 is a diagram showing an example of SWLD generated from the WLD shown in FIG. 20. As a result of feature extraction, VXL1 and VXL2 shown in FIG. 20 are determined to be FVXL1 and FVXL2, and are added to SWLD. On the other hand, VXL3 is not determined to be FVXL and is not included in SWLD. FIG. 23 is a diagram showing the 8-branch tree structure of SWLD shown in FIG. 22. In the 8-branch tree structure shown in FIG. 23, leaf 3 corresponding to VXL3 shown in FIG. 21 has been deleted. As a result, node 3 shown in FIG. 21 no longer has a valid VXL and has been changed to a leaf. In this way, the number of leaves in SWLD is generally smaller than the number of leaves in WLD, and the coded three-dimensional data in SWLD is also smaller than the coded three-dimensional data in WLD.

Modifications of this embodiment will be described below.

For example, when a client such as an in-vehicle device performs self-position estimation, it receives SWLD from the server, performs self-position estimation using SWLD, and when detecting an obstacle, uses a distance sensor such as a range finder, a stereo Obstacle detection may be performed based on surrounding three-dimensional information acquired by the user using various methods such as a camera or a combination of multiple monocular cameras.

Further, in general, SWLD does not easily include VXL data in a flat area. Therefore, the server may maintain a sub-sampled world (subWLD) obtained by subsampling the WLD for static obstacle detection, and may send the SWLD and subWLD to the client. This allows the client to perform self-position estimation and obstacle detection while suppressing network bandwidth.

Furthermore, when a client draws three-dimensional map data at high speed, it may be more convenient for the map information to have a mesh structure. Therefore, the server may generate a mesh from the WLD and hold it in advance as a mesh world (MWLD). For example, a client receives a MWLD when it requires coarse three-dimensional drawing, and receives a WLD when it requires detailed three-dimensional drawing. Thereby, network bandwidth can be suppressed.

Further, among the VXLs, the server sets the VXL whose feature amount is equal to or greater than the threshold value as the FVXL, but the FVXL may be calculated using a different method. For example, the server determines that VXL, VLM, SPC, or GOS that constitutes a traffic light or intersection is necessary for self-position estimation, driving assistance, or automated driving, and includes it in the SWLD as FVXL, FVLM, FSPC, or FGOS. It doesn't matter if you do it like this. Further, the above judgment may be made manually. Note that the FVXL etc. obtained by the above method may be added to the FVXL etc. set based on the feature amount. That is, the SWLD extraction unit 403 may further extract data corresponding to an object having a predetermined attribute from the input three-dimensional data 411 as extracted three-dimensional data 412.

Further, it is also possible to label the information necessary for those uses separately from the feature amounts. Further, the server may separately hold FVXL necessary for self-position estimation at traffic lights or intersections, driving assistance, automatic driving, etc. as an upper layer of SWLD (for example, lane world).

Further, the server may also add attributes to the VXL in the WLD for each random access unit or for each predetermined unit. The attributes include, for example, information indicating whether the attribute is necessary or unnecessary for self-position estimation, or information indicating whether the attribute is important as traffic information such as a signal or an intersection. Further, the attribute may include a correspondence relationship with a Feature (an intersection, a road, etc.) in lane information (GDF: Geographic Data Files, etc.).

Further, the following method may be used as a method for updating the WLD or SWLD.

Updated information indicating changes in people, construction, or tree lines (for trucks) is uploaded to the server as a point cloud or metadata. The server updates the WLD based on the upload, and then updates the SWLD using the updated WLD.

Additionally, if the client detects a mismatch between the 3D information it generated during self-position estimation and the 3D information received from the server, the client can send the 3D information it generated to the server along with an update notification. good. In this case, the server uses WLD to update SWLD. If the SWLD is not updated, the server determines that the WLD itself is outdated.

In addition, information that distinguishes between WLD and SWLD is added as header information of the encoded stream, but if there are many types of worlds, such as mesh worlds or lane worlds, it is necessary to distinguish between them. The information may be added to the header information. Furthermore, if there are many SWLDs with different feature amounts, information that distinguishes them from each other may be added to the header information.

Furthermore, although the SWLD is configured of FVXL, it may include VXL that is not determined to be FVXL. For example, the SWLD may include adjacent VXLs used when calculating the feature amount of the FVXL. Thereby, even if feature information is not added to each FVXL of the SWLD, the client can calculate the feature of the FVXL when receiving the SWLD. In this case, the SWLD may include information for distinguishing whether each VXL is an FVXL or a VXL.

As described above, the three-dimensional data encoding device 400 extracts extracted three-dimensional data 412 (second three-dimensional data) whose feature amount is equal to or greater than a threshold from input three-dimensional data 411 (first three-dimensional data), and By encoding the three-dimensional data 412, encoded three-dimensional data 414 (first encoded three-dimensional data) is generated.

According to this, the three-dimensional data encoding device 400 generates encoded three-dimensional data 414 by encoding data whose feature amount is equal to or greater than the threshold value. Thereby, the amount of data can be reduced compared to the case where the input three-dimensional data 411 is encoded as is. Therefore, the three-dimensional data encoding device 400 can reduce the amount of data to be transmitted.

Furthermore, the three-dimensional data encoding device 400 further encodes the input three-dimensional data 411 to generate encoded three-dimensional data 413 (second encoded three-dimensional data).

According to this, the three-dimensional data encoding device 400 can selectively transmit the encoded three-dimensional data 413 and the encoded three-dimensional data 414 depending on the intended use, for example.

Further, the extracted three-dimensional data 412 is encoded by a first encoding method, and the input three-dimensional data 411 is encoded by a second encoding method different from the first encoding method.

According to this, the three-dimensional data encoding device 400 can use encoding methods suitable for the input three-dimensional data 411 and the extracted three-dimensional data 412, respectively.

Furthermore, in the first encoding method, inter prediction is given priority over intra prediction and inter prediction over the second encoding method.

According to this, the three-dimensional data encoding device 400 can increase the priority of inter prediction for the extracted three-dimensional data 412 in which the correlation between adjacent data tends to be low.

Furthermore, the first encoding method and the second encoding method differ in the method of expressing the three-dimensional position. For example, in the second encoding method, a three-dimensional position is expressed by an octree, and in the first encoding method, a three-dimensional position is expressed by three-dimensional coordinates.

According to this, the three-dimensional data encoding device 400 can use a more suitable three-dimensional position expression method for three-dimensional data having different numbers of data (number of VXL or FVXL).

In addition, at least one of the encoded three-dimensional data 413 and 414 is encoded three-dimensional data obtained by encoding the input three-dimensional data 411, or the input three-dimensional data 411 It includes an identifier indicating whether the data is encoded three-dimensional data obtained by encoding a part of the data. That is, the identifier indicates whether the encoded three-dimensional data is WLD encoded three-dimensional data 413 or SWLD encoded three-dimensional data 414.

According to this, the decoding device can easily determine whether the acquired encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414.

Furthermore, the three-dimensional data encoding device 400 encodes the extracted three-dimensional data 412 such that the data amount of the encoded three-dimensional data 414 is smaller than the data amount of the encoded three-dimensional data 413.

According to this, the three-dimensional data encoding device 400 can make the data amount of the encoded three-dimensional data 414 smaller than the data amount of the encoded three-dimensional data 413.

Furthermore, the three-dimensional data encoding device 400 further extracts data corresponding to an object having a predetermined attribute from the input three-dimensional data 411 as extracted three-dimensional data 412. For example, an object having a predetermined attribute is an object necessary for self-position estimation, driving assistance, automatic driving, etc., such as a traffic light or an intersection.

According to this, the three-dimensional data encoding device 400 can generate encoded three-dimensional data 414 including data required by the decoding device.

Furthermore, the three-dimensional data encoding device 400 (server) further transmits one of the encoded three-dimensional data 413 and 414 to the client depending on the state of the client.

According to this, the three-dimensional data encoding device 400 can transmit appropriate data according to the state of the client.

Further, the client status includes the client's communication status (for example, network band) or the client's movement speed.

Furthermore, the three-dimensional data encoding device 400 further transmits one of the encoded three-dimensional data 413 and 414 to the client in response to a request from the client.

According to this, the three-dimensional data encoding device 400 can transmit appropriate data in response to a client's request.

Furthermore, the three-dimensional data decoding device 500 according to the present embodiment decodes the encoded three-dimensional data 413 or 414 generated by the three-dimensional data encoding device 400 described above.

In other words, the three-dimensional data decoding device 500 first decodes the encoded three-dimensional data 414 obtained by encoding the extracted three-dimensional data 412 whose feature quantity extracted from the input three-dimensional data 411 is equal to or higher than the threshold value. Decrypt using the following method. Furthermore, the three-dimensional data decoding device 500 decodes the encoded three-dimensional data 413 obtained by encoding the input three-dimensional data 411 using a second decoding method different from the first decoding method.

According to this, the three-dimensional data decoding device 500 selects the encoded three-dimensional data 414 and the encoded three-dimensional data 413, which are encoded data whose feature amount is equal to or higher than the threshold value, depending on the intended use, for example. can be received in a timely manner. Thereby, the three-dimensional data decoding device 500 can reduce the amount of data to be transmitted. Further, the three-dimensional data decoding device 500 can use a decoding method suitable for each of the input three-dimensional data 411 and the extracted three-dimensional data 412.

Furthermore, in the first decoding method, inter prediction is given priority over intra prediction and inter prediction over the second decoding method.

According to this, the three-dimensional data decoding device 500 can increase the priority of inter prediction for extracted three-dimensional data in which the correlation between adjacent data tends to be low.

Furthermore, the first decoding method and the second decoding method differ in the representation method of the three-dimensional position. For example, in the second decoding method, a three-dimensional position is expressed by an octree, and in the first decoding method, a three-dimensional position is expressed by three-dimensional coordinates.

According to this, the three-dimensional data decoding device 500 can use a more suitable three-dimensional position representation method for three-dimensional data having different numbers of data (number of VXL or FVXL).

In addition, at least one of the encoded three-dimensional data 413 and 414 is encoded three-dimensional data obtained by encoding the input three-dimensional data 411, or the input three-dimensional data 411 It includes an identifier indicating whether the data is encoded three-dimensional data obtained by encoding a part of the data. Three-dimensional data decoding device 500 identifies encoded three-dimensional data 413 and 414 by referring to the identifiers.

According to this, the three-dimensional data decoding device 500 can easily determine whether the acquired encoded three-dimensional data is the encoded three-dimensional data 413 or the encoded three-dimensional data 414.

Furthermore, the three-dimensional data decoding device 500 further notifies the server of the state of the client (the three-dimensional data decoding device 500). The three-dimensional data decoding device 500 receives one of the encoded three-dimensional data 413 and 414 transmitted from the server, depending on the state of the client.

According to this, the three-dimensional data decoding device 500 can receive appropriate data depending on the state of the client.

Further, the client status includes the client's communication status (for example, network band) or the client's movement speed.

Furthermore, the three-dimensional data decoding device 500 further requests one of the encoded three-dimensional data 413 and 414 from the server, and in response to the request, one of the encoded three-dimensional data 413 and 414 transmitted from the server. Receive.

According to this, the three-dimensional data decoding device 500 can receive data appropriate for the purpose.

(Embodiment 3)
In this embodiment, a method for transmitting and receiving three-dimensional data between vehicles will be described.

FIG. 24 is a schematic diagram showing how three-dimensional data 607 is transmitted and received between own vehicle 600 and surrounding vehicles 601.

When acquiring three-dimensional data using a sensor mounted on the own vehicle 600 (a distance sensor such as a range finder, a stereo camera, a combination of multiple monocular cameras, etc.), the sensor of the own vehicle 600 may be affected by obstacles such as surrounding vehicles 601. A region (hereinafter referred to as an occlusion region 604) occurs within the detection range 602 but in which three-dimensional data cannot be created. Further, as the space for acquiring three-dimensional data becomes larger, the accuracy of autonomous operation increases, but the sensor detection range of only the host vehicle 600 is limited.

A sensor detection range 602 of the own vehicle 600 includes an area 603 where three-dimensional data can be acquired and an occlusion area 604. The range in which the own vehicle 600 wants to acquire three-dimensional data includes the sensor detection range 602 of the own vehicle 600 and other areas. Further, the sensor detection range 605 of the surrounding vehicle 601 includes an occlusion region 604 and a region 606 that is not included in the sensor detection range 602 of the own vehicle 600.

The surrounding vehicle 601 transmits information detected by the surrounding vehicle 601 to the own vehicle 600. The own vehicle 600 acquires three-dimensional data 607 of the occlusion area 604 and the area 606 other than the sensor detection range 602 of the own vehicle 600 by acquiring information detected by a surrounding vehicle 601 such as a vehicle in front. becomes possible. The own vehicle 600 uses the information acquired by the surrounding vehicles 601 to complement the three-dimensional data of the occlusion area 604 and the area 606 outside the sensor detection range.

Applications of three-dimensional data in autonomous operation of vehicles or robots include self-position estimation, sensing of surrounding conditions, or both. For example, three-dimensional data generated by the own vehicle 600 based on sensor information of the own vehicle 600 is used for self-position estimation. In addition to the three-dimensional data generated by the host vehicle 600, three-dimensional data acquired from the surrounding vehicles 601 is also used to detect the surrounding situation.

The surrounding vehicle 601 that transmits the three-dimensional data 607 to the own vehicle 600 may be determined according to the state of the own vehicle 600. For example, the surrounding vehicle 601 is a vehicle in front when the vehicle 600 is traveling straight, an oncoming vehicle when the vehicle 600 is turning right, and a vehicle behind the vehicle when the vehicle 600 is reversing. Alternatively, the driver of the own vehicle 600 may directly specify the surrounding vehicle 601 to which the three-dimensional data 607 is to be transmitted to the own vehicle 600.

Further, the own vehicle 600 may search for nearby vehicles 601 that are included in the space from which the three-dimensional data 607 is desired and that possess three-dimensional data of an area that cannot be acquired by the own vehicle 600. The area that cannot be acquired by the own vehicle 600 includes an occlusion area 604 or an area 606 outside the sensor detection range 602.

Furthermore, the own vehicle 600 may identify the occlusion area 604 based on the sensor information of the own vehicle 600. For example, the own vehicle 600 identifies an area included within the sensor detection range 602 of the own vehicle 600 where three-dimensional data cannot be created as the occlusion area 604.

An example of the operation when the vehicle in front transmits the three-dimensional data 607 will be described below. FIG. 25 is a diagram showing an example of three-dimensional data transmitted in this case.

As shown in FIG. 25, the three-dimensional data 607 transmitted from the vehicle in front is, for example, a point cloud sparse world (SWLD). In other words, the vehicle in front creates three-dimensional WLD data (point cloud) from the information detected by the sensor of the vehicle in front, and extracts data whose feature amount is greater than a threshold value from the three-dimensional WLD data. Create three-dimensional data (point cloud). Then, the vehicle in front transmits the created three-dimensional data of the SWLD to its own vehicle 600.

The own vehicle 600 receives the SWLD and merges the received SWLD into the point cloud created by the own vehicle 600.

The transmitted SWLD has information on absolute coordinates (the position of the SWLD in the coordinate system of the three-dimensional map). The host vehicle 600 can realize the merging process by overwriting the point cloud generated by the host vehicle 600 based on these absolute coordinates.

The SWLD transmitted from the surrounding vehicle 601 is the SWLD of an area 606 outside the sensor detection range 602 of the own vehicle 600 and within the sensor detection range 605 of the surrounding vehicle 601, or the SWLD of the occlusion area 604 for the own vehicle 600, or Both SWLDs may be used. Further, the SWLD to be transmitted may be one of the above-mentioned SWLDs in an area that the surrounding vehicle 601 uses to detect the surrounding situation.

Further, the surrounding vehicle 601 may change the density of the transmitted point cloud depending on the communication available time based on the speed difference between the own vehicle 600 and the surrounding vehicle 601. For example, when the speed difference is large and the available communication time is short, the surrounding vehicle 601 can reduce the density (data amount) of the point cloud by extracting three-dimensional points with large feature amounts from the SWLD. good.

Furthermore, detection of surrounding conditions refers to determining the presence or absence of people, vehicles, and road construction equipment, identifying their types, and detecting their positions, moving directions, and moving speeds. .

Further, the own vehicle 600 may acquire braking information of the surrounding vehicle 601 instead of or in addition to the three-dimensional data 607 generated by the surrounding vehicle 601. Here, the braking information of the surrounding vehicle 601 is, for example, information indicating that the accelerator or brake of the surrounding vehicle 601 has been depressed, or the degree thereof.

Furthermore, in the point cloud generated by each vehicle, the three-dimensional space is subdivided into random access units in consideration of low-latency communication between vehicles. On the other hand, in the case of a three-dimensional map or the like which is map data downloaded from a server, the three-dimensional space is divided into larger random access units than in the case of inter-vehicle communication.

Data in areas that are likely to become occlusion areas, such as the area in front of the vehicle in front or the area behind the vehicle in rear, is divided into small random access units as data for low delay.

Since the front view becomes more important when driving at high speeds, each vehicle creates SWLDs with narrowed viewing angles in fine random access units when driving at high speeds.

If the SWLD created by the vehicle in front for transmission includes an area where the own vehicle 600 can obtain point clouds, the vehicle in front may reduce the amount of transmission by removing the point cloud in that area. good.

Next, the configuration and operation of the three-dimensional data creation device 620, which is the three-dimensional data receiving device according to this embodiment, will be explained.

FIG. 26 is a block diagram of a three-dimensional data creation device 620 according to this embodiment. This three-dimensional data creation device 620 is included in, for example, the own vehicle 600 described above, and combines the received second three-dimensional data 635 with the first three-dimensional data 632 created by the three-dimensional data creation device 620. , to create denser third three-dimensional data 636.

This three-dimensional data creation device 620 includes a three-dimensional data creation section 621, a required range determination section 622, a search section 623, a reception section 624, a decoding section 625, and a synthesis section 626. FIG. 27 is a flowchart showing the operation of the three-dimensional data creation device 620.

First, the three-dimensional data creation unit 621 creates first three-dimensional data 632 using sensor information 631 detected by a sensor included in the own vehicle 600 (S621). Next, the required range determination unit 622 determines a required range that is a three-dimensional spatial range in which data is insufficient in the created first three-dimensional data 632 (S622).

Next, the search unit 623 searches for surrounding vehicles 601 that have three-dimensional data of the requested range, and transmits requested range information 633 indicating the requested range to the surrounding vehicles 601 identified by the search (S623). Next, the receiving unit 624 receives encoded three-dimensional data 634, which is an encoded stream of the requested range, from the surrounding vehicle 601 (S624). Note that the search unit 623 may issue a request indiscriminately to all vehicles existing in a specific range, and may receive encoded three-dimensional data 634 from those that have responded. Furthermore, the search unit 623 may issue a request to an object such as a traffic light or a sign, and may receive encoded three-dimensional data 634 from the object, instead of being limited to a vehicle.

Next, the decoding unit 625 obtains second three-dimensional data 635 by decoding the received encoded three-dimensional data 634 (S625). Next, the combining unit 626 creates denser third three-dimensional data 636 by combining the first three-dimensional data 632 and the second three-dimensional data 635 (S626).

Next, the configuration and operation of the three-dimensional data transmitting device 640 according to this embodiment will be explained. FIG. 28 is a block diagram of the three-dimensional data transmitting device 640.

The three-dimensional data transmitting device 640 is, for example, included in the above-mentioned surrounding vehicle 601 and processes the fifth three-dimensional data 652 created by the surrounding vehicle 601 into sixth three-dimensional data 654 requested by the own vehicle 600, and The encoded three-dimensional data 634 is generated by encoding the three-dimensional data 654, and the encoded three-dimensional data 634 is transmitted to the host vehicle 600.

The three-dimensional data transmitting device 640 includes a three-dimensional data creating section 641, a receiving section 642, an extracting section 643, an encoding section 644, and a transmitting section 645. FIG. 29 is a flowchart showing the operation of the three-dimensional data transmitting device 640.

First, the three-dimensional data creation unit 641 creates fifth three-dimensional data 652 using sensor information 651 detected by a sensor included in the surrounding vehicle 601 (S641). Next, the receiving unit 642 receives the requested range information 633 transmitted from the own vehicle 600 (S642).

Next, the extraction unit 643 processes the fifth three-dimensional data 652 into sixth three-dimensional data 654 by extracting the three-dimensional data of the requested range indicated by the requested range information 633 from the fifth three-dimensional data 652. (S643). Next, the encoding unit 644 generates encoded three-dimensional data 634, which is an encoded stream, by encoding the sixth three-dimensional data 654 (S644). Then, the transmitter 645 transmits the encoded three-dimensional data 634 to the own vehicle 600 (S645).

Note that here, an example will be described in which the own vehicle 600 is equipped with the three-dimensional data generation device 620 and the surrounding vehicle 601 is equipped with the three-dimensional data transmission device 640. It may also have a function with the transmitting device 640.

The configuration and operation in the case where the three-dimensional data creation device 620 is a surrounding situation detection device that realizes processing for detecting the surrounding situation of the own vehicle 600 will be described below. FIG. 30 is a block diagram showing the configuration of the three-dimensional data creation device 620A in this case. In addition to the configuration of the three-dimensional data creation device 620 shown in FIG. 26, the three-dimensional data creation device 620A shown in FIG. Be prepared. Further, the three-dimensional data creation device 620A is included in the own vehicle 600.

FIG. 31 is a flowchart of the process of detecting the surrounding situation of the host vehicle 600 by the three-dimensional data creation device 620A.

First, the three-dimensional data creation unit 621 creates first three-dimensional data 632, which is a point cloud, using sensor information 631 of the detection range of the own vehicle 600 detected by a sensor included in the own vehicle 600 (S661). Note that the three-dimensional data creation device 620A may further perform self-position estimation using the sensor information 631.

Next, the detection area determining unit 627 determines a detection target range, which is a spatial area in which the surrounding situation is to be detected (S662). For example, the detection area determining unit 627 calculates an area necessary for detecting the surrounding situation to safely perform autonomous operation, depending on the autonomous operation (automatic driving) situation such as the traveling direction and speed of the own vehicle 600, The area is determined as the detection target range.

Next, the required range determining unit 622 determines the occlusion area 604 and a spatial area that is outside the detection range of the sensor of the own vehicle 600 but is necessary for detecting the surrounding situation as the required range (S663).

If the requested range determined in step S663 exists (Yes in S664), the search unit 623 searches for nearby vehicles that have information regarding the requested range. For example, the search unit 623 may inquire of nearby vehicles whether they have information regarding the requested range, or whether the nearby vehicle owns information regarding the requested range based on the position of the requested range and the surrounding vehicle. may be judged. Next, the search unit 623 transmits a request signal 637 requesting transmission of three-dimensional data to the surrounding vehicle 601 identified through the search. After receiving the permission signal indicating that the request signal 637 is accepted, which is transmitted from the nearby vehicle 601, the search unit 623 transmits request range information 633 indicating the requested range to the nearby vehicle 601 (S665). .

Next, the receiving unit 624 detects a transmission notification of transmission data 638, which is information regarding the request range, and receives the transmission data 638 (S666).

Note that the three-dimensional data creation device 620A indiscriminately issues requests to all vehicles existing in a specific range without searching for the other party to whom the request is sent, and receives a response if it has information regarding the requested range. The transmission data 638 may be received from the other party. Furthermore, the search unit 623 may issue a request to an object such as a traffic light or a sign, and may receive the transmission data 638 from the object, without being limited to a vehicle.

Furthermore, the transmission data 638 includes at least one of encoded three-dimensional data 634 generated by the surrounding vehicle 601 in which the three-dimensional data of the requested range is encoded, and a surrounding situation detection result 639 of the requested range. The surrounding situation detection result 639 indicates the positions, moving directions, moving speeds, etc. of people and vehicles detected by the surrounding vehicle 601. Further, the transmission data 638 may include information indicating the position, movement, etc. of the surrounding vehicle 601. For example, the transmission data 638 may include braking information of surrounding vehicles 601.

If the encoded three-dimensional data 634 is included in the received transmission data 638 (Yes in S667), the decoding unit 625 obtains the second three-dimensional data 635 of SWLD by decoding the encoded three-dimensional data 634. (S668). That is, the second three-dimensional data 635 is three-dimensional data (SWLD) generated by extracting data whose feature amount is equal to or greater than the threshold value from the fourth three-dimensional data (WLD).

Next, the combining unit 626 generates third three-dimensional data 636 by combining the first three-dimensional data 632 and the second three-dimensional data 635 (S669).

Next, the surrounding situation detection unit 628 detects the surrounding situation of the host vehicle 600 using the third three-dimensional data 636, which is a point cloud of a spatial region necessary for surrounding situation detection (S670). Note that when the received transmission data 638 includes the surrounding situation detection result 639, the surrounding situation detection unit 628 uses the surrounding situation detection result 639 in addition to the third three-dimensional data 636 to detect the own vehicle 600. Detect the surrounding situation. Further, when the received transmission data 638 includes braking information of the surrounding vehicle 601, the surrounding situation detection unit 628 uses the braking information in addition to the third three-dimensional data 636 to detect the surroundings of the own vehicle 600. Detect the situation.

Next, the autonomous operation control unit 629 controls the autonomous operation (automatic driving) of the host vehicle 600 based on the surrounding situation detection result by the surrounding situation detection unit 628 (S671). Note that the surrounding situation detection results may be presented to the driver through a UI (user interface) or the like.

On the other hand, if the requested range does not exist in step S663 (No in S664), that is, if information on all the spatial areas necessary for detecting the surrounding situation can be created based on the sensor information 631, the surrounding situation detection unit 628 , the surrounding situation of the host vehicle 600 is detected using the first three-dimensional data 632, which is a point cloud of a spatial region necessary for detecting the surrounding situation (S672). Then, the autonomous operation control unit 629 controls the autonomous operation (automatic driving) of the host vehicle 600 based on the surrounding situation detection result by the surrounding situation detection unit 628 (S671).

Further, if the received transmission data 638 does not include encoded three-dimensional data 634 (No in S667), that is, if the transmission data 638 includes only the surrounding situation detection result 639 of the surrounding vehicle 601 or braking information. The surrounding situation detection unit 628 detects the surrounding situation of the own vehicle 600 using the first three-dimensional data 632 and the surrounding situation detection result 639 or the braking information (S673). Then, the autonomous operation control unit 629 controls the autonomous operation (automatic driving) of the host vehicle 600 based on the surrounding situation detection result by the surrounding situation detection unit 628 (S671).

Next, a three-dimensional data transmitting device 640A that transmits transmission data 638 to the three-dimensional data creating device 620A will be described. FIG. 32 is a block diagram of this three-dimensional data transmitting device 640A.

In addition to the configuration of the three-dimensional data transmitting device 640 shown in FIG. 28, the three-dimensional data transmitting device 640A shown in FIG. 32 further includes a transmission permission determining section 646. Further, the three-dimensional data transmitting device 640A is included in the surrounding vehicle 601.

FIG. 33 is a flowchart showing an example of the operation of the three-dimensional data transmitting device 640A. First, the three-dimensional data creation unit 641 creates fifth three-dimensional data 652 using sensor information 651 detected by a sensor included in the surrounding vehicle 601 (S681).

Next, the receiving unit 642 receives a request signal 637 requesting a three-dimensional data transmission request from the host vehicle 600 (S682). Next, the transmission permission determining unit 646 determines whether to comply with the request indicated by the request signal 637 (S683). For example, the transmission permission determining unit 646 determines whether to comply with the request based on the content set in advance by the user. Note that the receiving unit 642 may first receive a request from the other party, such as the request range, and the transmission permission determining unit 646 may determine whether to comply with the request based on the contents. For example, the transmission permission determining unit 646 determines to accept the request when it has the three-dimensional data within the requested range, and determines not to accept the request when it does not possess the three-dimensional data within the requested range. You may.

When responding to the request (Yes in S683), the three-dimensional data transmitting device 640A transmits a permission signal to the own vehicle 600, and the receiving unit 642 receives the requested range information 633 indicating the requested range (S684). Next, the extraction unit 643 cuts out a point cloud in the requested range from the fifth three-dimensional data 652 which is a point cloud, and creates transmission data 638 including sixth three-dimensional data 654 which is the SWLD of the cut out point cloud ( S685).

In other words, the three-dimensional data transmitting device 640A creates the seventh three-dimensional data (WLD) from the sensor information 651, and extracts data whose feature amount is equal to or higher than the threshold value from the seventh three-dimensional data (WLD). Original data 652 (SWLD) is created. Note that the three-dimensional data creation unit 641 may create the three-dimensional data of the SWLD in advance, and the extraction unit 643 may extract the three-dimensional data of the SWLD within the requested range from the three-dimensional data of the SWLD. However, the three-dimensional data of the SWLD in the requested range may be generated from the three-dimensional data of the WLD in the requested range.

Further, the transmission data 638 may include a surrounding situation detection result 639 of the requested range by the surrounding vehicle 601 and braking information of the surrounding vehicle 601. Further, the transmission data 638 may not include the sixth three-dimensional data 654 and may include only at least one of the surrounding situation detection result 639 of the requested range by the surrounding vehicle 601 and the braking information of the surrounding vehicle 601.

When the transmission data 638 includes the sixth three-dimensional data 654 (Yes in S686), the encoding unit 644 generates encoded three-dimensional data 634 by encoding the sixth three-dimensional data 654 (S687).

Then, the transmitter 645 transmits transmission data 638 including encoded three-dimensional data 634 to the own vehicle 600 (S688).

On the other hand, if the transmission data 638 does not include the sixth three-dimensional data 654 (No in S686), the transmission unit 645 transmits at least the surrounding situation detection result 639 of the requested range by the surrounding vehicle 601 and the braking information of the surrounding vehicle 601. Transmission data 638 including one is transmitted to own vehicle 600 (S688).

Modifications of this embodiment will be described below.

For example, the information transmitted from the surrounding vehicle 601 may not be three-dimensional data created by the surrounding vehicle or the surrounding situation detection result, but may be accurate feature point information of the surrounding vehicle 601 itself. The host vehicle 600 uses the feature point information of the surrounding vehicle 601 to correct the feature point information of the preceding vehicle in the point cloud acquired by the host vehicle 600. Thereby, the own vehicle 600 can improve matching accuracy when estimating the own position.

Further, the feature point information of the vehicle in front is, for example, three-dimensional point information consisting of color information and coordinate information. Thereby, whether the sensor of own vehicle 600 is a laser sensor or a stereo camera, the feature point information of the vehicle in front can be used regardless of the type.

Note that the host vehicle 600 may use the SWLD point cloud not only during transmission but also when calculating the accuracy during self-position estimation. For example, if the sensor of the own vehicle 600 is an imaging device such as a stereo camera, the own vehicle 600 detects a two-dimensional point on an image taken by the camera, and estimates its own position using the two-dimensional point. Further, the own vehicle 600 creates a point cloud of surrounding objects at the same time as the own position. The host vehicle 600 reprojects the three-dimensional points of the SWLD therein onto a two-dimensional image, and evaluates the accuracy of self-position estimation based on the error between the detection point and the reprojection point on the two-dimensional image.

Further, when the sensor of the own vehicle 600 is a laser sensor such as LiDAR, the own vehicle 600 determines its own position based on the error calculated by Iterative Closest Point using the SWLD of the created point cloud and the SWLD of the three-dimensional map. Evaluate the accuracy of the estimation.

Further, when the communication condition via a base station or server such as 5G is poor, the host vehicle 600 may acquire a three-dimensional map from the surrounding vehicle 601.

Additionally, information from a distance that cannot be obtained from vehicles around the host vehicle 600 may be obtained through inter-vehicle communication. For example, the host vehicle 600 may obtain information on a traffic accident that occurred immediately after it occurred several hundred meters or several kilometers away, using a passing communication from an oncoming vehicle, or a relay method in which the information is sequentially transmitted to surrounding vehicles. At this time, the data format of the transmitted data is transmitted as upper layer meta information of the dynamic three-dimensional map.

Further, the detection result of the surrounding situation and the information detected by the host vehicle 600 may be presented to the user through a user interface. For example, this information can be presented by superimposing it on a car navigation screen or a front window.

Further, a vehicle that does not support automatic driving and has cruise control may discover a nearby vehicle that is running in automatic driving mode and track the nearby vehicle.

Furthermore, when self-position estimation is not possible due to the inability to obtain a three-dimensional map or too many occlusion areas, the own vehicle 600 may switch its operating mode from automatic driving mode to surrounding vehicle tracking mode. good.

Further, the vehicle being tracked may be equipped with a user interface that warns the user that the vehicle is being tracked and allows the user to specify whether or not to permit the tracking. At that time, a mechanism may be provided such as displaying an advertisement on the vehicle being tracked and paying incentives to the tracked party.

Furthermore, the transmitted information is basically SWLD, which is three-dimensional data, but may also be information according to the request settings set in the own vehicle 600 or the disclosure settings of the vehicle in front. For example, the information to be transmitted may be WLD, which is a dense point cloud, the detection result of the surrounding situation by the vehicle in front, or the braking information of the vehicle in front.

Further, the host vehicle 600 may receive the WLD, visualize the three-dimensional data of the WLD, and present the visualized three-dimensional data to the driver using the GUI. At this time, the host vehicle 600 may display the information in a color-coded manner so that the user can distinguish between the point cloud created by the host vehicle 600 and the received point cloud.

Furthermore, when presenting the information detected by the own vehicle 600 and the detection results of the surrounding vehicles 601 to the driver on the GUI, the own vehicle 600 provides a way for the user to distinguish between the information detected by the own vehicle 600 and the received detection results. Alternatively, the information may be presented using color coding or the like.

As described above, in the three-dimensional data creation device 620 according to the present embodiment, the three-dimensional data creation unit 621 creates the first three-dimensional data 632 from the sensor information 631 detected by the sensor. The receiving unit 624 receives encoded three-dimensional data 634 in which second three-dimensional data 635 is encoded. The decoding unit 625 obtains second three-dimensional data 635 by decoding the received encoded three-dimensional data 634. The combining unit 626 creates third three-dimensional data 636 by combining the first three-dimensional data 632 and the second three-dimensional data 635.

According to this, the three-dimensional data creation device 620 can create detailed third three-dimensional data 636 using the created first three-dimensional data 632 and the received second three-dimensional data 635.

Furthermore, by combining the first three-dimensional data 632 and the second three-dimensional data 635, the combining unit 626 generates third three-dimensional data 636 having a higher density than the first three-dimensional data 632 and the second three-dimensional data 635. Create.

Further, the second three-dimensional data 635 (for example, SWLD) is three-dimensional data generated by extracting data whose feature amount is equal to or greater than a threshold value from the fourth three-dimensional data (for example, WLD).

According to this, the three-dimensional data creation device 620 can reduce the amount of three-dimensional data to be transmitted.

Furthermore, the three-dimensional data creation device 620 further includes a search unit 623 that searches for a transmitter that is the source of the encoded three-dimensional data 634. The receiving unit 624 receives encoded three-dimensional data 634 from the searched transmitting device.

According to this, the three-dimensional data creation device 620 can identify, for example, a transmitting device that owns necessary three-dimensional data by searching.

Furthermore, the three-dimensional data creation device further includes a request range determination unit 622 that determines a request range that is a range of three-dimensional space for which three-dimensional data is requested. The search unit 623 transmits request range information 633 indicating the request range to the transmitting device. The second three-dimensional data 635 includes three-dimensional data within the requested range.

According to this, the three-dimensional data creation device 620 can receive necessary three-dimensional data and reduce the amount of three-dimensional data to be transmitted.

Further, the required range determination unit 622 determines a spatial range including the occlusion area 604 that cannot be detected by the sensor as the required range.

Furthermore, in the three-dimensional data transmitting device 640 according to the present embodiment, the three-dimensional data creation unit 641 creates fifth three-dimensional data 652 from the sensor information 651 detected by the sensor. The extraction unit 643 creates sixth three-dimensional data 654 by extracting a part of the fifth three-dimensional data 652. The encoding unit 644 generates encoded three-dimensional data 634 by encoding the sixth three-dimensional data 654. The transmitter 645 transmits encoded three-dimensional data 634.

According to this, the three-dimensional data transmitting device 640 can transmit the three-dimensional data created by itself to other devices, and can reduce the amount of three-dimensional data to be transmitted.

Further, the three-dimensional data creation unit 641 creates seventh three-dimensional data (for example, WLD) from the sensor information 651 detected by the sensor, and extracts data whose feature amount is equal to or higher than a threshold value from the seventh three-dimensional data. 5 Create three-dimensional data 652 (for example, SWLD).

According to this, the three-dimensional data transmitting device 640 can reduce the amount of three-dimensional data to be transmitted.

Furthermore, the three-dimensional data transmitting device 640 further includes a receiving unit 642 that receives request range information 633 indicating a request range, which is a three-dimensional space range for which three-dimensional data is requested, from the receiving device. The extraction unit 643 creates sixth three-dimensional data 654 by extracting three-dimensional data within the requested range from the fifth three-dimensional data 652 . The transmitting unit 645 transmits encoded three-dimensional data 634 to the receiving device.

According to this, the three-dimensional data transmitting device 640 can reduce the amount of three-dimensional data to be transmitted.

(Embodiment 4)
In this embodiment, the operation of an abnormal system in self-position estimation based on a three-dimensional map will be described.

Applications are expected to expand in the future, such as self-driving cars and autonomously moving moving objects such as robots and flying objects such as drones. As an example of means for realizing such autonomous movement, there is a method in which a moving object moves according to a map while estimating its own position within a three-dimensional map (self-position estimation).

Self-position estimation involves matching a 3D map with 3D information around the vehicle (hereinafter referred to as vehicle detection 3D data) acquired by a sensor such as a rangefinder (LiDAR, etc.) installed in the vehicle or a stereo camera. This can be achieved by estimating the vehicle's position within the three-dimensional map.

3D maps include not only 3D point clouds, such as the HD maps proposed by HERE, but also 2D map data such as road and intersection shape information, or real-time changes such as traffic jams and accidents. It may also include information that A three-dimensional map is constructed from multiple layers such as three-dimensional data, two-dimensional data, and metadata that changes in real time, and the device can also acquire or refer to only the necessary data.

The point cloud data may be the above-mentioned SWLD, or may include point cloud data that is not feature points. Further, data transmission and reception of point cloud data is performed on the basis of one or more random access units.

The following method can be used as a matching method for the three-dimensional map and the three-dimensional data detected by the own vehicle. For example, the device compares the shapes of point groups in each other's point clouds and determines that regions with high similarity between feature points are at the same location. Further, when the three-dimensional map is composed of SWLD, the device performs matching by comparing the feature points making up the SWLD and the three-dimensional feature points extracted from the own vehicle detection three-dimensional data.

In order to perform self-position estimation with high accuracy, (A) a 3D map and 3D vehicle detection data must be acquired, and (B) their accuracy must meet predetermined standards. This is necessary. However, in the following abnormal cases, (A) or (B) cannot be satisfied.

(1) 3D map cannot be obtained via communication.

(2) The 3D map does not exist, or the 3D map was obtained but is damaged.

(3) The generation accuracy of the three-dimensional data detected by the own vehicle is not sufficient because the sensor of the own vehicle is malfunctioning or due to bad weather.

Operations for dealing with these abnormal cases will be described below. Although the operation will be described below using a car as an example, the following method can be applied to any autonomously moving animal such as a robot or a drone.

Hereinafter, the configuration and operation of the three-dimensional information processing apparatus according to the present embodiment for dealing with abnormal cases in the three-dimensional map or self-vehicle detection three-dimensional data will be described. FIG. 34 is a block diagram showing a configuration example of a three-dimensional information processing device 700 according to this embodiment. FIG. 35 is a flowchart of a three-dimensional information processing method by the three-dimensional information processing apparatus 700.

The three-dimensional information processing device 700 is mounted, for example, on a moving object such as a car. As shown in FIG. 34, the three-dimensional information processing device 700 includes a three-dimensional map acquisition section 701, an own vehicle detection data acquisition section 702, an abnormality case determination section 703, a countermeasure action determination section 704, and an operation control section 705. Equipped with.

Note that the three-dimensional information processing device 700 includes a camera (not shown) for detecting structures or moving objects around the vehicle, such as a camera that acquires two-dimensional images, or a one-dimensional data sensor using ultrasonic waves or lasers. A two-dimensional or one-dimensional sensor may be provided. Furthermore, the three-dimensional information processing device 700 may include a communication unit (not shown) for acquiring a three-dimensional map through a mobile communication network such as 4G or 5G, or vehicle-to-vehicle communication or road-to-vehicle communication. .

As shown in FIG. 35, the three-dimensional map acquisition unit 701 acquires a three-dimensional map 711 near the travel route (S701). For example, the three-dimensional map acquisition unit 701 acquires the three-dimensional map 711 through a mobile communication network, vehicle-to-vehicle communication, or road-to-vehicle communication.

Next, the own vehicle detection data acquisition unit 702 acquires own vehicle detection three-dimensional data 712 based on the sensor information (S702). For example, the own vehicle detection data acquisition unit 702 generates own vehicle detection three-dimensional data 712 based on sensor information acquired by a sensor included in the own vehicle.

Next, the abnormal case determination unit 703 detects an abnormal case by performing a predetermined check on at least one of the acquired three-dimensional map 711 and the own vehicle detection three-dimensional data 712 (S703). That is, the abnormal case determination unit 703 determines whether at least one of the acquired three-dimensional map 711 and the own vehicle detection three-dimensional data 712 is abnormal.

If an abnormal case is detected in step S703 (Yes in S704), the countermeasure action determining unit 704 determines a countermeasure action for the abnormal case (S705). Next, the operation control unit 705 controls the operation of each processing unit, such as the three-dimensional map acquisition unit 701, that is necessary for implementing the countermeasure operation (S706).

On the other hand, if no abnormal case is detected in step S703 (No in S704), the three-dimensional information processing apparatus 700 ends the process.

Further, the three-dimensional information processing device 700 uses the three-dimensional map 711 and the own vehicle detection three-dimensional data 712 to estimate the self-position of the vehicle including the three-dimensional information processing device 700. Next, the three-dimensional information processing device 700 automatically drives the vehicle using the result of self-position estimation.

In this way, the three-dimensional information processing device 700 acquires map data (three-dimensional map 711) including first three-dimensional position information via the communication channel. For example, the first three-dimensional position information is encoded in units of subspaces having three-dimensional coordinate information, each of which is a collection of one or more subspaces, and each of which can be decoded independently. Contains access units. For example, the first three-dimensional position information is data (SWLD) in which feature points whose three-dimensional feature amount is equal to or greater than a predetermined threshold are encoded.

Furthermore, the three-dimensional information processing device 700 generates second three-dimensional position information (self-vehicle detection three-dimensional data 712) from information detected by the sensor. Next, the three-dimensional information processing device 700 performs an abnormality determination process on the first three-dimensional position information or the second three-dimensional position information, thereby determining the first three-dimensional position information or the second three-dimensional position information. Determine whether the three-dimensional position information is abnormal.

When it is determined that the first three-dimensional position information or the second three-dimensional position information is abnormal, the three-dimensional information processing apparatus 700 determines a countermeasure operation for the abnormality. Next, the three-dimensional information processing device 700 performs control necessary to implement the countermeasure action.

Thereby, the three-dimensional information processing device 700 can detect an abnormality in the first three-dimensional position information or the second three-dimensional position information, and can perform a countermeasure operation.

Hereinafter, a description will be given of a countermeasure operation for a case where the three-dimensional map 711 cannot be obtained via communication, which is abnormal case 1.

A three-dimensional map 711 is required for self-position estimation, but if the vehicle has not previously acquired a three-dimensional map 711 corresponding to the route to the destination, it is necessary to acquire the three-dimensional map 711 through communication. There is. However, the vehicle may not be able to acquire the three-dimensional map 711 on the travel route due to congestion of the communication channel or deterioration of the radio wave reception condition.

The abnormal case determination unit 703 checks whether three-dimensional maps 711 have been acquired for all sections on the route to the destination or for sections within a predetermined range from the current position, and if they have not been acquired, Determined as abnormal case 1. That is, the abnormal case determination unit 703 determines whether the three-dimensional map 711 (first three-dimensional position information) can be acquired via the communication channel, and if the three-dimensional map 711 cannot be acquired via the communication channel, It is determined that the three-dimensional map 711 is abnormal.

When it is determined that it is abnormal case 1, the countermeasure action determining unit 704 selects one of two types of countermeasure actions: (1) continue self-position estimation, and (2) stop self-position estimation. do.

First, a specific example of the countermeasure operation when (1) self-position estimation is continued will be described. If self-position estimation is to be continued, a three-dimensional map 711 on the route to the destination is required.

For example, the vehicle determines a location where a communication channel is available within the range where the three-dimensional map 711 has been acquired, moves to that location, and acquires the three-dimensional map 711. At this time, the vehicle may acquire all three-dimensional maps 711 up to the destination, or acquire three-dimensional maps 711 for each random access unit within the upper limit size that can be stored in the vehicle's memory or storage unit such as HDD. may be obtained.

In addition, the vehicle separately obtains the communication status on the route, and if it is predicted that the communication status on the route will be poor, the vehicle acquires the three-dimensional map of the section before reaching the section where the communication status is poor. The three-dimensional map 711 may be obtained in advance, or the three-dimensional map 711 of the maximum obtainable range may be obtained in advance. In other words, the three-dimensional information processing device 700 predicts whether the vehicle will enter an area where communication conditions are poor. When it is predicted that a vehicle will enter an area where communication conditions are poor, the three-dimensional information processing device 700 acquires a three-dimensional map 711 before the vehicle enters the area.

In addition, even if the vehicle specifies a random access unit that constitutes the minimum three-dimensional map 711 necessary for self-position estimation on the route, which has a narrower range than usual, and receives the specified random access unit, good. That is, when the three-dimensional information processing device 700 cannot acquire the three-dimensional map 711 (first three-dimensional position information) via the communication channel, the three-dimensional information processing device 700 acquires a third three-dimensional position having a narrower range than the first three-dimensional position information. Information may also be obtained via a communication channel.

Furthermore, in the case where the vehicle cannot access the distribution server of the 3D map 711, the 3D map 711 on the route to the destination, such as another vehicle traveling around the vehicle, has been acquired, and The three-dimensional map 711 may be acquired from a mobile object that can communicate with the own vehicle.

Next, a specific example of the countermeasure operation when (2) self-position estimation is stopped will be described. In this case, the three-dimensional map 711 on the route to the destination is unnecessary.

For example, the vehicle notifies the driver that functions such as automatic driving based on self-position estimation cannot be continued, and shifts the operating mode to a manual mode in which the driver drives the vehicle.

Normally, when estimating the vehicle's own position, autonomous driving is performed, although there are differences in the level of human intervention. On the other hand, the results of self-position estimation can also be used for navigation when a person is driving. Therefore, the result of self-position estimation does not necessarily have to be used for automatic driving.

In addition, if a normally used communication channel such as a mobile communication network such as 4G or 5G is not available, the vehicle may use another method such as road-to-vehicle Wi-Fi (registered trademark) or millimeter wave communication, or vehicle-to-vehicle communication. The user may check whether the three-dimensional map 711 can be acquired via the communication channel, and then switch the communication channel to be used to one that can acquire the three-dimensional map 711.

Further, if the vehicle cannot acquire the three-dimensional map 711, the vehicle may acquire a two-dimensional map and continue automatic driving using the two-dimensional map and the own vehicle detected three-dimensional data 712. That is, when the three-dimensional map 711 cannot be acquired via the communication channel, the three-dimensional information processing device 700 acquires map data (two-dimensional map) including two-dimensional position information via the communication channel, and acquires the two-dimensional position information. The self-position of the vehicle may be estimated using the self-vehicle detection three-dimensional data 712.

Specifically, the vehicle uses the two-dimensional map and the own-vehicle detection three-dimensional data 712 to estimate its own position, and uses the own-vehicle detected three-dimensional data to detect surrounding vehicles, pedestrians, obstacles, etc. 712 is used.

Here, map data such as an HD map includes a three-dimensional map 711 composed of a three-dimensional point cloud, etc., two-dimensional map data (two-dimensional map), and road shapes or intersections from the two-dimensional map data. It can include simplified map data that extracts characteristic information such as, and metadata representing real-time information such as traffic jams, accidents, or construction. For example, map data has a layer structure in which three-dimensional data (three-dimensional map 711), two-dimensional data (two-dimensional map), and metadata are arranged in order from the lower layer.

Here, two-dimensional data has a smaller data size than three-dimensional data. Therefore, even if the communication condition is poor, the vehicle may be able to obtain a two-dimensional map. Alternatively, the vehicle can collectively acquire a two-dimensional map of a wide range in a section with good communication conditions. Therefore, if the condition of the communication channel is poor and it is difficult to obtain the three-dimensional map 711, the vehicle may receive a layer including the two-dimensional map without receiving the three-dimensional map 711. Note that since the data size of metadata is small, for example, a vehicle always receives metadata regardless of the communication state.

There are, for example, the following two methods for estimating the self-position using the two-dimensional map and the self-vehicle detected three-dimensional data 712.

The first method is a method of matching two-dimensional special quantities. Specifically, the vehicle extracts a two-dimensional feature amount from the vehicle-detected three-dimensional data 712, and matches the extracted two-dimensional feature amount with a two-dimensional map.

For example, the vehicle projects the own vehicle detected three-dimensional data 712 onto the same plane as the two-dimensional map, and matches the two-dimensional data obtained with the two-dimensional map. Matching is performed using two-dimensional image features extracted from both.

When the 3D map 711 includes SWLD, the 3D map 711 may store the 3D feature values at the feature points in the 3D space as well as the 2D feature values on the same plane as the 2D map. good. For example, identification information is attached to two-dimensional feature amounts. Alternatively, the two-dimensional feature amount is stored in a layer separate from the three-dimensional data and the two-dimensional map, and the vehicle acquires the data of the two-dimensional feature amount along with the two-dimensional map.

If a two-dimensional map shows information on locations with different heights from the ground (not on the same plane), such as white lines in the road, guardrails, and buildings, the vehicle uses the own vehicle detection three-dimensional data. A feature quantity is extracted from the plurality of height data in step 712.

Further, information indicating the correspondence between the feature points in the two-dimensional map and the feature points in the three-dimensional map 711 may be stored as meta information of the map data.

The second method is a method of matching three-dimensional feature amounts. Specifically, the vehicle acquires the three-dimensional feature amount corresponding to the feature point in the two-dimensional map, and matches the acquired three-dimensional feature amount with the three-dimensional feature amount of the own vehicle detection three-dimensional data 712.

Specifically, three-dimensional feature amounts corresponding to feature points in the two-dimensional map are stored in the map data. The vehicle also acquires this three-dimensional feature amount when acquiring the two-dimensional map. Note that when the three-dimensional map 711 includes SWLD, by adding information that identifies feature points corresponding to feature points in the two-dimensional map among the feature points in SWLD, the vehicle can, based on the identification information, It is possible to determine the 3D features to be acquired together with the 2D map. Note that in this case, since it is sufficient to express a two-dimensional position, the amount of data can be reduced compared to the case where a three-dimensional position is expressed.

Further, when estimating the self-position using the two-dimensional map, the accuracy of self-position estimation is lower than that using the three-dimensional map 711. Therefore, the vehicle may determine whether automatic driving can be continued even if the estimation accuracy is reduced, and may continue automatic driving only when it is determined that automatic driving can be continued.

Whether autonomous driving can continue depends on whether the road the vehicle is traveling on is in an urban area or on a road with few other vehicles or pedestrians, such as a highway, the width of the road, or the degree of congestion (density of vehicles or pedestrians). ) and other driving environments. Furthermore, it is also possible to place markers for recognition by sensors such as cameras on the premises of business offices, in towns, or inside buildings. In these specific areas, markers can be recognized with high precision by two-dimensional sensors, so for example, by including position information of markers in a two-dimensional map, self-position estimation can be performed with high precision.

Furthermore, by including identification information indicating whether each area is a specific area in the map, the vehicle can determine whether the vehicle is present within the specific area. The vehicle determines to continue automatic driving if the vehicle is within the specific area. In this way, the vehicle may determine whether to continue automatic driving based on the accuracy of self-position estimation when using a two-dimensional map or the driving environment of the vehicle.

In this way, the three-dimensional information processing device 700 uses the results of estimating the vehicle's own position using the two-dimensional map and the own-vehicle detected three-dimensional data 712 based on the driving environment of the vehicle (the moving environment of the moving object). Determine whether or not to perform automatic operation of the vehicle.

Further, the vehicle may switch the level (mode) of automatic driving depending on the accuracy of self-position estimation or the driving environment of the vehicle, rather than whether or not automatic driving can be continued. Here, switching the level (mode) of autonomous driving means, for example, limiting the speed, increasing the amount of driver operation (lowering the automatic driving level), obtaining driving information of the car in front and using it as a reference. These include switching to an autonomous driving mode, obtaining driving information from cars set to the same destination, and using that information to switch to an autonomous driving mode.

Further, the map may include information associated with the position information and indicating a recommended level of automatic driving when estimating the self-position using the two-dimensional map. The recommendation level may be metadata that changes dynamically depending on traffic volume or the like. Thereby, the vehicle can determine the level simply by acquiring information in the map, without having to sequentially determine the level according to the surrounding environment or the like. Furthermore, by having multiple vehicles refer to the same map, it is possible to maintain the level of automatic driving of each vehicle at a constant level. Note that the recommended level may not be a recommendation, but may be a level that requires compliance.

Further, the vehicle may switch the level of automatic driving depending on the presence or absence of a driver (manned or unmanned). For example, if the vehicle is manned, the level of automatic driving will be lowered, and if it is unmanned, it will stop. A vehicle determines where it can safely stop by recognizing nearby passersby, vehicles, and traffic signs. Alternatively, the map may include position information indicating a position where the vehicle can safely stop, and the vehicle may refer to the position information to determine a position where the vehicle can safely stop.

Next, a description will be given of a countermeasure operation for abnormal case 2, in which the three-dimensional map 711 does not exist or the three-dimensional map 711 is obtained but is damaged.

The abnormal case determination unit 703 determines whether (1) the three-dimensional map 711 for part or all of the route to the destination does not exist on the distribution server to be accessed and cannot be obtained; or (2) It is checked whether part or all of the acquired three-dimensional map 711 is damaged, and if this is the case, it is determined to be abnormal case 2. That is, the abnormal case determination unit 703 determines whether the data of the three-dimensional map 711 is complete, and if the data of the three-dimensional map 711 is not complete, determines that the three-dimensional map 711 is abnormal.

If it is determined to be abnormal case 2, the following countermeasure actions are performed. First, (1) an example of a countermeasure operation when the three-dimensional map 711 cannot be obtained will be described.

For example, the vehicle sets a route that does not pass through sections where the three-dimensional map 711 does not exist.

In addition, if an alternative route cannot be set because an alternative route does not exist, or an alternative route exists but the distance will increase significantly, the vehicle may select a section where the three-dimensional map 711 does not exist. Set a route that includes Additionally, the vehicle notifies the driver that the driving mode will be switched in the relevant section, and switches the driving mode to manual mode.

(2) If part or all of the acquired three-dimensional map 711 is damaged, the following countermeasures are performed.

The vehicle identifies the damaged part in the three-dimensional map 711, requests data on the damaged part through communication, acquires the data on the damaged part, and updates the three-dimensional map 711 using the acquired data. At this time, the vehicle may specify the damaged part using position information such as absolute coordinates or relative coordinates on the three-dimensional map 711, or may specify the damaged part using an index number of a random access unit that constitutes the damaged part. . In this case, the vehicle replaces the random access unit that includes the damaged part with the acquired random access unit.

Next, a description will be given of an operation to deal with abnormal case 3, which is a case where the own vehicle detection three-dimensional data 712 cannot be generated due to a malfunction of the own vehicle's sensor or bad weather.

The abnormal case determination unit 703 checks whether the generation error of the own vehicle detection three-dimensional data 712 is within the allowable range, and if it is not within the permissible range, determines abnormal case 3. In other words, the abnormal case determination unit 703 determines whether the data generation accuracy of the own vehicle detection three-dimensional data 712 is equal to or higher than the reference value, and determines whether the data generation precision of the own vehicle detection three-dimensional data 712 is not equal to or higher than the reference value. In this case, it is determined that the own vehicle detection three-dimensional data 712 is abnormal.

The following method can be used to confirm whether the generation error of the own vehicle detection three-dimensional data 712 is within an allowable range.

The spatial resolution of the vehicle detected three-dimensional data 712 during normal operation is based on the resolution in the depth direction and scanning direction of the vehicle's three-dimensional sensor such as a range finder or stereo camera, or the density of the point cloud that can be generated. Determined in advance. Further, the vehicle acquires the spatial resolution of the three-dimensional map 711 from meta information included in the three-dimensional map 711.

The vehicle uses the spatial resolution of both to estimate a reference value for a matching error when matching the own vehicle detected three-dimensional data 712 and the three-dimensional map 711 based on three-dimensional feature amounts and the like. Matching errors include errors in 3D feature values for each feature point, statistics such as the average value of errors in 3D feature values between multiple feature points, or errors in spatial distances between multiple feature points. etc. can be used. The allowable range of deviation from the reference value is set in advance.

If the matching error between the own vehicle detection three-dimensional data 712 and the three-dimensional map 711 generated before or during traveling is not within the permissible range, the vehicle determines abnormality case 3.

Alternatively, the vehicle uses a test pattern with a known three-dimensional shape for accuracy check, and acquires own vehicle detection three-dimensional data 712 for the test pattern before starting driving, and checks whether the shape error is within an allowable range. It may be determined whether it is abnormal case 3 based on the following.

For example, the vehicle performs the above determination every time before starting to drive. Alternatively, the vehicle obtains time-series changes in the matching error by making the above determination at regular time intervals while driving. When the matching error tends to increase, the vehicle may determine that it is abnormal case 3 even if the error is within the allowable range. Additionally, if an abnormality is predicted based on time-series changes, the vehicle will notify the user of the predicted abnormality, such as by displaying a message urging inspection or repair. It's okay. Further, the vehicle may determine whether an abnormality is due to a temporary factor such as bad weather or an abnormality due to a sensor failure based on a time-series change, and may notify the user only of the abnormality due to a sensor failure.

In addition, when the vehicle is determined to be abnormal case 3, the vehicle (1) activates an emergency alternative sensor (rescue mode), (2) switches the driving mode, (3) corrects the operation of the three-dimensional sensor. Either or selectively perform one of the three types of countermeasure actions.

First, (1) the case where an emergency alternative sensor is activated will be explained. The vehicle operates an emergency alternative sensor that is different from the three-dimensional sensor used during normal driving. In other words, when the data generation accuracy of the own vehicle detection 3D data 712 is not higher than the reference value, the 3D information processing device 700 generates the own vehicle detection 3D data 712 ( 4th three-dimensional position information).

Specifically, when the vehicle uses multiple cameras or LiDAR to acquire the own vehicle detection three-dimensional data 712, the vehicle detects the direction in which the matching error of the own vehicle detection three-dimensional data 712 exceeds the allowable range. Identify malfunctioning sensors based on the following. The vehicle then activates a substitute sensor corresponding to the malfunctioning sensor.

The alternative sensor may be a three-dimensional sensor, a camera capable of acquiring a two-dimensional image, or a one-dimensional sensor such as an ultrasonic sensor. If the alternative sensor is a sensor other than a three-dimensional sensor, the accuracy of self-position estimation may decrease or self-position estimation may not be possible. You may switch.

For example, the vehicle continues in autonomous driving mode if the alternative sensor is a three-dimensional sensor. Further, when the alternative sensor is a two-dimensional sensor, the vehicle changes the driving mode from fully automatic driving to a semi-automatic driving mode that assumes human driving operation. Furthermore, if the alternative sensor is a one-dimensional sensor, the vehicle switches the driving mode to a manual mode in which automatic braking control is not performed.

Additionally, the vehicle may switch automatic driving modes based on the driving environment. For example, when the alternative sensor is a two-dimensional sensor, the vehicle continues the fully automatic driving mode when driving on a highway, and switches the driving mode to the semi-automatic driving mode when driving in a city area.

Furthermore, even if there is no alternative sensor, the vehicle may continue estimating its own position as long as a sufficient number of feature points can be acquired using only normally operating sensors. However, since detection in a specific direction becomes impossible, the vehicle switches the driving mode to semi-automatic driving or manual mode.

Next, (2) countermeasure action for switching the driving mode will be explained. The vehicle switches its driving mode from automatic driving mode to manual mode. Alternatively, the vehicle may continue to operate automatically until it reaches a place where it can safely stop, such as the shoulder of the road, and then stop. Further, the vehicle may switch the driving mode to manual mode after stopping. In this way, the three-dimensional information processing device 700 switches the automatic driving mode when the generation accuracy of the self-vehicle detection three-dimensional data 712 is not equal to or higher than the reference value.

Next, (3) a countermeasure operation for correcting the operation of the three-dimensional sensor will be described. The vehicle identifies malfunctioning three-dimensional sensors based on the direction in which matching errors occur, and calibrates the identified sensors. Specifically, when a plurality of LiDARs or cameras are used as sensors, a portion of the three-dimensional space reconstructed by each sensor overlaps. That is, data in the overlapping portion is acquired by multiple sensors. The three-dimensional point cloud data obtained for the overlapping portion is different between a normal sensor and a malfunctioning sensor. Therefore, in order for the malfunctioning sensor to acquire the same three-dimensional point cloud data as a normal sensor, the vehicle must correct the origin of LiDAR, or change the operation of predetermined points such as camera exposure or focus. Adjust.

After the adjustment, if the matching error falls within the allowable range, the vehicle continues the previous driving mode. On the other hand, if the matching accuracy does not fall within the allowable range even after the adjustment, the vehicle performs the above-mentioned (1) countermeasure operation of activating an emergency alternative sensor, or (2) countermeasure operation of switching the driving mode.

In this manner, the three-dimensional information processing device 700 corrects the sensor operation when the data generation accuracy of the vehicle-detected three-dimensional data 712 is not equal to or higher than the reference value.

The method for selecting a countermeasure action will be explained below. The countermeasure action may be selected by a user such as a driver, or may be automatically selected by the vehicle without the intervention of the user.

Further, the vehicle may switch control depending on whether a driver is riding along with the vehicle. For example, when a driver is in the same vehicle, the vehicle prioritizes switching to manual mode. On the other hand, if the driver is not in the vehicle, the vehicle prioritizes a mode in which it moves to a safe location and stops.

Information indicating the stopping location may be included in the three-dimensional map 711 as meta information. Alternatively, the vehicle may issue a response request for the stop location to a service that manages the operation information of the automatic driver, and obtain information indicating the stop location.

Furthermore, when the vehicle operates on a predetermined route, the vehicle operation mode may be shifted to a mode in which the operator manages the operation of the vehicle via a communication channel. In particular, an abnormality in the self-position estimation function of a vehicle running in fully automatic driving mode is highly dangerous. Therefore, when an abnormality is detected, or when the detected abnormality cannot be corrected, the vehicle notifies the service that manages the operation information of the occurrence of the abnormality via the communication channel. The service may notify vehicles traveling around the vehicle of the presence of the abnormal vehicle or instruct them to vacate a nearby parking spot.

Further, when an abnormal case is detected, the vehicle may reduce its traveling speed compared to normal times.

The vehicle is a self-driving vehicle that provides a dispatch service such as a taxi, and if an abnormality occurs in the vehicle, the vehicle will contact the operation control center and stop in a safe location. Additionally, the ride-hailing service will dispatch a replacement vehicle. Alternatively, a user of a ride-hailing service may drive the vehicle. In these cases, discounts on fees or the provision of bonus points may also be used.

Further, in the method for dealing with abnormal case 1, a method of estimating the self-position based on the two-dimensional map has been described, but the self-position may be estimated using the two-dimensional map even in normal times. FIG. 36 is a flowchart of the self-position estimation process in this case.

First, the vehicle acquires a three-dimensional map 711 near the driving route (S711). Next, the vehicle acquires self-vehicle detection three-dimensional data 712 based on the sensor information (S712).

Next, the vehicle determines whether the three-dimensional map 711 is necessary for self-position estimation (S713). Specifically, the vehicle determines whether or not the three-dimensional map 711 is necessary based on the accuracy of self-position estimation when using the two-dimensional map and the driving environment. For example, a method similar to the method for dealing with abnormal case 1 described above may be used.

If it is determined that the three-dimensional map 711 is not necessary (No in S714), the vehicle acquires a two-dimensional map (S715). At this time, the vehicle may also acquire additional information as described in the handling method for abnormal case 1. Additionally, the vehicle may generate a two-dimensional map from the three-dimensional map 711. For example, the vehicle may generate a two-dimensional map by cutting out an arbitrary plane from the three-dimensional map 711.

Next, the vehicle estimates its own position using the detected three-dimensional data 712 and the two-dimensional map (S716). Note that the method of self-position estimation using a two-dimensional map is, for example, the same as the method described in the method for dealing with abnormal case 1 described above.

On the other hand, if it is determined that the three-dimensional map 711 is necessary (Yes in S714), the vehicle acquires the three-dimensional map 711 (S717). Next, the vehicle estimates its own position using the detected three-dimensional data 712 and the three-dimensional map 711 (S718).

Note that the vehicle may switch between using the two-dimensional map as the basis or the three-dimensional map 711 as the basis, depending on the speed corresponding to the communication device of the own vehicle or the situation of the communication path. For example, the communication speed required when driving while receiving the three-dimensional map 711 is set in advance, and if the communication speed during driving is less than the set value, the vehicle uses the two-dimensional map as a base, and when driving, the communication speed is set in advance. If the communication speed is higher than the setting, the three-dimensional map 711 may be used as a basis. Note that the vehicle may use the two-dimensional map as a base without determining whether to use the two-dimensional map or the three-dimensional map.

(Embodiment 5)
In this embodiment, a method of transmitting three-dimensional data to a following vehicle, etc. will be described. FIG. 37 is a diagram showing an example of a target space of three-dimensional data to be transmitted to a following vehicle or the like.

The vehicle 801 collects three-dimensional data, such as a point cloud, included in a rectangular parallelepiped space 802 with a width W, a height H, and a depth D, located at a distance L from the vehicle 801 in front of the vehicle 801, over a time period of Δt. At intervals, it is sent to a traffic monitoring cloud that monitors the road conditions or to the following vehicle.

When a change occurs in the three-dimensional data included in the space 802 that has been transmitted in the past due to a vehicle or person entering the space 802 from the outside, the vehicle 801 updates the three-dimensional data of the space where the change occurred. Also send.

Note that although FIG. 37 shows an example in which the shape of the space 802 is a rectangular parallelepiped, the space 802 does not necessarily have to be a rectangular parallelepiped as long as it includes a space on the road ahead that is a blind spot from the following vehicle. .

It is desirable that the distance L is set to a distance at which a following vehicle that has received the three-dimensional data can safely stop. For example, the distance L is the distance that the following vehicle travels while it takes to receive three-dimensional data, the distance that the following vehicle travels until it starts decelerating according to the received data, and the distance that the following vehicle travels until it starts decelerating according to the received data. It is set to the sum of the distance required to stop safely after starting. Since these distances change depending on the speed, the distance L may change depending on the speed V of the vehicle, such as L=a×V+b (a and b are constants).

The width W is set to a value larger than at least the width of the lane in which the vehicle 801 is traveling. More preferably, the width W is set to a size that includes adjacent spaces such as left and right lanes or roadside strips.

The depth D may be a fixed value, but it may also be changed according to the speed V of the vehicle, such as D=c×V+d (c and d are constants). Further, by setting D so that D>V×Δt, the space to be transmitted can overlap with the space that has been transmitted in the past. Thereby, the vehicle 801 can more reliably transmit information about the space on the road to the following vehicle etc. without omission.

In this way, by limiting the 3D data transmitted by the vehicle 801 to a space useful for the following vehicle, the capacity of the 3D data to be transmitted can be effectively reduced, achieving low communication delay and cost. can.

Next, the configuration of the three-dimensional data creation device 810 according to this embodiment will be explained. FIG. 38 is a block diagram showing a configuration example of a three-dimensional data creation device 810 according to this embodiment. This three-dimensional data creation device 810 is mounted on a vehicle 801, for example. The three-dimensional data creation device 810 transmits and receives three-dimensional data to and from an external traffic monitoring cloud, a preceding vehicle, or a following vehicle, and creates and accumulates three-dimensional data.

The three-dimensional data creation device 810 includes a data reception section 811, a communication section 812, a reception control section 813, a format conversion section 814, a plurality of sensors 815, a three-dimensional data creation section 816, and a three-dimensional data synthesis section. 817, a three-dimensional data storage section 818, a communication section 819, a transmission control section 820, a format conversion section 821, and a data transmission section 822.

The data receiving unit 811 receives three-dimensional data 831 from the traffic monitoring cloud or the vehicle in front. The three-dimensional data 831 includes information such as a point cloud, a visible light image, depth information, sensor position information, or speed information, including an area that cannot be detected by the sensor 815 of the own vehicle.

The communication unit 812 communicates with the traffic monitoring cloud or the vehicle in front, and transmits a data transmission request or the like to the traffic monitoring cloud or the vehicle in front.

The reception control unit 813 exchanges information such as compatible formats with the communication destination via the communication unit 812, and establishes communication with the communication destination.

The format conversion unit 814 generates three-dimensional data 832 by performing format conversion or the like on the three-dimensional data 831 received by the data reception unit 811. Further, if the three-dimensional data 831 is compressed or encoded, the format conversion unit 814 performs decompression or decoding processing.

The plurality of sensors 815 are a group of sensors, such as LiDAR, a visible light camera, or an infrared camera, that acquire information outside the vehicle 801, and generate sensor information 833. For example, when the sensor 815 is a laser sensor such as LiDAR, the sensor information 833 is three-dimensional data such as a point cloud (point cloud data). Note that the number of sensors 815 does not have to be plural.

The three-dimensional data generation unit 816 generates three-dimensional data 834 from the sensor information 833. The three-dimensional data 834 includes information such as a point cloud, visible light image, depth information, sensor position information, or speed information, for example.

The three-dimensional data synthesis unit 817 synthesizes the three-dimensional data 834 created based on the sensor information 833 of the own vehicle with the three-dimensional data 832 created by the traffic monitoring cloud or the vehicle in front, etc. Three-dimensional data 835 is constructed that includes the space in front of the vehicle in front that cannot be detected by the sensor 815.

The three-dimensional data storage unit 818 stores the generated three-dimensional data 835 and the like.

The communication unit 819 communicates with the traffic monitoring cloud or the following vehicle, and transmits data transmission requests and the like to the traffic monitoring cloud or the following vehicle.

The transmission control unit 820 exchanges information such as compatible formats with the communication destination via the communication unit 819, and establishes communication with the communication destination. The transmission control unit 820 also controls the space of the 3D data to be transmitted based on the 3D data construction information of the 3D data 832 generated by the 3D data synthesis unit 817 and the data transmission request from the communication destination. Determine a certain transmission area.

Specifically, in response to a data transmission request from the traffic monitoring cloud or a following vehicle, the transmission control unit 820 determines a transmission area that includes the space in front of the own vehicle that cannot be detected by the sensor of the following vehicle. Furthermore, the transmission control unit 820 determines the transmission area by determining whether or not the transmittable space or the transmitted space has been updated based on the three-dimensional data construction information. For example, the transmission control unit 820 determines, as the transmission area, an area that is specified in the data transmission request and in which the corresponding three-dimensional data 835 exists. Then, the transmission control unit 820 notifies the format conversion unit 821 of the format and transmission area compatible with the communication destination.

The format conversion unit 821 converts the three-dimensional data 836 in the transmission area out of the three-dimensional data 835 stored in the three-dimensional data storage unit 818 into a format compatible with the receiving side, thereby converting the three-dimensional data 837 into three-dimensional data 837. generate. Note that the format conversion unit 821 may reduce the amount of data by compressing or encoding the three-dimensional data 837.

The data transmitter 822 transmits the three-dimensional data 837 to the traffic monitoring cloud or the following vehicle. This three-dimensional data 837 includes, for example, information such as a point cloud in front of the own vehicle, a visible light image, depth information, or sensor position information, including an area that becomes a blind spot of a following vehicle.

Note that although an example has been described here in which the format conversion units 814 and 821 perform format conversion, etc., format conversion does not need to be performed.

With such a configuration, the three-dimensional data creation device 810 acquires three-dimensional data 831 from the outside in an area that cannot be detected by the sensor 815 of the own vehicle, and combines the three-dimensional data 831 and sensor information 833 detected by the sensor 815 of the own vehicle. Three-dimensional data 835 is generated by combining the three-dimensional data 834 based on the three-dimensional data. Thereby, the three-dimensional data creation device 810 can generate three-dimensional data in a range that cannot be detected by the sensor 815 of the own vehicle.

In addition, in response to a data transmission request from the traffic monitoring cloud or a following vehicle, the three-dimensional data creation device 810 transmits three-dimensional data including the space in front of the vehicle that cannot be detected by the sensors of the following vehicle to the traffic monitoring cloud or the following vehicle. Can be sent to vehicles, etc.

Next, a procedure for transmitting three-dimensional data to a following vehicle in the three-dimensional data creation device 810 will be explained. FIG. 39 is a flowchart showing an example of a procedure for transmitting three-dimensional data to the traffic monitoring cloud or a following vehicle by the three-dimensional data creation device 810.

First, the three-dimensional data creation device 810 generates and updates the three-dimensional data 835 of a space including the space 802 on the road ahead of the own vehicle 801 (S801). Specifically, the three-dimensional data creation device 810 combines the three-dimensional data 834 created based on the sensor information 833 of the host vehicle 801 with the three-dimensional data 831 created by the traffic monitoring cloud, the vehicle in front, etc. Then, three-dimensional data 835 is constructed that includes the space in front of the vehicle in front that cannot be detected by the sensor 815 of the own vehicle.

Next, the three-dimensional data creation device 810 determines whether the three-dimensional data 835 included in the transmitted space has changed (S802).

If a change occurs in the three-dimensional data 835 included in the transmitted space due to a vehicle or person entering the space from the outside (Yes in S802), the three-dimensional data creation device 810 detects the change. The three-dimensional data including the three-dimensional data 835 of the generated space is transmitted to the traffic monitoring cloud or the following vehicle (S803).

Note that the three-dimensional data creation device 810 may transmit the three-dimensional data of the space where the change has occurred in accordance with the transmission timing of the three-dimensional data transmitted at predetermined intervals, but may transmit the three-dimensional data immediately after detecting the change. You may. In other words, the three-dimensional data creation device 810 may transmit three-dimensional data of a space in which a change has occurred with priority over three-dimensional data that is transmitted at predetermined intervals.

Furthermore, the three-dimensional data creation device 810 may transmit all of the three-dimensional data of the space where the change has occurred as the three-dimensional data of the space where the change has occurred, or may transmit the difference of the three-dimensional data (for example, appearance or disappearance). information on the three-dimensional point, displacement information on the three-dimensional point, etc.) may be transmitted.

Further, the three-dimensional data creation device 810 may transmit metadata related to the danger avoidance operation of the host vehicle, such as a sudden braking warning, to the following vehicle, prior to the three-dimensional data of the space where the change has occurred. According to this, the following vehicle can quickly recognize the sudden braking of the vehicle in front, and can start risk avoidance actions such as deceleration earlier.

If there is no change in the three-dimensional data 835 included in the transmitted space (No in S802), or after step S803, the three-dimensional data creation device 810 selects a predetermined point at a distance L in front of the own vehicle 801. The three-dimensional data included in the shape space is transmitted to the traffic monitoring cloud or the following vehicle (S804).

Further, for example, the processes of steps S801 to S804 are repeatedly performed at predetermined time intervals.

Further, the three-dimensional data creation device 810 does not need to transmit the three-dimensional data 837 of the space 802 if there is no difference between the three-dimensional data 835 of the space 802 that is currently being transmitted and the three-dimensional map. .

FIG. 40 is a flowchart showing the operation of the three-dimensional data creation device 810 in this case.

First, the three-dimensional data creation device 810 generates and updates the three-dimensional data 835 of a space including the space 802 on the road ahead of the own vehicle 801 (S811).

Next, the three-dimensional data creation device 810 determines whether the three-dimensional data 835 of the generated space 802 has been updated from the three-dimensional map (S812). That is, the three-dimensional data creation device 810 determines whether there is a difference between the generated three-dimensional data 835 of the space 802 and the three-dimensional map. Here, the three-dimensional map is three-dimensional map information managed by an infrastructure-side device such as a traffic monitoring cloud. For example, this three-dimensional map is acquired as three-dimensional data 831.

If there is an update (Yes in S812), the three-dimensional data creation device 810 transmits the three-dimensional data included in the space 802 to the traffic monitoring cloud or the following vehicle, similarly to the above (S813).

On the other hand, if there is no update (No in S812), the three-dimensional data creation device 810 does not transmit the three-dimensional data included in the space 802 to the traffic monitoring cloud and the following vehicle (S814). Note that the three-dimensional data creation device 810 may control the three-dimensional data of the space 802 not to be transmitted by setting the volume of the space 802 to zero. Additionally, the three-dimensional data creation device 810 may transmit information indicating that there is no update in the space 802 to the traffic monitoring cloud or the following vehicle.

As described above, for example, if there are no obstacles on the road, there will be no difference between the generated three-dimensional data 835 and the three-dimensional map on the infrastructure side, and no data will be transmitted. In this way, transmission of unnecessary data can be suppressed.

In addition, although the above description described an example in which the three-dimensional data creation device 810 is mounted on a vehicle, the three-dimensional data creation device 810 is not limited to a vehicle, and may be mounted on any moving body.

As described above, the three-dimensional data creation device 810 according to the present embodiment is a mobile object that includes a sensor 815 and a communication unit (data receiving unit 811 or data transmitting unit 822, etc.) that transmits and receives three-dimensional data to and from the outside. will be installed on. The three-dimensional data creation device 810 generates three-dimensional data 835 (second three-dimensional data) based on sensor information 833 detected by the sensor 815 and three-dimensional data 831 (first three-dimensional data) received by the data receiving unit 811. data). Three-dimensional data creation device 810 transmits three-dimensional data 837, which is part of three-dimensional data 835, to the outside.

Thereby, the three-dimensional data creation device 810 can generate three-dimensional data in a range that cannot be detected by the own vehicle. Further, the three-dimensional data creation device 810 can transmit three-dimensional data in a range that cannot be detected by other vehicles to the other vehicle.

Furthermore, the three-dimensional data creation device 810 repeatedly creates three-dimensional data 835 and transmits three-dimensional data 837 at predetermined intervals. The three-dimensional data 837 is three-dimensional data of a small space 802 of a predetermined size located a predetermined distance L from the current position of the vehicle 801 in the forward movement direction of the vehicle 801.

This limits the range of the three-dimensional data 837 to be transmitted, so the amount of three-dimensional data 837 to be transmitted can be reduced.

Further, the predetermined distance L changes depending on the moving speed V of the vehicle 801. For example, the higher the moving speed V, the longer the predetermined distance L becomes. Thereby, the vehicle 801 can set an appropriate small space 802 according to the moving speed V of the vehicle 801, and transmit the three-dimensional data 837 of the small space 802 to a following vehicle or the like.

Furthermore, the predetermined size changes depending on the moving speed V of the vehicle 801. For example, as the moving speed V increases, the predetermined size increases. For example, as the moving speed V increases, the depth D, which is the length of the small space 802 in the moving direction of the vehicle, increases. Thereby, the vehicle 801 can set an appropriate small space 802 according to the moving speed V of the vehicle 801, and transmit the three-dimensional data 837 of the small space 802 to a following vehicle or the like.

Furthermore, the three-dimensional data creation device 810 determines whether there is a change in the three-dimensional data 835 of the small space 802 that corresponds to the three-dimensional data 837 that has already been sent. When determining that there is a change, the three-dimensional data creation device 810 transmits three-dimensional data 837 (fourth three-dimensional data), which is at least a part of the three-dimensional data 835 that has changed, to an external following vehicle or the like.

Thereby, the vehicle 801 can transmit the three-dimensional data 837 of the changed space to the following vehicle or the like.

Additionally, the three-dimensional data creation device 810 gives priority to the changed three-dimensional data 837 (fourth three-dimensional data) over the regular three-dimensional data 837 (third three-dimensional data) that is transmitted periodically. Send. Specifically, the three-dimensional data creation device 810 generates three-dimensional data 837 (fourth three-dimensional data) that has changed before sending normal three-dimensional data 837 (third three-dimensional data) that is transmitted periodically. data). In other words, the three-dimensional data creation device 810 irregularly transmits the changed three-dimensional data 837 (fourth three-dimensional data) without waiting for the regular three-dimensional data 837 to be transmitted periodically. .

As a result, the vehicle 801 can preferentially transmit the three-dimensional data 837 of the changed space to the following vehicle, etc., so that the following vehicle, etc. can quickly make a judgment based on the three-dimensional data.

In addition, the changed three-dimensional data 837 (fourth three-dimensional data) is calculated by calculating the difference between the three-dimensional data 835 of the small space 802 corresponding to the three-dimensional data 837 that has already been sent and the three-dimensional data 835 after the change. show. This allows the amount of three-dimensional data 837 to be transmitted to be reduced.

Further, the three-dimensional data creation device 810 does not transmit the three-dimensional data 837 of the small space 802 if there is no difference between the three-dimensional data 837 of the small space 802 and the three-dimensional data 831 of the small space 802. Furthermore, the three-dimensional data creation device 810 may transmit information indicating that there is no difference between the three-dimensional data 837 of the small space 802 and the three-dimensional data 831 of the small space 802 to the outside.

As a result, unnecessary three-dimensional data 837 can be suppressed from being transmitted, so the amount of transmitted three-dimensional data 837 can be reduced.

(Embodiment 6)
In this embodiment, a display device and display method for displaying information obtained from a three-dimensional map, etc., and a storage device and storage method for the three-dimensional map, etc. will be described.

A mobile object such as a car or robot uses a three-dimensional map obtained from a server or communication with another vehicle, a two-dimensional image obtained from a sensor installed in the own car, or a self-driving object for autonomous driving of the car or autonomous movement of the robot. Utilize vehicle detection 3D data. Of these data, the data that the user wants to view or save may vary depending on the situation. A display device that switches display according to the situation will be described below.

FIG. 41 is a flowchart showing an overview of a display method by a display device. The display device is mounted on a moving object such as a car or a robot. Note that below, an example in which the moving object is a vehicle (automobile) will be described.

First, the display device determines whether to display two-dimensional surrounding information or three-dimensional surrounding information according to the driving situation of the vehicle (S901). Note that the two-dimensional peripheral information corresponds to first peripheral information in the claims, and the three-dimensional peripheral information corresponds to second peripheral information in the claims. Here, the surrounding information is information indicating the surroundings of the moving body, and is, for example, an image viewed from a vehicle in a predetermined direction or a map of the surroundings of the vehicle.

Two-dimensional peripheral information is information generated using two-dimensional data. The two-dimensional data here refers to two-dimensional map information or video. For example, the two-dimensional surrounding information is a map around the vehicle obtained from a two-dimensional map, or an image obtained by a camera mounted on the vehicle. Furthermore, the two-dimensional peripheral information does not include, for example, three-dimensional information. That is, when the two-dimensional surrounding information is a map around the vehicle, the map does not include information in the height direction. Further, when the two-dimensional peripheral information is an image obtained by a camera, the image does not include information in the depth direction.

Furthermore, three-dimensional peripheral information is information generated using three-dimensional data. Here, the three-dimensional data is, for example, a three-dimensional map. Note that the three-dimensional data may be information indicating the three-dimensional position or three-dimensional shape of an object around the vehicle, which is acquired from another vehicle or a server, or detected by the own vehicle. For example, the three-dimensional surrounding information is a two-dimensional or three-dimensional image or map of the surroundings of the vehicle generated using a three-dimensional map. Further, the three-dimensional peripheral information includes, for example, three-dimensional information. For example, when the three-dimensional surrounding information is an image in front of the vehicle, the image includes information indicating the distance to the object in the image. Alternatively, in the video, for example, a pedestrian or the like existing behind the vehicle in front is displayed. Further, the three-dimensional surrounding information may be obtained by superimposing information indicating these distances, pedestrians, etc. on an image obtained from a sensor mounted on a vehicle. Further, the three-dimensional surrounding information may be information in the height direction superimposed on a two-dimensional map.

Further, the three-dimensional data may be displayed three-dimensionally, or a two-dimensional image or a two-dimensional map obtained from the three-dimensional data may be displayed on a two-dimensional display or the like.

If it is determined in step S901 to display the three-dimensional surrounding information (Yes in S902), the display device displays the three-dimensional surrounding information (S903). On the other hand, if it is determined in step S901 to display the two-dimensional surrounding information (No in S902), the display device displays the two-dimensional surrounding information (S904). In this manner, the display device displays the three-dimensional peripheral information or two-dimensional peripheral information determined to be displayed in step S901.

A specific example will be explained below. In the first example, the display device switches the surrounding information to be displayed depending on whether the vehicle is being driven automatically or manually. Specifically, during automatic driving, the driver does not need to know detailed surrounding road information, so the display device displays two-dimensional surrounding information (for example, a two-dimensional map). On the other hand, during manual driving, three-dimensional surrounding information (for example, a three-dimensional map) is displayed so that detailed surrounding road information can be seen for safe driving.

In addition, during automatic driving, in order to show the user what kind of information the own vehicle is driving based on, the display device displays information that affected the driving operation (for example, the SWLD used for self-position estimation, (lanes, road signs, surrounding situation detection results, etc.) may also be displayed. For example, the display device may display this information in addition to the two-dimensional map.

The surrounding information displayed during automatic driving and manual driving is merely an example, and the display device may display three-dimensional surrounding information during automatic driving and two-dimensional surrounding information during manual driving. In addition, the display device may display metadata or surrounding situation detection results in addition to a two-dimensional or three-dimensional map or video, or may display two-dimensional or three-dimensional Metadata or surrounding situation detection results may be displayed instead of the original map or video. Here, metadata is information indicating the three-dimensional position or three-dimensional shape of an object acquired from a server or another vehicle. Further, the surrounding situation detection result is information indicating the three-dimensional position or three-dimensional shape of the object detected by the host vehicle.

In the second example, the display device switches the surrounding information to be displayed depending on the driving environment. For example, a display device switches the displayed peripheral information depending on the brightness of the outside world. Specifically, when the area around the vehicle is bright, the display device displays a two-dimensional image obtained by a camera mounted on the vehicle, or three-dimensional surrounding information created using the two-dimensional image. On the other hand, when the surroundings of the vehicle are dark, the two-dimensional images obtained from the camera mounted on the vehicle are dark and difficult to view, so the display device displays three-dimensional surrounding information created using lidar or millimeter wave radar. indicate.

Further, the display device may switch the surrounding information displayed depending on the driving region, which is the region in which the host vehicle currently exists. For example, the display device displays three-dimensional surrounding information so as to provide the user with information on surrounding buildings, etc. at a tourist spot, a city center, or near a destination. On the other hand, the display device displays two-dimensional surrounding information because it is considered that detailed surrounding information is not necessary in many cases, such as in mountainous areas or suburbs.

Further, the display device may switch the surrounding information to be displayed based on weather conditions. For example, when the weather is clear, the display device displays three-dimensional surrounding information created using a camera or lidar. On the other hand, in the case of rain or dense fog, the display device displays three-dimensional surrounding information created using a millimeter wave radar because the three-dimensional surrounding information provided by a camera or lidar tends to include noise.

Furthermore, the system may automatically switch between these displays, or the user may manually switch between these displays.

In addition, the three-dimensional surrounding information includes dense point group data generated based on WLD, mesh data generated based on MWLD, sparse data generated based on SWLD, and lane data generated based on lane world. , two-dimensional map data including three-dimensional shape information of roads and intersections, metadata including three-dimensional position or three-dimensional shape information that changes in real time, or own vehicle detection results. Generated from data.

Note that, as described above, WLD is three-dimensional point group data, and SWLD is data obtained by extracting a point group whose feature amount is equal to or greater than a threshold value from WLD. Further, MWLD is data having a mesh structure generated from WLD. The lane world is data obtained by extracting from WLD a point group whose feature amount is equal to or greater than a threshold value and which is necessary for self-position estimation, driving assistance, automatic driving, or the like.

Here, MWLD and SWLD have a smaller amount of data than WLD. Therefore, when more detailed data is required, the amount of communication data and processing amount can be appropriately reduced by using WLD, and otherwise using MWLD or SWLD. Furthermore, Lane World has a smaller amount of data than SWLD. Therefore, by using Lane World, the amount of communication data and processing amount can be further reduced.

Furthermore, although an example of switching between two-dimensional peripheral information and three-dimensional peripheral information has been described above, the display device determines the type of data (WLD, SWLD, etc.) used to generate three-dimensional peripheral information based on the above conditions. You may switch. That is, in the above description, in the case of displaying three-dimensional peripheral information, the display device displays three-dimensional peripheral information generated from first data (for example, WLD or SWLD) with a larger amount of data, and displays two-dimensional peripheral information. In the case of displaying the two-dimensional surrounding information, three-dimensional surrounding information generated from second data (for example, SWLD or lane world) having a smaller amount of data than the first data may be displayed instead of the two-dimensional surrounding information.

Further, the display device displays two-dimensional surrounding information or three-dimensional surrounding information on, for example, a two-dimensional display, a head-up display, or a head-mounted display mounted on the own vehicle. Alternatively, the display device may transfer and display the two-dimensional surrounding information or three-dimensional surrounding information to a mobile terminal such as a smartphone via wireless communication. In other words, the display device is not limited to one that is mounted on a moving object, but may be any device that operates in cooperation with the moving object. For example, when a user with a display device such as a smartphone boards a mobile object or drives a mobile object, information about the mobile object such as the position of the mobile object based on self-position estimation of the mobile object is displayed on the display device. displayed, or these pieces of information are displayed on a display device together with surrounding information.

Furthermore, when displaying a three-dimensional map, the display device may render the three-dimensional map and display it as two-dimensional data, or display it as three-dimensional data using a three-dimensional display or a three-dimensional hologram. .

Next, a method for saving a three-dimensional map will be explained. A mobile object such as a car or a robot uses a three-dimensional map obtained by communication with a server or another vehicle, a two-dimensional image obtained from a sensor installed in the own car, and Alternatively, use 3D data detected by your own vehicle. Of these data, the data that the user wants to view or save may vary depending on the situation. Below, we will explain how to save data depending on the situation.

The storage device is mounted on a moving object such as a car or a robot. Note that below, an example in which the moving object is a vehicle (automobile) will be described. Further, the storage device may be included in the display device described above.

In a first example, the storage device determines whether to store the three-dimensional map based on the region. Here, by saving the three-dimensional map in the storage medium of the own vehicle, automatic driving becomes possible within the saved space without communication with a server. However, due to limited storage capacity, only a limited amount of data can be saved. Therefore, the storage device limits the storage area as shown below.

For example, the storage device preferentially saves 3D maps of frequently traveled areas such as commuting routes or the vicinity of one's home. This eliminates the need to acquire data for frequently used areas each time, so the amount of communication data can be effectively reduced. Note that storing with priority means storing data with higher priority within a predetermined storage capacity. For example, if new data cannot be saved within the storage capacity, data with a lower priority than the new data is deleted.

Alternatively, the storage device preferentially stores three-dimensional maps of areas with poor communication environment. This eliminates the need to acquire data through communication in areas with poor communication environments, so it is possible to suppress the occurrence of cases in which a three-dimensional map cannot be acquired due to poor communication.

Alternatively, the storage device preferentially stores 3D maps of areas with high traffic volume. This makes it possible to preferentially save 3D maps of areas where accidents occur frequently. Therefore, in such areas, it is possible to prevent a three-dimensional map from being unable to be obtained due to poor communication and a decrease in the accuracy of automatic driving or driving assistance.

Alternatively, the storage device preferentially stores three-dimensional maps of areas with low traffic volume. Here, in areas with low traffic volume, there is a high possibility that the automatic driving mode, which automatically follows the vehicle in front, cannot be used. This may require more detailed peripheral information. Therefore, by preferentially saving three-dimensional maps of areas with low traffic volume, it is possible to improve the accuracy of automatic driving or driving assistance in such areas.

Note that the plurality of storage methods described above may be combined. Further, the area in which these three-dimensional maps are to be saved preferentially may be automatically determined by the system, or may be specified by the user.

Further, the storage device may delete a three-dimensional map that has been stored for a predetermined period of time, or may update the three-dimensional map to the latest data. This prevents old map data from being used. In addition, when updating map data, the storage device detects a difference area, which is a spatial area where there is a difference, by comparing the old map and the new map, and then updates the old map with the data of the difference area of the new map. Only the data in the area that has changed may be updated by adding or removing the data in the difference area from the old map.

Also, in this example, the saved three-dimensional map is used for autonomous driving. Therefore, by using SWLD as this three-dimensional map, the amount of communication data, etc. can be reduced. Note that the three-dimensional map is not limited to SWLD, and may be other types of data such as WLD.

In a second example, the storage device stores the three-dimensional map based on the event.

For example, the storage device stores special events encountered while driving as a three-dimensional map. This allows the user to view details of the event later. An example of an event saved as a three-dimensional map is shown below. Note that the storage device may store three-dimensional surrounding information generated from the three-dimensional map.

For example, the storage device stores the three-dimensional map before and after a collision, or when danger is detected.

Alternatively, the storage device stores a three-dimensional map of a characteristic scene, such as a beautiful scenery, a crowded place, or a tourist spot.

These events to be saved may be automatically determined by the system, or may be specified in advance by the user. For example, machine learning may be used as a method for determining these events.

Also, in this example, the saved three-dimensional map is used for viewing. Therefore, by using WLD as this three-dimensional map, high-quality video can be provided. Note that the three-dimensional map is not limited to WLD, but may be other types of data such as SWLD.

Hereinafter, a method for the display device to control display according to the user will be described. When the display device superimposes and displays the surrounding situation detection results obtained through vehicle-to-vehicle communication on a map, it expresses the surrounding vehicles in a wire frame or by adding transparency to the surrounding vehicles. Make the detected object located deep inside the vehicle visible. Alternatively, the display device may display an image from a viewpoint from above, so that the own vehicle, surrounding vehicles, and the surrounding situation detection results can be viewed from above.

When superimposing surrounding situation detection results or point cloud data using a head-up display on the surrounding environment visible through the windshield as shown in Figure 42, information is superimposed depending on differences in the user's posture, body shape, or eye position. The position may shift. FIG. 43 is a diagram showing a display example of the head-up display when the position is shifted.

In order to correct such a shift, the display device detects the user's posture, body shape, or eye position using information from an in-vehicle camera or a sensor mounted on the seat. The display device adjusts the position at which information is superimposed according to the detected posture, body shape, or eye position of the user. FIG. 44 is a diagram showing a display example of the head-up display after adjustment.

Note that such adjustment of the superimposition position may be performed manually by the user using a control device installed in the vehicle.

Further, the display device may show a safe place on a map and present it to the user in the event of a disaster. Alternatively, the vehicle may inform the user of the details of the disaster and the intention to go to a safe place, and then automatically drive to the safe place.

For example, vehicles are set up for areas with high altitudes above sea level to avoid being caught in a tsunami in the event of an earthquake. At this time, the vehicle may acquire information on roads that have become difficult to pass due to the earthquake through communication with the server, and perform processing according to the nature of the disaster, such as taking a route that avoids the roads.

Further, automatic driving may include a plurality of modes such as a movement mode and a drive mode.

In travel mode, the vehicle determines a route to the destination, taking into consideration factors such as quick arrival time, low fare, short driving distance, and low energy consumption, and then automatically drives according to the determined route. conduct.

In drive mode, the vehicle automatically determines a route to reach the destination at the time specified by the user. For example, when a user sets a destination and arrival time, the vehicle visits surrounding tourist spots, determines a route that will allow it to arrive at the destination at the set time, and automatically drives according to the determined route.

(Embodiment 7)
In the fifth embodiment, an example has been described in which a client device such as a vehicle transmits three-dimensional data to another vehicle or a server such as a traffic monitoring cloud. In this embodiment, a client device transmits sensor information obtained by a sensor to a server or another client device.

First, the configuration of the system according to this embodiment will be explained. FIG. 45 is a diagram showing the configuration of a three-dimensional map and sensor information transmission/reception system according to this embodiment. This system includes a server 901 and client devices 902A and 902B. Note that if the client devices 902A and 902B are not particularly distinguished, they are also referred to as client devices 902.

The client device 902 is, for example, an in-vehicle device mounted on a moving body such as a vehicle. The server 901 is, for example, a traffic monitoring cloud or the like, and is capable of communicating with a plurality of client devices 902 .

The server 901 transmits to the client device 902 a three-dimensional map composed of a point cloud. Note that the configuration of the three-dimensional map is not limited to a point cloud, and may represent other three-dimensional data such as a mesh structure.

The client device 902 transmits the sensor information acquired by the client device 902 to the server 901. The sensor information includes, for example, at least one of LiDAR acquisition information, a visible light image, an infrared image, a depth image, sensor position information, and speed information.

Data sent and received between the server 901 and the client device 902 may be compressed to reduce data, or may remain uncompressed to maintain data accuracy. When compressing data, a three-dimensional compression method based on, for example, an octree structure can be used for the point cloud. Furthermore, a two-dimensional image compression method can be used for visible light images, infrared images, and depth images. The two-dimensional image compression method is, for example, MPEG-4 AVC or HEVC standardized by MPEG.

Further, the server 901 transmits the three-dimensional map managed by the server 901 to the client device 902 in response to a three-dimensional map transmission request from the client device 902 . Note that the server 901 may transmit the three-dimensional map without waiting for a three-dimensional map transmission request from the client device 902. For example, server 901 may broadcast a three-dimensional map to one or more client devices 902 in a predetermined space. Further, the server 901 may transmit a three-dimensional map suitable for the position of the client device 902 at regular intervals to the client device 902, which has once received a transmission request. Further, the server 901 may transmit the three-dimensional map to the client device 902 every time the three-dimensional map managed by the server 901 is updated.

The client device 902 issues a three-dimensional map transmission request to the server 901. For example, if the client device 902 wants to estimate its own position while driving, the client device 902 transmits a three-dimensional map transmission request to the server 901.

Note that in the following cases, the client device 902 may issue a request to the server 901 to transmit a three-dimensional map. If the three-dimensional map held by the client device 902 is old, the client device 902 may issue a three-dimensional map transmission request to the server 901. For example, if a certain period of time has passed since the client device 902 acquired the three-dimensional map, the client device 902 may request the server 901 to send the three-dimensional map.

The client device 902 may issue a three-dimensional map transmission request to the server 901 at a certain time before the client device 902 leaves the space indicated by the three-dimensional map held by the client device 902 . For example, when the client device 902 exists within a predetermined distance from the boundary of a space indicated by a three-dimensional map held by the client device 902, the client device 902 issues a three-dimensional map transmission request to the server 901. It's okay. Furthermore, if the moving route and moving speed of the client device 902 are known, based on these, the time when the client device 902 will leave the space indicated by the three-dimensional map held by the client device 902 is predicted. It's okay.

If the error in alignment between the three-dimensional data created by the client device 902 from sensor information and the three-dimensional map is greater than a certain value, the client device 902 may issue a request to the server 901 to transmit the three-dimensional map.

The client device 902 transmits sensor information to the server 901 in response to the sensor information transmission request transmitted from the server 901 . Note that the client device 902 may send sensor information to the server 901 without waiting for a sensor information transmission request from the server 901. For example, once the client device 902 receives a sensor information transmission request from the server 901, it may periodically transmit the sensor information to the server 901 for a certain period of time. In addition, if the error in alignment between the three-dimensional data created by the client device 902 based on sensor information and the three-dimensional map obtained from the server 901 exceeds a certain level, the client device 902 detects the surrounding area of the client device 902. It may be determined that there is a possibility that a change has occurred in the three-dimensional map, and transmit that fact and sensor information to the server 901.

The server 901 issues a sensor information transmission request to the client device 902. For example, the server 901 receives location information of the client device 902, such as GPS, from the client device 902. If the server 901 determines that the client device 902 is approaching a space with little information in the 3D map managed by the server 901 based on the position information of the client device 902, the server 901 sends the client device 902 to generate a new 3D map. A request to send sensor information is issued to the device 902. Additionally, the server 901 issues a request to send sensor information when it wants to update a three-dimensional map, check road conditions during snowfall or disasters, traffic jams, incidents and accidents, etc. Good too.

Further, the client device 902 may set the amount of data of sensor information to be transmitted to the server 901 according to the communication state or band at the time of receiving the sensor information transmission request received from the server 901. Setting the amount of sensor information to be transmitted to the server 901 means, for example, increasing or decreasing the data itself, or selecting an appropriate compression method.

FIG. 46 is a block diagram showing a configuration example of the client device 902. The client device 902 receives a three-dimensional map composed of a point cloud or the like from the server 901, and estimates the self-position of the client device 902 from three-dimensional data created based on the sensor information of the client device 902. Further, the client device 902 transmits the acquired sensor information to the server 901.

The client device 902 includes a data reception unit 1011, a communication unit 1012, a reception control unit 1013, a format conversion unit 1014, a plurality of sensors 1015, a three-dimensional data creation unit 1016, a three-dimensional image processing unit 1017, It includes a three-dimensional data storage section 1018, a format conversion section 1019, a communication section 1020, a transmission control section 1021, and a data transmission section 1022.

The data receiving unit 1011 receives the three-dimensional map 1031 from the server 901. The three-dimensional map 1031 is data including a point cloud such as WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

The communication unit 1012 communicates with the server 901 and sends a data transmission request (for example, a three-dimensional map transmission request) to the server 901 .

The reception control unit 1013 exchanges information such as compatible formats with the communication destination via the communication unit 1012, and establishes communication with the communication destination.

The format conversion unit 1014 generates a three-dimensional map 1032 by performing format conversion or the like on the three-dimensional map 1031 received by the data receiving unit 1011. Further, if the three-dimensional map 1031 is compressed or encoded, the format conversion unit 1014 performs expansion or decoding processing. Note that if the three-dimensional map 1031 is uncompressed data, the format conversion unit 1014 does not perform expansion or decoding processing.

The plurality of sensors 1015 are a group of sensors, such as LiDAR, a visible light camera, an infrared camera, or a depth sensor, that acquire external information of the vehicle in which the client device 902 is mounted, and generate sensor information 1033. For example, when the sensor 1015 is a laser sensor such as LiDAR, the sensor information 1033 is three-dimensional data such as a point cloud (point cloud data). Note that the number of sensors 1015 does not have to be plural.

The three-dimensional data creation unit 1016 creates three-dimensional data 1034 around the host vehicle based on the sensor information 1033. For example, the three-dimensional data creation unit 1016 creates point cloud data with color information around the own vehicle using information acquired by LiDAR and visible light images acquired by a visible light camera.

The three-dimensional image processing unit 1017 performs self-position estimation processing of the own vehicle, etc. using the received three-dimensional map 1032 such as a point cloud and three-dimensional data 1034 around the own vehicle generated from the sensor information 1033. . Note that the three-dimensional image processing unit 1017 creates three-dimensional data 1035 around the own vehicle by combining the three-dimensional map 1032 and the three-dimensional data 1034, and uses the created three-dimensional data 1035 to estimate the own position. Processing may be performed.

The three-dimensional data storage unit 1018 stores a three-dimensional map 1032, three-dimensional data 1034, three-dimensional data 1035, and the like.

The format conversion unit 1019 generates sensor information 1037 by converting the sensor information 1033 into a format compatible with the receiving side. Note that the format conversion unit 1019 may reduce the amount of data by compressing or encoding the sensor information 1037. Further, the format conversion unit 1019 may omit the process if there is no need to perform format conversion. Further, the format conversion unit 1019 may control the amount of data to be transmitted according to the specification of the transmission range.

The communication unit 1020 communicates with the server 901 and receives data transmission requests (sensor information transmission requests) and the like from the server 901 .

The transmission control unit 1021 exchanges information such as compatible formats with the communication destination via the communication unit 1020, and establishes communication.

Data transmitter 1022 transmits sensor information 1037 to server 901. The sensor information 1037 includes, for example, information acquired by LiDAR, a brightness image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, speed information, etc. 1015.

Next, the configuration of the server 901 will be explained. FIG. 47 is a block diagram showing a configuration example of the server 901. The server 901 receives sensor information transmitted from the client device 902, and creates three-dimensional data based on the received sensor information. The server 901 updates the three-dimensional map managed by the server 901 using the created three-dimensional data. Further, the server 901 transmits the updated three-dimensional map to the client device 902 in response to a three-dimensional map transmission request from the client device 902.

The server 901 includes a data reception section 1111, a communication section 1112, a reception control section 1113, a format conversion section 1114, a three-dimensional data creation section 1116, a three-dimensional data synthesis section 1117, and a three-dimensional data storage section 1118. , a format conversion section 1119, a communication section 1120, a transmission control section 1121, and a data transmission section 1122.

The data receiving unit 1111 receives sensor information 1037 from the client device 902. The sensor information 1037 includes, for example, information acquired by LiDAR, a brightness image acquired by a visible light camera, an infrared image acquired by an infrared camera, a depth image acquired by a depth sensor, sensor position information, speed information, and the like.

The communication unit 1112 communicates with the client device 902 and transmits a data transmission request (for example, a request to transmit sensor information) to the client device 902 .

The reception control unit 1113 exchanges information such as compatible formats with the communication destination via the communication unit 1112, and establishes communication.

If the received sensor information 1037 is compressed or encoded, the format conversion unit 1114 generates sensor information 1132 by performing decompression or decoding processing. Note that if the sensor information 1037 is uncompressed data, the format conversion unit 1114 does not perform expansion or decoding processing.

The three-dimensional data creation unit 1116 creates three-dimensional data 1134 around the client device 902 based on the sensor information 1132. For example, the three-dimensional data creation unit 1116 creates point cloud data with color information around the client device 902 using information acquired by LiDAR and visible light images acquired by a visible light camera.

The three-dimensional data synthesis unit 1117 updates the three-dimensional map 1135 by synthesizing the three-dimensional data 1134 created based on the sensor information 1132 with the three-dimensional map 1135 managed by the server 901.

The three-dimensional data storage unit 1118 stores three-dimensional maps 1135 and the like.

The format conversion unit 1119 generates a three-dimensional map 1031 by converting the three-dimensional map 1135 into a format compatible with the receiving side. Note that the format conversion unit 1119 may reduce the amount of data by compressing or encoding the three-dimensional map 1135. Furthermore, the format conversion unit 1119 may omit the process if there is no need to perform format conversion. Further, the format conversion unit 1119 may control the amount of data to be transmitted according to the specification of the transmission range.

The communication unit 1120 communicates with the client device 902 and receives data transmission requests (three-dimensional map transmission requests) and the like from the client device 902 .

The transmission control unit 1121 exchanges information such as compatible formats with the communication destination via the communication unit 1120 and establishes communication.

The data transmitter 1122 transmits the three-dimensional map 1031 to the client device 902. The three-dimensional map 1031 is data including a point cloud such as WLD or SWLD. The three-dimensional map 1031 may include either compressed data or uncompressed data.

Next, the operation flow of the client device 902 will be explained. FIG. 48 is a flowchart showing the operation of the client device 902 when acquiring a three-dimensional map.

First, the client device 902 requests the server 901 to transmit a three-dimensional map (point cloud, etc.) (S1001). At this time, the client device 902 may request the server 901 to transmit a three-dimensional map related to the location information by also transmitting the location information of the client device 902 obtained by GPS or the like.

Next, the client device 902 receives a three-dimensional map from the server 901 (S1002). If the received three-dimensional map is compressed data, the client device 902 decodes the received three-dimensional map to generate an uncompressed three-dimensional map (S1003).

Next, the client device 902 creates three-dimensional data 1034 around the client device 902 from the sensor information 1033 obtained by the plurality of sensors 1015 (S1004). Next, the client device 902 estimates its own position using the three-dimensional map 1032 received from the server 901 and the three-dimensional data 1034 created from the sensor information 1033 (S1005).

FIG. 49 is a flowchart showing the operation of the client device 902 when transmitting sensor information. First, the client device 902 receives a sensor information transmission request from the server 901 (S1011). The client device 902 that has received the transmission request transmits the sensor information 1037 to the server 901 (S1012). Note that when the sensor information 1033 includes a plurality of pieces of information obtained by a plurality of sensors 1015, the client device 902 generates the sensor information 1037 by compressing each piece of information using a compression method suitable for each piece of information. good.

Next, the operation flow of the server 901 will be explained. FIG. 50 is a flowchart showing the operation of the server 901 when acquiring sensor information. First, the server 901 requests the client device 902 to transmit sensor information (S1021). Next, the server 901 receives sensor information 1037 transmitted from the client device 902 in response to the request (S1022). Next, the server 901 creates three-dimensional data 1134 using the received sensor information 1037 (S1023). Next, the server 901 reflects the created three-dimensional data 1134 on the three-dimensional map 1135 (S1024).

FIG. 51 is a flowchart showing the operation of the server 901 when transmitting a three-dimensional map. First, the server 901 receives a three-dimensional map transmission request from the client device 902 (S1031). The server 901, which has received the three-dimensional map transmission request, transmits the three-dimensional map 1031 to the client device 902 (S1032). At this time, the server 901 may extract a three-dimensional map of the vicinity according to the position information of the client device 902, and transmit the extracted three-dimensional map. Further, the server 901 may compress a three-dimensional map composed of a point cloud using a compression method using an octree structure, for example, and transmit the compressed three-dimensional map.

Modifications of this embodiment will be described below.

The server 901 uses the sensor information 1037 received from the client device 902 to create three-dimensional data 1134 around the location of the client device 902 . Next, the server 901 calculates the difference between the three-dimensional data 1134 and the three-dimensional map 1135 by matching the created three-dimensional data 1134 with the three-dimensional map 1135 of the same area managed by the server 901. . If the difference is greater than or equal to a predetermined threshold, the server 901 determines that some abnormality has occurred around the client device 902. For example, when ground subsidence occurs due to a natural disaster such as an earthquake, a large difference may occur between the three-dimensional map 1135 managed by the server 901 and the three-dimensional data 1134 created based on sensor information 1037. It is possible that

The sensor information 1037 may include information indicating at least one of the sensor type, sensor performance, and sensor model number. Furthermore, a class ID or the like may be added to the sensor information 1037 depending on the performance of the sensor. For example, if the sensor information 1037 is information acquired by LiDAR, a sensor that can acquire information with an accuracy of several millimeters is class 1, a sensor that can acquire information with an accuracy of several centimeters is class 2, and a sensor that can acquire information with an accuracy of several centimeters is class 2. It is conceivable to assign an identifier to the performance of a sensor, such as a class 3 sensor that can acquire information with high accuracy. Further, the server 901 may estimate sensor performance information and the like from the model number of the client device 902. For example, when the client device 902 is installed in a vehicle, the server 901 may determine sensor spec information from the vehicle type of the vehicle. In this case, the server 901 may obtain information on the vehicle type in advance, or the sensor information may include the information. Further, the server 901 may use the acquired sensor information 1037 to switch the degree of correction for the three-dimensional data 1134 created using the sensor information 1037. For example, when the sensor performance is high accuracy (class 1), the server 901 does not perform correction on the three-dimensional data 1134. If the sensor performance is low accuracy (class 3), the server 901 applies correction to the three-dimensional data 1134 according to the accuracy of the sensor. For example, the server 901 increases the degree (strength) of correction as the accuracy of the sensor decreases.

The server 901 may issue a sensor information transmission request to multiple client devices 902 in a certain space at the same time. When the server 901 receives a plurality of pieces of sensor information from a plurality of client devices 902, it is not necessary to use all the sensor information to create the three-dimensional data 1134. Information may be selected. For example, when updating the three-dimensional map 1135, the server 901 selects highly accurate sensor information (class 1) from a plurality of received sensor information, and creates three-dimensional data 1134 using the selected sensor information. You may.

The server 901 is not limited to a server such as a traffic monitoring cloud, but may be another client device (mounted in a vehicle). FIG. 52 is a diagram showing the system configuration in this case.

For example, the client device 902C issues a sensor information transmission request to the nearby client device 902A, and acquires the sensor information from the client device 902A. The client device 902C then creates three-dimensional data using the acquired sensor information of the client device 902A, and updates the three-dimensional map of the client device 902C. Thereby, the client device 902C can generate a three-dimensional map of the space that can be obtained from the client device 902A by taking advantage of the performance of the client device 902C. For example, such a case is considered to occur when the performance of the client device 902C is high.

Further, in this case, the client device 902A that provided the sensor information is given the right to acquire the highly accurate three-dimensional map generated by the client device 902C. Client device 902A receives a highly accurate three-dimensional map from client device 902C in accordance with its rights.

Further, the client device 902C may issue a sensor information transmission request to a plurality of nearby client devices 902 (client device 902A and client device 902B). If the sensor of client device 902A or client device 902B has high performance, client device 902C can create three-dimensional data using sensor information obtained by this high-performance sensor.

FIG. 53 is a block diagram showing the functional configuration of the server 901 and client device 902. The server 901 includes, for example, a three-dimensional map compression/decoding processing unit 1201 that compresses and decodes a three-dimensional map, and a sensor information compression/decoding processing unit 1202 that compresses and decodes sensor information.

The client device 902 includes a three-dimensional map decoding processing section 1211 and a sensor information compression processing section 1212. The three-dimensional map decoding processing unit 1211 receives encoded data of a compressed three-dimensional map, decodes the encoded data, and obtains a three-dimensional map. The sensor information compression processing unit 1212 compresses the sensor information itself instead of the three-dimensional data created from the acquired sensor information, and transmits encoded data of the compressed sensor information to the server 901. With this configuration, the client device 902 only needs to internally hold a processing unit (device or LSI) that performs processing to decode a three-dimensional map (point cloud, etc.), and the client device 902 can store three-dimensional data of the three-dimensional map (point cloud, etc.). There is no need to internally maintain a processing unit that performs the process of compressing the data. Thereby, the cost, power consumption, etc. of the client device 902 can be reduced.

As described above, the client device 902 according to the present embodiment is mounted on a moving object, and uses sensor information 1033 indicating the surrounding situation of the moving object obtained by the sensor 1015 installed on the moving object. Three-dimensional data 1034 of the surrounding area is created. The client device 902 estimates the self-position of the mobile object using the created three-dimensional data 1034. The client device 902 transmits the acquired sensor information 1033 to the server 901 or another mobile object 902.

According to this, the client device 902 transmits sensor information 1033 to the server 901 or the like. Thereby, it is possible to reduce the amount of transmitted data compared to the case of transmitting three-dimensional data. Further, since there is no need for the client device 902 to perform processing such as compression or encoding of three-dimensional data, the processing amount of the client device 902 can be reduced. Therefore, the client device 902 can reduce the amount of data to be transmitted or simplify the configuration of the device.

Further, the client device 902 further transmits a three-dimensional map transmission request to the server 901 and receives the three-dimensional map 1031 from the server 901. In estimating the self-position, the client device 902 uses the three-dimensional data 1034 and the three-dimensional map 1032 to estimate the self-position.

Further, the sensor information 1033 includes at least one of information obtained by a laser sensor, a brightness image, an infrared image, a depth image, sensor position information, and sensor speed information.

Further, the sensor information 1033 includes information indicating the performance of the sensor.

Further, the client device 902 encodes or compresses the sensor information 1033, and transmits the encoded or compressed sensor information 1037 to the server 901 or another mobile object 902 when transmitting the sensor information. According to this, the client device 902 can reduce the amount of data to be transmitted.

For example, the client device 902 includes a processor and a memory, and the processor uses the memory to perform the above processing.

Further, the server 901 according to the present embodiment is capable of communicating with a client device 902 mounted on a mobile body, and sensor information 1037 indicating the surrounding situation of the mobile body obtained by a sensor 1015 mounted on the mobile body. is received from the client device 902. The server 901 creates three-dimensional data 1134 around the moving object from the received sensor information 1037.

According to this, the server 901 creates three-dimensional data 1134 using sensor information 1037 sent from the client device 902. This may reduce the amount of transmitted data compared to when the client device 902 transmits three-dimensional data. Further, since there is no need for the client device 902 to perform processing such as compression or encoding of three-dimensional data, the processing amount of the client device 902 can be reduced. Therefore, the server 901 can reduce the amount of data to be transmitted or simplify the configuration of the device.

Furthermore, the server 901 further transmits a sensor information transmission request to the client device 902.

Further, the server 901 further updates the 3D map 1135 using the created 3D data 1134, and transmits the 3D map 1135 to the client device 902 in response to a request to transmit the 3D map 1135 from the client device 902. Send.

Further, the sensor information 1037 includes at least one of information obtained by a laser sensor, a brightness image, an infrared image, a depth image, sensor position information, and sensor speed information.

Further, the sensor information 1037 includes information indicating the performance of the sensor.

Furthermore, the server 901 further corrects the three-dimensional data according to the performance of the sensor. According to this, the three-dimensional data creation method can improve the quality of three-dimensional data.

Further, in receiving sensor information, the server 901 receives a plurality of sensor information 1037 from a plurality of client devices 902, and generates three-dimensional data 1134 based on a plurality of pieces of information indicating the performance of the sensor included in the plurality of sensor information 1037. Select sensor information 1037 to be used for creation. According to this, the server 901 can improve the quality of the three-dimensional data 1134.

Further, the server 901 decodes or decompresses the received sensor information 1037 and creates three-dimensional data 1134 from the decoded or decompressed sensor information 1132. According to this, the server 901 can reduce the amount of data to be transmitted.

For example, the server 901 includes a processor and a memory, and the processor uses the memory to perform the above processing.

(Embodiment 8)
In this embodiment, a modification of the seventh embodiment will be described. FIG. 54 is a diagram showing the configuration of a system according to this embodiment. The system shown in FIG. 54 includes a server 2001, a client device 2002A, and a client device 2002B.

The client device 2002A and the client device 2002B are mounted on a moving body such as a vehicle, and transmit sensor information to the server 2001. Server 2001 transmits a three-dimensional map (point cloud) to client device 2002A and client device 2002B.

The client device 2002A includes a sensor information acquisition unit 2011, a storage unit 2012, and a data transmission permission determination unit 2013. Note that the configuration of the client device 2002B is also similar. Furthermore, hereinafter, unless the client device 2002A and the client device 2002B are particularly distinguished, they will also be referred to as the client device 2002.

FIG. 55 is a flowchart showing the operation of client device 2002 according to this embodiment.

The sensor information acquisition unit 2011 acquires various sensor information using sensors (sensor group) mounted on a moving body. That is, the sensor information acquisition unit 2011 acquires sensor information indicating the surrounding situation of the mobile body, which is obtained by a sensor (sensor group) mounted on the mobile body. Further, the sensor information acquisition unit 2011 stores the acquired sensor information in the storage unit 2012. This sensor information includes at least one of LiDAR acquisition information, a visible light image, an infrared image, and a depth image. Further, the sensor information may include at least one of sensor position information, speed information, acquisition time information, and acquisition location information. The sensor position information indicates the position of the sensor that acquired the sensor information. The speed information indicates the speed of the moving body when the sensor acquires the sensor information. The acquisition time information indicates the time when the sensor information was acquired by the sensor. The acquisition location information indicates the position of the moving object or the sensor when the sensor information was acquired by the sensor.

Next, the data transmission permission determining unit 2013 determines whether the mobile object (client device 2002) exists in an environment where sensor information can be transmitted to the server 2001 (S2002). For example, the data transmission permission determining unit 2013 may use information such as GPS to identify the location and time of the client device 2002, and determine whether data can be transmitted. Further, the data transmission possibility determining unit 2013 may determine whether data can be transmitted based on whether or not it is possible to connect to a specific access point.

When the client device 2002 determines that the mobile object exists in an environment where sensor information can be transmitted to the server 2001 (S2002: Yes), the client device 2002 transmits the sensor information to the server 2001 (S2003). That is, when the client device 2002 becomes able to send sensor information to the server 2001, the client device 2002 sends the sensor information it holds to the server 2001. For example, millimeter wave access points capable of high-speed communication are installed at intersections and the like. When the client device 2002 enters the intersection, it quickly transmits sensor information held by the client device 2002 to the server 2001 using millimeter wave communication.

Next, the client device 2002 deletes the sensor information already sent to the server 2001 from the storage unit 2012 (S2004). Note that the client device 2002 may delete sensor information that has not been sent to the server 2001 when the sensor information satisfies a predetermined condition. For example, the client device 2002 may delete sensor information from the storage unit 2012 when the acquisition time of the sensor information it holds becomes older than a certain time before the current time. That is, the client device 2002 may delete sensor information from the storage unit 2012 when the difference between the time when the sensor information was acquired by the sensor and the current time exceeds a predetermined time. Further, the client device 2002 may delete the sensor information from the storage unit 2012 when the acquired location of the retained sensor information becomes more than a certain distance away from the current location. In other words, the client device 2002 receives sensor information when the difference between the position of the moving body or sensor when the sensor information was acquired by the sensor and the current position of the moving body or sensor exceeds a predetermined distance. may be deleted from the storage unit 2012. Thereby, the capacity of the storage unit 2012 of the client device 2002 can be suppressed.

If the acquisition of sensor information by the client device 2002 is not completed (No in S2005), the client device 2002 performs the processing from step S2001 again. Furthermore, when the acquisition of sensor information by the client device 2002 is completed (Yes in S2005), the client device 2002 ends the process.

Further, the client device 2002 may select the sensor information to be transmitted to the server 2001 according to the communication situation. For example, if high-speed communication is possible, the client device 2002 preferentially transmits sensor information held in the storage unit 2012 that is large in size (for example, LiDAR acquisition information, etc.). Furthermore, if high-speed communication is difficult, the client device 2002 transmits sensor information (for example, a visible light image) that is small in size and has a high priority, which is held in the storage unit 2012. Thereby, the client device 2002 can efficiently transmit the sensor information held in the storage unit 2012 to the server 2001 according to the network situation.

Further, the client device 2002 may obtain time information indicating the current time and location information indicating the current location from the server 2001. Further, the client device 2002 may determine the acquisition time and acquisition location of the sensor information based on the acquired time information and location information. That is, the client device 2002 may acquire time information from the server 2001 and generate acquisition time information using the acquired time information. Further, the client device 2002 may acquire location information from the server 2001 and generate the acquired location information using the acquired location information.

For example, regarding time information, the server 2001 and the client device 2002 perform time synchronization using a mechanism such as NTP (Network Time Protocol) or PTP (Precision Time Protocol). This allows the client device 2002 to obtain accurate time information. Furthermore, since the time can be synchronized between the server 2001 and a plurality of client devices, the times in the sensor information acquired by different client devices 2002 can be synchronized. Therefore, the server 2001 can handle sensor information indicating synchronized time. Note that the time synchronization mechanism may be any method other than NTP or PTP. Furthermore, GPS information may be used as the time information and location information.

The server 2001 may acquire sensor information from a plurality of client devices 2002 by specifying a time or location. For example, when an accident occurs, the server 2001 specifies the time and location of the accident and broadcasts a sensor information transmission request to a plurality of client devices 2002 in order to search for clients who were nearby. Then, the client device 2002 that has the sensor information of the relevant time and location transmits the sensor information to the server 2001. That is, the client device 2002 receives a sensor information transmission request including specification information specifying the location and time from the server 2001. The client device 2002 determines that the storage unit 2012 stores sensor information obtained at the location and time indicated by the specified information, and that the mobile object exists in an environment where sensor information can be transmitted to the server 2001. In this case, sensor information obtained at the location and time indicated by the specified information is transmitted to the server 2001. Thereby, the server 2001 can acquire sensor information related to the occurrence of an accident from a plurality of client devices 2002 and use it for accident analysis and the like.

Note that when the client device 2002 receives a sensor information transmission request from the server 2001, it may refuse to transmit the sensor information. Further, the client device 2002 may set in advance which sensor information among the plurality of sensor information can be transmitted. Alternatively, the server 2001 may inquire of the client device 2002 each time whether sensor information can be transmitted.

Furthermore, points may be given to the client device 2002 that has transmitted sensor information to the server 2001. These points can be used, for example, to pay for gasoline purchase costs, EV (Electric Vehicle) charging costs, expressway tolls, rental car costs, and the like. Further, after acquiring the sensor information, the server 2001 may delete information for identifying the client device 2002 that is the source of the sensor information. For example, this information is information such as the network address of client device 2002. This allows the sensor information to be anonymized, so the user of the client device 2002 can send the sensor information from the client device 2002 to the server 2001 with peace of mind. Further, the server 2001 may be composed of a plurality of servers. For example, by sharing sensor information among multiple servers, even if one server fails, other servers can communicate with the client device 2002. This makes it possible to avoid service suspension due to server failure.

Furthermore, the specified location specified in the sensor information transmission request indicates the location where the accident occurred, and may be different from the location of the client device 2002 at the specified time specified in the sensor information transmission request. Therefore, the server 2001 can request information acquisition from the client devices 2002 existing within the specified range, for example, by specifying a range within XXm of the surrounding area as the specified location. Similarly, for the designated time, the server 2001 may designate a range, such as within N seconds before or after a certain time. Thereby, the server 2001 can acquire sensor information from the client device 2002 that was present at "time: t-N to t+N, location: within XXm from the absolute position S". When transmitting three-dimensional data such as LiDAR, the client device 2002 may transmit data generated immediately after time t.

Furthermore, the server 2001 may separately designate information indicating the location of the client device 2002 from which sensor information is to be obtained and a location from which sensor information is desired, as the designated location. For example, the server 2001 specifies that sensor information including at least a range of YYm from the absolute position S is to be acquired from the client device 2002 existing within XXm from the absolute position S. When selecting three-dimensional data to transmit, the client device 2002 selects one or more randomly accessible units of three-dimensional data so as to include at least sensor information in a specified range. Further, when transmitting a visible light image, the client device 2002 may transmit a plurality of temporally continuous image data including at least a frame immediately before or after time t.

If the client device 2002 can use multiple physical networks to transmit sensor information, such as 5G, WiFi, or multiple modes in 5G, the client device 2002 selects the network to use according to the priority order notified from the server 2001. You may choose. Alternatively, the client device 2002 itself may select a network that can secure an appropriate bandwidth based on the size of the transmitted data. Alternatively, the client device 2002 may select a network to use based on the cost of data transmission and the like. Further, the transmission request from the server 2001 may include information indicating a transmission deadline, such as that the client device 2002 will transmit if it can start transmission by time T. The server 2001 may issue the transmission request again if sufficient sensor information cannot be acquired within the time limit.

The sensor information may include compressed or uncompressed sensor data, as well as header information indicating characteristics of the sensor data. Client device 2002 may send the header information to server 2001 via a different physical network or communication protocol than the sensor data. For example, the client device 2002 transmits header information to the server 2001 before transmitting sensor data. The server 2001 determines whether to acquire sensor data of the client device 2002 based on the analysis result of the header information. For example, the header information may include information indicating the LiDAR point cloud acquisition density, elevation angle, or frame rate, or the resolution, signal-to-noise ratio, or frame rate of the visible light image. Thereby, the server 2001 can acquire sensor information from the client device 2002 having sensor data of the determined quality.

As described above, the client device 2002 is mounted on a moving object, acquires sensor information indicating the surrounding situation of the moving object obtained by a sensor mounted on the moving object, and stores the sensor information in the storage unit 2012. . The client device 2002 determines whether the mobile object exists in an environment where it can transmit sensor information to the server 2001, and if it determines that the mobile object exists in an environment where it can transmit sensor information to the server, it transmits the sensor information to the server 2001. Send to.

Furthermore, the client device 2002 further creates three-dimensional data around the moving object from the sensor information, and estimates the self-position of the moving object using the created three-dimensional data.

Further, the client device 2002 further transmits a three-dimensional map transmission request to the server 2001 and receives the three-dimensional map from the server 2001. In estimating its own position, the client device 2002 uses three-dimensional data and a three-dimensional map to estimate its own position.

Note that the above processing by the client device 2002 may be implemented as an information transmission method in the client device 2002.

Further, the client device 2002 may include a processor and a memory, and the processor may perform the above processing using the memory.

Next, a sensor information collection system according to this embodiment will be explained. FIG. 56 is a diagram showing the configuration of a sensor information collection system according to this embodiment. As shown in FIG. 56, the sensor information collection system according to this embodiment includes a terminal 2021A, a terminal 2021B, a communication device 2022A, a communication device 2022B, a network 2023, a data collection server 2024, and a map server 2025. , client device 2026. Note that if the terminal 2021A and the terminal 2021B are not particularly distinguished, they are also referred to as the terminal 2021. The communication device 2022A and the communication device 2022B are also referred to as the communication device 2022 unless they are particularly distinguished.

The data collection server 2024 collects data such as sensor data obtained by a sensor included in the terminal 2021 as position-related data associated with a position in a three-dimensional space.

The sensor data is, for example, data obtained by using a sensor included in the terminal 2021, such as the state around the terminal 2021 or the state inside the terminal 2021. The terminal 2021 sends sensor data collected from one or more sensor devices located at a position where it can communicate directly with the terminal 2021 or can communicate with the terminal 2021 by relaying one or more relay devices using the same communication method to the data collection server 2024. Send.

The data included in the location-related data may include, for example, information indicating the operating status of the terminal itself or the equipment included in the terminal, an operation log, a service usage status, and the like. Furthermore, the data included in the location-related data may include information that associates the identifier of the terminal 2021 with the location or movement route of the terminal 2021, or the like.

Information indicating a position included in the position-related data is associated with information indicating a position in three-dimensional data such as three-dimensional map data, for example. Details of the information indicating the position will be described later.

Location-related data includes, in addition to location information that indicates location, the aforementioned time information, attributes of data included in location-related data, or information that indicates the type of sensor that generated the data (for example, model number, etc.) It may contain at least one of the following. The position information and time information may be stored in a header area of position-related data or a header area of a frame storing position-related data. Further, the location information and time information may be transmitted and/or stored separately from the location-related data as metadata associated with the location-related data.

The map server 2025 is connected to the network 2023, for example, and transmits three-dimensional data such as three-dimensional map data in response to a request from another device such as the terminal 2021. Further, as described in each of the above-described embodiments, the map server 2025 may have a function of updating three-dimensional data using sensor information transmitted from the terminal 2021.

The data collection server 2024 is connected to the network 2023, for example, and collects location-related data from other devices such as the terminal 2021, and stores the collected location-related data in a storage device internally or in another server. Further, the data collection server 2024 transmits the collected location-related data or metadata of three-dimensional map data generated based on the location-related data to the terminal 2021 in response to a request from the terminal 2021.

Network 2023 is, for example, a communication network such as the Internet. Terminal 2021 is connected to network 2023 via communication device 2022. The communication device 2022 communicates with the terminal 2021 while switching between one communication method or a plurality of communication methods. The communication device 2022 is, for example, (1) a base station such as LTE (Long Term Evolution), (2) an access point (AP) such as WiFi or millimeter wave communication, (3) LPWA such as SIGFOX, LoRaWAN, or Wi-SUN. (Low Power Wide Area) Network gateway, or (4) a communication satellite that communicates using a satellite communication method such as DVB-S2.

Note that the base station may communicate with the terminal 2021 using a method classified as LPWA such as NB-IoT (Narrow Band-IoT) or LTE-M, or may communicate with the terminal 2021 while switching between these methods. may also be communicating.

Here, the terminal 2021 has a function of communicating with a communication device 2022 that uses two types of communication methods, and uses one of these communication methods, or uses multiple communication methods and a communication device that is a direct communication partner. 2022 is switched while communicating with the map server 2025 or the data collection server 2024, but the configurations of the sensor information collection system and the terminal 2021 are not limited to this. For example, the terminal 2021 may not have a communication function using a plurality of communication methods, but may have a function of communicating using any one communication method. Further, the terminal 2021 may support three or more communication methods. Further, the communication method supported by each terminal 2021 may be different.

The terminal 2021 includes, for example, the configuration of the client device 902 shown in FIG. 46. The terminal 2021 uses the received three-dimensional data to estimate a position such as its own position. Furthermore, the terminal 2021 generates position-related data by associating the sensor data acquired from the sensor with the position information obtained through the position estimation process.

The position information added to the position-related data indicates, for example, a position in a coordinate system used in three-dimensional data. For example, the location information is coordinate values represented by latitude and longitude values. At this time, the terminal 2021 may include in the position information, along with the coordinate values, information indicating a coordinate system serving as a reference for the coordinate values and three-dimensional data used for position estimation. Further, the coordinate values may include altitude information.

Furthermore, the position information may be associated with a unit of data or a unit of space that can be used for encoding the three-dimensional data described above. This unit is, for example, WLD, GOS, SPC, VLM, or VXL. At this time, the location information is expressed, for example, by an identifier for specifying a data unit such as an SPC corresponding to location-related data. In addition to an identifier for specifying a data unit such as an SPC, location information includes information indicating three-dimensional data that encodes a three-dimensional space that includes a data unit such as the SPC, or details within the SPC. It may also include information indicating the location. The information indicating three-dimensional data is, for example, the file name of the three-dimensional data.

In this way, the system generates location-related data that is associated with location information based on location estimation using three-dimensional data, thereby determining the self-location of the client device (terminal 2021) obtained using GPS. It is possible to add more accurate position information to sensor information than when adding position information based on sensor information to sensor information. As a result, even when location-related data is used by other devices for other services, the location can be estimated based on the same three-dimensional data, allowing more accurate identification of the location corresponding to the location-related data in real space. There is a possibility that it can be done.

Note that although this embodiment has been described using an example in which the data transmitted from the terminal 2021 is location-related data, the data transmitted from the terminal 2021 may be data that is not associated with location information. good. That is, the transmission and reception of three-dimensional data or sensor data described in other embodiments may be performed via the network 2023 described in this embodiment.

Next, different examples of position information indicating a position in three-dimensional or two-dimensional real space or map space will be explained. The position information added to the position-related data may be information indicating a relative position to a feature point in the three-dimensional data. Here, the feature points serving as the reference for the position information are, for example, feature points encoded as SWLD and notified to the terminal 2021 as three-dimensional data.

The information indicating the relative position with respect to the feature point may be expressed, for example, by a vector from the feature point to the point indicated by the position information, or may be information indicating the direction and distance from the feature point to the point indicated by the position information. Alternatively, the information indicating the relative position to the feature point may be information indicating the amount of displacement in each of the X-axis, Y-axis, and Z-axis from the feature point to the point indicated by the position information. Further, the information indicating the relative position to the feature point may be information indicating the distance from each of the three or more feature points to the point indicated by the position information. Note that the relative position may not be the relative position of the point indicated by the position information expressed with each feature point as a reference, but may be the relative position of each feature point expressed with the point indicated by the position information as a reference. An example of position information based on a relative position to a feature point includes information for specifying a reference feature point and information indicating a relative position of a point indicated by the position information with respect to the feature point. In addition, if information indicating the relative position to the feature point is provided separately from the three-dimensional data, the information indicating the relative position to the feature point may include the coordinate axes used to derive the relative position, information indicating the type of three-dimensional data, Or/and information indicating the size (scale, etc.) per unit amount of the value of the information indicating the relative position may be included.

Further, the position information may include information indicating the relative position of each feature point for a plurality of feature points. When position information is expressed as a relative position to a plurality of feature points, the terminal 2021 that attempts to identify the position indicated by the position information in real space extracts the position information from the position of the feature point estimated from sensor data for each feature point. A candidate point at the position indicated by is calculated, and a point obtained by averaging the plurality of calculated candidate points may be determined to be the point indicated by the position information. According to this configuration, it is possible to reduce the influence of errors when estimating the position of a feature point from sensor data, and thus it is possible to improve the accuracy of estimating a point indicated by position information in real space. In addition, if the position information includes information indicating relative positions with respect to multiple feature points, even if there are feature points that cannot be detected due to constraints such as the type or performance of the sensor included in the terminal 2021, any of the multiple feature points If even one of them can be detected, it becomes possible to estimate the value of the point indicated by the position information.

Points that can be identified from sensor data can be used as feature points. A point that can be specified from sensor data is, for example, a point or a point within a region that satisfies a predetermined condition for feature point detection, such as the aforementioned three-dimensional feature amount or feature amount of visible light data being equal to or greater than a threshold value.

Furthermore, markers placed in real space may be used as feature points. In this case, it is sufficient that the marker can be detected and located from data acquired using a sensor such as a LiDER or a camera. For example, the marker is expressed by a change in color or brightness value (reflectance), or by a three-dimensional shape (irregularities, etc.). Further, coordinate values indicating the position of the marker, or a two-dimensional code or barcode generated from the identifier of the marker may be used.

Furthermore, a light source that transmits an optical signal may be used as a marker. When a light source of an optical signal is used as a marker, not only information for obtaining a position such as coordinate values or an identifier, but also other data may be transmitted by the optical signal. For example, the optical signal includes the content of the service according to the position of the marker, an address such as a URL for acquiring the content, or an identifier of the wireless communication device to receive the service, and a link to the wireless communication device. The information may also include information indicating the wireless communication method for the communication. By using an optical communication device (light source) as a marker, it becomes easy to transmit data other than information indicating a position, and it becomes possible to dynamically switch the data.

The terminal 2021 grasps the correspondence of feature points between mutually different data using, for example, an identifier commonly used among the data, or information or a table indicating the correspondence of feature points between data. In addition, if there is no information indicating the correspondence between feature points, the terminal 2021 converts the coordinates of the feature points in one three-dimensional data to the position in the other three-dimensional data space, A point may be determined to be a corresponding feature point.

When position information based on the relative position explained above is used, even between terminals 2021 or services that use different three-dimensional data, information included in each three-dimensional data or associated with each three-dimensional data The position indicated by the position information can be specified or estimated based on common feature points. As a result, it becomes possible to specify or estimate the same position with higher accuracy between terminals 2021 or services that use different three-dimensional data.

In addition, even when using map data or three-dimensional data expressed using different coordinate systems, it is possible to reduce the effects of errors caused by coordinate system conversion, thereby providing services based on more accurate location information. Cooperation becomes possible.

Examples of functions provided by the data collection server 2024 will be described below. Data collection server 2024 may forward the received location-related data to other data servers. If there are multiple data servers, the data collection server 2024 determines which data server to transfer the received location-related data to, and transfers the location-related data to the data server determined as the transfer destination.

The data collection server 2024 determines the transfer destination based on, for example, a transfer destination server determination rule set in the data collection server 2024 in advance. The transfer destination server determination rule is set, for example, in a transfer destination table that associates an identifier associated with each terminal 2021 with a transfer destination data server.

The terminal 2021 adds an identifier associated with the terminal 2021 to the location-related data to be transmitted, and transmits the data to the data collection server 2024. The data collection server 2024 identifies the forwarding destination data server corresponding to the identifier added to the location-related data based on a forwarding destination server determination rule using a forwarding destination table, etc., and determines whether the location-related data has been identified. Send to data server. Further, the determination rule for the transfer destination server may be specified by a determination condition using the time or place at which the location-related data was acquired. Here, the identifier associated with the above-mentioned transmission source terminal 2021 is, for example, an identifier unique to each terminal 2021 or an identifier indicating a group to which the terminal 2021 belongs.

Further, the transfer destination table does not need to directly associate the identifier associated with the sender terminal with the transfer destination data server. For example, the data collection server 2024 maintains a management table that stores tag information assigned to each unique identifier of the terminal 2021, and a transfer destination table that associates the tag information with transfer destination data servers. The data collection server 2024 may use the management table and the transfer destination table to determine the transfer destination data server based on the tag information. Here, the tag information is, for example, the type, model number, owner, group to which the terminal 2021 corresponds to the identifier, the group to which it belongs, or other management control information or service provision control information given to the identifier. Furthermore, instead of the identifier associated with the transmission source terminal 2021, a unique identifier for each sensor may be used in the transfer destination table. Further, the determination rule for the transfer destination server may be set from the client device 2026.

The data collection server 2024 may determine a plurality of data servers as transfer destinations and may transfer the received location-related data to the plurality of data servers. According to this configuration, for example, when automatically backing up location-related data or in order to use location-related data in common with different services, location-related data is sent to the data server that provides each service. If necessary, data can be transferred as intended by changing the settings for the data collection server 2024. As a result, the number of man-hours required for system construction and modification can be reduced compared to the case where the transmission destination of location-related data is set in each individual terminal 2021.

In response to the transfer request signal received from the data server, the data collection server 2024 registers the data server specified in the transfer request signal as a new transfer destination, and transfers subsequently received location-related data to the data server. It's okay.

The data collection server 2024 stores the location-related data received from the terminal 2021 in a recording device, and in response to the transmission request signal received from the terminal 2021 or the data server, transmits the location-related data specified in the transmission request signal to the requestor. The information may be sent to the terminal 2021 or the data server.

The data collection server 2024 determines whether the location-related data can be provided to the requesting data server or terminal 2021, and if it is determined that the location-related data can be provided, transfers or transmits the location-related data to the requesting data server or terminal 2021. You may do so.

When receiving a request for current location-related data from the client device 2026, the data collection server 2024 requests the terminal 2021 to send the location-related data, even if it is not the timing for the terminal 2021 to send the location-related data. 2021 may transmit location-related data in response to the transmission request.

In the above description, it is assumed that the terminal 2021 transmits location information data to the data collection server 2024. However, the data collection server 2024 has functions for collecting location-related data from the terminal 2021, such as a function to manage the terminal 2021. The terminal 2021 may also include functions necessary for the terminal 2021 or functions used when collecting location-related data from the terminal 2021.

The data collection server 2024 may have a function of transmitting a data request signal requesting transmission of location information data to the terminal 2021 and collecting location-related data.

In the data collection server 2024, management information such as an address for communicating with the terminal 2021 to be data collected or an identifier unique to the terminal 2021 is registered in advance. The data collection server 2024 collects location-related data from the terminal 2021 based on registered management information. The management information may include information such as the type of sensor included in the terminal 2021, the number of sensors included in the terminal 2021, and the communication method supported by the terminal 2021.

The data collection server 2024 may collect information such as the operating state or current location of the terminal 2021 from the terminal 2021.

Registration of management information may be performed from the client device 2026, or registration processing may be started when the terminal 2021 sends a registration request to the data collection server 2024. The data collection server 2024 may have a function of controlling communication with the terminal 2021.

Communication between the data collection server 2024 and the terminal 2021 is performed using a dedicated line provided by a service provider such as an MNO (Mobile Network Operator) or an MVNO (Mobile Virtual Network Operator), or a VPN (Virtual Private). network) It may also be a virtual dedicated line. According to this configuration, communication between the terminal 2021 and the data collection server 2024 can be performed safely.

The data collection server 2024 may have a function of authenticating the terminal 2021 or a function of encrypting data sent to and received from the terminal 2021. Here, the authentication process of the terminal 2021 or the data encryption process is performed using an identifier unique to the terminal 2021 or a terminal group including a plurality of terminals 2021, which is shared in advance between the data collection server 2024 and the terminal 2021. This is done using a unique identifier etc. This identifier is, for example, IMSI (International Mobile Subscriber Identity), which is a unique number stored in a SIM (Subscriber Identity Module) card. The identifier used for authentication processing and the identifier used for data encryption processing may be the same or different.

Authentication or data encryption processing between the data collection server 2024 and the terminal 2021 can be provided if both the data collection server 2024 and the terminal 2021 have the function to perform the processing, and relaying is possible. It does not depend on the communication method used by the communication device 2022. Therefore, a common authentication or encryption process can be used without considering which communication method the terminal 2021 uses, improving the convenience of system construction for the user. However, not depending on the communication method used by the communication device 2022 that performs the relay means that it is not essential to change according to the communication method. In other words, for the purpose of improving transmission efficiency or ensuring safety, authentication or data encryption processing between the data collection server 2024 and the terminal 2021 may be switched depending on the communication method used by the relay device.

The data collection server 2024 may provide the client device 2026 with a UI for managing data collection rules such as the type of location-related data collected from the terminal 2021 and the data collection schedule. This allows the user to use the client device 2026 to specify the terminal 2021 that collects data, as well as the time and frequency of data collection. Further, the data collection server 2024 may specify an area on a map for which data is to be collected, and may collect position-related data from the terminals 2021 included in the area.

When managing data collection rules on a terminal 2021 basis, the client device 2026 presents, for example, a list of terminals 2021 or sensors to be managed on the screen. The user sets whether or not data should be collected or the collection schedule for each item in the list.

When specifying an area on a map for which data is to be collected, the client device 2026 presents, for example, a two-dimensional or three-dimensional map of the area to be managed on the screen. The user selects an area on the displayed map to collect data. The area selected on the map may be a circular or rectangular area centered on a point specified on the map, or may be a circular or rectangular area that can be specified by a drag operation. The client device 2026 may also select an area in preset units such as a city, an area within a city, a block, or a major road. Also, instead of specifying the area using a map, the area may be set by inputting numerical values of latitude and longitude, or the area may be selected from a list of candidate areas derived based on the input text information. may be done. The text information is, for example, the name of a region, city, or landmark.

In addition, by specifying one or more terminals 2021 by the user and setting conditions such as being within a 100 meter radius of the terminal 2021, data collection can be performed while dynamically changing the specified area. Good too.

Further, when the client device 2026 includes a sensor such as a camera, an area on the map may be specified based on the position of the client device 2026 in real space obtained from sensor data. For example, the client device 2026 estimates its own position using sensor data, and uses the data to map an area within a predetermined distance from a point on the map corresponding to the estimated position or a distance specified by the user. may be specified as the area to be collected. Further, the client device 2026 may specify the sensing area of the sensor, that is, the area corresponding to the acquired sensor data, as the area for collecting data. Alternatively, the client device 2026 may specify an area based on a position corresponding to sensor data specified by the user as an area for data collection. The region or position on the map corresponding to the sensor data may be estimated by the client device 2026 or by the data collection server 2024.

When specifying an area on a map, the data collection server 2024 identifies the terminals 2021 within the specified area by collecting the current location information of each terminal 2021, and assigns a location to the specified terminal 2021. The transmission of related data may also be requested. Furthermore, instead of the data collection server 2024 specifying the terminal 2021 within the area, the data collection server 2024 sends information indicating the specified area to the terminal 2021, and if the terminal 2021 itself is within the specified area. The location-related data may be transmitted if it is determined that the location is within the designated area.

The data collection server 2024 transmits to the client device 2026 data such as a list or map for providing the above-mentioned UI (User Interface) in an application executed by the client device 2026. Data collection server 2024 may send application programs to client device 2026 as well as data such as lists or maps. Furthermore, the above-described UI may be provided as content created in HTML or the like that can be displayed on a browser. Note that some data such as map data may be provided from a server other than the data collection server 2024, such as the map server 2025.

When the client device 2026 receives an input notifying that the input has been completed, such as when the user presses a settings button, the client device 2026 transmits the input information to the data collection server 2024 as setting information. Based on the setting information received from the client device 2026, the data collection server 2024 transmits a signal to each terminal 2021 to notify the request for location-related data or the collection rule for location-related data, and collects the location-related data. conduct.

Next, an example will be described in which the operation of the terminal 2021 is controlled based on additional information added to three-dimensional or two-dimensional map data.

In this configuration, object information indicating the position of a power feeding unit such as a power feeding antenna or a power feeding coil for wireless power feeding buried in a road or a parking lot is included in the three-dimensional data or is associated with the three-dimensional data, It is provided to a terminal 2021, such as a car or a drone.

The vehicle or drone that has acquired the object information for charging automatically drives itself so that the position of the charging part such as the charging antenna or charging coil that the vehicle has is opposite to the area indicated by the object information. Move the position and start charging. In the case of vehicles or drones that do not have automatic driving functions, the driver or operator is shown the direction in which to move or the operations to be performed using images or sounds displayed on the screen. Ru. Content that causes the driver to stop driving or maneuvering when it is determined that the position of the charging unit calculated based on the estimated self-position is within the area indicated by the object information or within a predetermined distance from the area. The image or sound to be presented is switched to , and charging is started.

Furthermore, the object information may not be information indicating the position of the power feeding unit, but may be information indicating an area where charging efficiency equal to or higher than a predetermined threshold value can be obtained when the charging unit is placed within the area. The position of object information may be expressed as the center point of the area indicated by the object information, or may be expressed as an area or line in a two-dimensional plane, or as an area, line, or plane in three-dimensional space. good.

According to this configuration, it is possible to grasp the position of the power feeding antenna that cannot be grasped using LiDER sensing data or images taken with a camera, so it is possible to grasp the position of the power feeding antenna that cannot be grasped using LiDER sensing data or images taken with a camera. Positioning with the feeding antenna can be performed with higher precision. As a result, it is possible to shorten the charging speed during wireless charging and improve charging efficiency.

The object information may be an object other than the feeding antenna. For example, the three-dimensional data includes the location of an AP for millimeter wave wireless communication as object information. As a result, the terminal 2021 can know the position of the AP in advance, and can start communication by directing the beam direction toward the object information. As a result, it is possible to improve communication quality by increasing transmission speed, shortening the time required to start communication, and extending the period during which communication is possible.

The object information may include information indicating the type of object corresponding to the object information. In addition, the object information should be executed by the terminal 2021 when the terminal 2021 is included within a region in real space corresponding to the position on the three-dimensional data of the object information or within a predetermined distance from the region. It may also include information indicating processing.

The object information may be provided by a server different from the server that provides the three-dimensional data. When object information is provided separately from three-dimensional data, object groups storing object information used in the same service may be provided as separate data depending on the type of target service or target device. .

The three-dimensional data used in combination with the object information may be WLD point cloud data or SWLD feature point data.

Although the server, client device, etc. according to the embodiment of the present disclosure have been described above, the present disclosure is not limited to this embodiment.

Furthermore, each processing unit included in the server, client device, etc. according to the above embodiments is typically realized as an LSI, which is an integrated circuit. These may be integrated into one chip individually, or may be integrated into one chip including some or all of them.

Further, circuit integration is not limited to LSI, and may be realized using a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may be used.

Furthermore, in each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

Further, the present disclosure may be implemented as a three-dimensional data creation method executed by a server, a client device, and the like.

Furthermore, the division of functional blocks in the block diagram is just an example; multiple functional blocks can be realized as one functional block, one functional block can be divided into multiple functional blocks, or some functions can be moved to other functional blocks. It's okay. Further, functions of a plurality of functional blocks having similar functions may be processed in parallel or in a time-sharing manner by a single piece of hardware or software.

Further, the order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and may be in an order other than the above. Further, some of the above steps may be executed simultaneously (in parallel) with other steps.

Although the server, client device, etc. according to one or more aspects have been described above based on the embodiment, the present disclosure is not limited to this embodiment. Unless departing from the spirit of the present disclosure, various modifications that can be thought of by those skilled in the art to this embodiment, and forms constructed by combining components of different embodiments are also within the scope of one or more aspects. may be included within.

The present disclosure can be applied to client devices and the like.

100, 400 Three-dimensional data encoding device 101, 201, 401, 501 Acquisition unit 102, 402 Encoding area determining unit 103 Division unit 104, 644 Encoding unit 111, 607 Three-dimensional data 112, 211, 413, 414, 511 , 634 encoded three-dimensional data 200, 500 three-dimensional data decoding device 202 decoding start GOS determining section 203 decoding SPC determining section 204, 625 decoding section 212, 512, 513 decoding three-dimensional data 403 SWLD extracting section 404 WLD encoding section 405 SWLD encoding unit 411 Input 3D data 412 Extracted 3D data 502 Header analysis unit 503 WLD decoding unit 504 SWLD decoding unit 600 Own vehicle 601 Surrounding vehicles 602, 605 Sensor detection range 603, 606 Area 604 Occlusion area 620, 620A 3D Data creation device 621, 641 Three-dimensional data creation unit 622 Request range determination unit 623 Search unit 624, 642 Receiving unit 626 Combination unit 627 Detection area determination unit 628 Surrounding situation detection unit 629 Autonomous operation control unit 631, 651 Sensor information 632 First 3D data 633 Request range information 635 2nd 3D data 636 3rd 3D data 637 Request signal 638 Transmission data 639 Surrounding situation detection result 640, 640A 3D data transmission device 643 Extraction unit 645 Transmission unit 646 Transmission possibility determination unit 652 Fifth three-dimensional data 654 Sixth three-dimensional data 700 Three-dimensional information processing device 701 Three-dimensional map acquisition unit 702 Self-vehicle detection data acquisition unit 703 Abnormal case determination unit 704 Corrective action determination unit 705 Operation control unit 711 Three-dimensional map 712 Self Vehicle detection three-dimensional data 801 Vehicle 802 Space 810 Three-dimensional data creation device 811 Data reception section 812, 819 Communication section 813 Reception control section 814, 821 Format conversion section 815 Sensor 816 Three-dimensional data creation section 817 Three-dimensional data synthesis section 818 Tertiary Original data storage unit 820 Transmission control unit 822 Data transmission unit 831, 832, 834, 835, 836, 837 Three-dimensional data 833 Sensor information 901 Server 902, 902A, 902B, 902C Client device 1011, 1111 Data reception unit 1012, 1020, 1112, 1120 Communication section 1013, 1113 Reception control section 1014, 1019, 1114, 1119 Format conversion section 1015 Sensor 1016, 1116 Three-dimensional data creation section 1017 Three-dimensional image processing section 1018, 1118 Three-dimensional data storage section 1021, 1121 Transmission control Sections 1022, 1122 Data transmission section 1031, 1032, 1135 Three-dimensional map 1033, 1037, 1132 Sensor information 1034, 1035, 1134 Three-dimensional data 1117 Three-dimensional data synthesis section 1201 Three-dimensional map compression/decoding processing section 1202 Sensor information compression/ Decoding processing unit 1211 Three-dimensional map decoding processing unit 1212 Sensor information compression processing unit 2001 Server 2002, 2002A, 2002B Client device 2011 Sensor information acquisition unit 2012 Storage unit 2013 Data transmission permission determination unit 2021, 2021A, 2021B Terminal 2022, 2022A, 2022B Communication device 2023 Network 2024 Data collection server 2025 Map server 2026 Client device

Claims (16)

An information transmission method in a client device installed in a mobile object, the method comprising:
acquiring first sensor information indicating a surrounding situation of the mobile body obtained by a first sensor mounted on the mobile body, and storing the first sensor information in a storage unit included in the client device;
Obtaining second sensor information, which is obtained by a second sensor mounted on the moving body and has a smaller amount of data than the first sensor information,
determining whether the client device can connect to the access point capable of high-speed communication by directing the beam directionality in the direction of the position of the access point capable of high-speed communication indicated in the map data ;
If the client device determines that it is connectable to the access point capable of high-speed communication, transmitting the first sensor information to the server;
If it is determined that the client device is not connectable to the access point capable of high-speed communication, transmitting the second sensor information to the server;
How to send information. An information transmission method in a client device installed in a mobile object, the method comprising:
acquiring first sensor information obtained by a first sensor mounted on the mobile body, and storing the first sensor information in a storage unit included in the client device;
acquiring second sensor information obtained by a second sensor mounted on the mobile body;
If it is determined that the client device can connect to the access point using the millimeter wave communication method by directing the beam directionality in the direction of the location of the access point shown in the map data , the first sensor information to the server via the access point,
transmitting the second sensor information to the server using a communication method different from the millimeter wave communication method;
How to send information. The information transmission method according to claim 1 or 2, wherein the first sensor is a LiDAR sensor, and the second sensor is an image sensor. Further, determining whether the client device can use a plurality of networks for transmitting the first sensor information or the second sensor information,
When the plurality of networks can be used to transmit sensor information, the first sensor information or the second sensor information is transmitted from the plurality of networks based on the data amount of the first sensor information or the second sensor information. Select the network you want to use,
The information transmission method according to claim 1, wherein the first sensor information or the second sensor information is transmitted to the server using the selected network. The information transmission method according to claim 1, further comprising transmitting header information indicating characteristics of the first sensor information to the server before transmitting the first sensor information. The information transmission method further includes:
creating three-dimensional data around the moving object from the first sensor information;
The information transmission method according to any one of claims 1 to 5, wherein the self-position of the mobile object is estimated using the created three-dimensional data. The information transmission method further includes:
Sending a three-dimensional map transmission request to the server,
receiving the three-dimensional map from the server;
The information transmission method according to claim 6, wherein in estimating the self-position, the self-position is estimated using the three-dimensional data and the three-dimensional map. The first sensor information includes acquisition location information indicating the position of the mobile object or the first sensor when the first sensor information was acquired by the first sensor. How to send information as described in . The information transmission method according to claim 1, wherein the first sensor information includes acquisition time information indicating a time when the first sensor information was acquired by the first sensor. The information transmission method further includes:
Obtaining time information from the server,
The information transmission method according to claim 9, wherein the acquired time information is generated using the acquired time information. The information transmission method further includes:
receiving a sensor information transmission request including specification information specifying a location and time from the server;
If the first sensor information obtained at the location and time indicated by the specified information is stored in the storage unit, and the mobile object exists in an environment where the first sensor information can be transmitted to the server. If determined, the information transmission method according to any one of claims 1 to 10, further comprising transmitting the first sensor information obtained at the location and time indicated by the designated information to the server. The information transmission method further includes:
The information transmission method according to any one of claims 1 to 11, further comprising deleting the first sensor information that has already been transmitted to the server from the storage unit. The information transmission method further includes:
The difference between the position of the moving object or the first sensor when the first sensor information was acquired by the first sensor and the current position of the moving object or the first sensor is a predetermined distance. 13. The information transmitting method according to claim 1, wherein the first sensor information is deleted from the storage unit when the number of sensor information exceeds the first sensor information. The information transmission method further includes:
The first sensor information is deleted from the storage unit when a difference between a time when the first sensor information was acquired by the first sensor and a current time exceeds a predetermined time. The information transmission method according to any one of items 1 to 13. A client device mounted on a mobile object,
a processor;
Equipped with memory,
The processor uses the memory to:
acquiring first sensor information indicating a surrounding situation of the mobile body obtained by a first sensor mounted on the mobile body, and storing the first sensor information in a storage unit included in the client device;
Obtaining second sensor information, which is obtained by a second sensor mounted on the moving body and has a smaller amount of data than the first sensor information,
determining whether the client device can connect to the access point capable of high-speed communication by directing the beam directionality in the direction of the position of the access point capable of high-speed communication indicated in the map data ;
If the client device determines that it is connectable to the access point capable of high-speed communication, transmitting the first sensor information to the server;
If it is determined that the client device is not connectable to the access point capable of high-speed communication, transmitting the second sensor information to the server;
Client device. A client device mounted on a mobile object,
a processor;
Equipped with memory,
The processor uses the memory to:
An information transmission method in a client device installed in a mobile object, the method comprising:
acquiring first sensor information obtained by a first sensor mounted on the mobile body, and storing the first sensor information in a storage unit included in the client device;
acquiring second sensor information obtained by a second sensor mounted on the mobile body;
If it is determined that the client device can connect to the access point using the millimeter wave communication method by directing the beam directionality in the direction of the location of the access point that is capable of high-speed communication and is shown in the map data, transmitting first sensor information to a server via the access point;
transmitting the second sensor information to the server using a communication method different from the millimeter wave communication method;
Client device.

Images