JP2021086342A - Information processing device and information processing method - Google Patents

Abstract

To provide an information processing device and an information processing method which enable a user to easily grasp a situation around a vehicle.SOLUTION: An information processing device comprises an acquisition unit, a detection unit and a generation unit. The acquisition unit acquires vehicle information including a picked-up image which is picked up in a vehicle. The detection unit detects a positional relation between the vehicle and a surrounding object which is present around the vehicle, on the basis of the vehicle information acquired by the acquisition unit. The generation unit generates an overhead view image including images of the vehicle and the surrounding object in the positional relation detected by the detection unit on the basis of the vehicle information, the overhead view image being in predetermined timing in accordance with a road situation of a road on which the vehicle travels.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing apparatus and an information processing method.

Conventionally, a technique has been known in which a drive recorder mounted on a vehicle is used to image and record the periphery of the vehicle. Further, various techniques have been proposed in which the user is made to grasp the situation around the vehicle by providing the captured image captured by the drive recorder to the user (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2011-227571

However, there is room for improvement in the prior art in that the user can easily grasp the situation around the vehicle.

The present invention has been made in view of the above, and an object of the present invention is to provide an information processing device and an information processing method capable of allowing a user to easily grasp the situation around a vehicle.

In order to solve the above problems and achieve the object, the present invention includes an acquisition unit, a detection unit, and a generation unit in the information processing apparatus. The acquisition unit acquires vehicle information including the captured image captured in the vehicle. The detection unit detects the positional relationship between the vehicle and peripheral objects existing around the vehicle based on the vehicle information acquired by the acquisition unit. The generation unit is a bird's-eye view image including images of the vehicle and the peripheral objects in the positional relationship detected by the detection unit based on the vehicle information, and depends on the road condition of the road on which the vehicle travels. The bird's-eye view image is generated at a predetermined timing.

According to the present invention, the user can easily grasp the situation around the vehicle.

FIG. 1 is a diagram showing an outline of an information processing method according to an embodiment. FIG. 2 is a block diagram showing a configuration example of an information processing system including an information processing device. FIG. 3 is a block diagram showing a configuration example of an in-vehicle device. FIG. 4 is a block diagram showing a configuration example of the information processing device. FIG. 5 is a diagram showing an example of vehicle information. FIG. 6 is a diagram showing an example of information of the animation image. FIG. 7 is a diagram showing an example of information on the bird's-eye view image. FIG. 8 is a diagram showing an example of a bird's-eye view image. FIG. 9 is a flowchart showing a processing procedure executed by the information processing apparatus.

Hereinafter, embodiments of the information processing apparatus and information processing method disclosed in the present application will be described in detail with reference to the accompanying drawings. The present invention is not limited to the embodiments shown below.

<1. Overview of information processing method>
Hereinafter, first, an outline of the information processing method according to the embodiment will be described with reference to FIG. FIG. 1 is a diagram showing an outline of an information processing method according to an embodiment.

The information processing method according to the embodiment is executed by, for example, the information processing device 50 included in the information processing system 1. More specifically, as shown in FIG. 1, the information processing system 1 includes an in-vehicle device 10 and the above-mentioned information processing device 50.

The in-vehicle device 10 is mounted on the vehicle C1. The in-vehicle device 10 includes, for example, a drive recorder. The in-vehicle device 10 can image the periphery of the vehicle C1 and record the captured image.

By the way, in the prior art, the captured image captured by the in-vehicle device 10 is provided to, for example, the user of the vehicle C1. Specifically, for example, when a predetermined event occurs, such as when the vehicle C1 comes into contact with or is about to come into contact with a peripheral vehicle C2 existing around the vehicle C1, the captured image is provided to provide an image at the time of the event occurrence. The user was made to grasp the situation around the vehicle C1. In FIG. 1, a case where the vehicle C1 and the peripheral vehicle C2 enter the intersection D and the vehicle C1 traveling straight at the intersection D and the peripheral vehicle C2 turning right at the intersection D come into contact with each other will be described as an example.

However, it has been difficult for the user to sufficiently grasp the situation around the vehicle C1 at the time of the event occurrence or before the event occurrence only by the captured image captured by the in-vehicle device 10. Further, it was difficult for the user to identify the cause of the event unless the situation around the vehicle C1 was sufficiently grasped. Specifically, whether the cause of the event was on the vehicle C1 side or the peripheral vehicle C2 side. It was difficult to identify if it was in.

Therefore, the information processing device 50 according to the present embodiment is configured so that the user can easily grasp the situation around the vehicle C1.

Specifically, the information processing device 50 is a device capable of communicating various types of information with the in-vehicle device 10. As the information processing device 50, for example, a personal computer, a server, or the like can be used, but the information processing device 50 is not limited thereto.

Note that FIG. 1 shows a configuration example in which the vehicle-mounted device 10 and the information processing device 50 are separate bodies, but the present invention is not limited to this, and for example, the vehicle-mounted device 10 and the information processing device 50 are integrated into the vehicle-mounted device 10. May function as the information processing device 50.

The information processing device 50 acquires vehicle information including the captured image captured by the vehicle C1 (step S1). Here, the captured image includes the peripheral vehicle C2, the traffic lights E1 and E2 around the vehicle C1 (for example, in front), the stop lines F1 and F2 of the road on which the vehicle C1 and the peripheral vehicle C2 travel, and the like around the vehicle C1. It is assumed that existing peripheral objects are included. Further, the captured image may include position information indicating the position where the captured image was captured, date and time information indicating the date and time when the captured image was captured, and the like.

As shown by an imaginary line in FIG. 1, the in-vehicle device 10 may also be mounted on the peripheral vehicle C2. In such a case, the information processing device 50 can acquire vehicle information including a captured image from the in-vehicle device 10 of the peripheral vehicle C2. That is, the in-vehicle device 10 mounted on the peripheral vehicle C2 captures the periphery of the peripheral vehicle C2 and records the captured image (hereinafter, may be referred to as "peripheral vehicle captured image"). Then, the information processing device 50 can acquire vehicle information including a peripheral vehicle captured image from the in-vehicle device 10 of the peripheral vehicle C2.

It is assumed that the peripheral vehicle captured image includes peripheral objects existing around the peripheral vehicle C2, such as the traffic lights E1 and E2 around the vehicle C1 and the peripheral vehicle C2, and the road stop lines F1 and F2. Further, the peripheral vehicle captured image may include position information indicating the position where the peripheral vehicle captured image was captured, date and time information indicating the date and time when the image was captured, and the like, as in the captured image.

In the above description, the information processing device 50 acquires vehicle information from the vehicle-mounted device 10 of the vehicle C1 and the vehicle-mounted device 10 of the peripheral vehicle C2, respectively, but the information processing device 50 is not limited to this. That is, for example, the in-vehicle device 10 of the peripheral vehicle C2 transmits the vehicle information including the captured image of the peripheral vehicle to the in-vehicle device 10 of the vehicle C1 via wireless communication such as inter-vehicle communication. Then, the information processing device 50 acquires the vehicle information including the peripheral vehicle captured image and the vehicle information including the captured image captured by the vehicle-mounted device 10 of the vehicle C1 from the vehicle-mounted device 10 of the vehicle C1 together. May be good. In this case, the vehicle information of the vehicle C1 and the peripheral vehicles may be recorded in the memory card of the vehicle-mounted device 10 of the vehicle C1, and the information processing device 50 may acquire each vehicle information from the memory card.

Next, the information processing device 50 detects the positional relationship between the vehicle C1 and the peripheral objects existing around the vehicle C1 based on the acquired vehicle information (step S2). For example, the information processing device 50 analyzes the captured image and the peripheral vehicle captured image included in the vehicle information, and analyzes the vehicle C1 and the peripheral objects (here, for example, the peripheral vehicle C2, the traffic lights E1, E2, the stop lines F1, F2, etc.). Detects the relative position with and the positional relationship such as the distance from the vehicle C1 to the surrounding objects.

Next, the information processing device 50 generates a bird's-eye view image A based on the vehicle information (step S3). The bird's-eye view image A is an image (for example, an animation image) when the place where the captured image included in the vehicle information is captured (here, the intersection D) is viewed from a predetermined virtual viewpoint. Specifically, in the example shown in FIG. 1, the bird's-eye view image A is an image when the intersection D is viewed from above.

For example, the information processing device 50 captures images of the detected vehicle C1 and peripheral objects (here, for example, peripheral vehicles C2, traffic lights E1, E2, stop lines F1, F2, etc.) based on vehicle information. Generate a bird's-eye view image A including.

In the bird's-eye view image A, as the image of the vehicle C1 and the peripheral objects, for example, an image prepared in advance in the information processing device 50, in other words, an image stored in advance in the information processing device 50 may be used or captured. An image drawn by the information processing apparatus 50 based on an analysis result such as an image may be used. Further, in the bird's-eye view image A, the entire image of the intersection D including the sidewalk, the crosswalk, and the like may be, for example, an image prepared in advance in the information processing device 50, or the information processing device 50 may be used as the captured image. An image acquired from a map information server (external server, not shown) based on the included position information may be used, or an image drawn based on an analysis result such as an captured image may be used.

Further, the information processing device 50 generates a bird's-eye view image A at a predetermined timing according to the road condition of the road on which the vehicle C1 travels. As a result, the user can easily grasp the situation around the vehicle C1 by confirming (visually) the generated bird's-eye view image A. In addition, the user can identify the cause of the occurrence of a predetermined event by grasping the situation around the vehicle C1.

That is, when a predetermined event occurs, for example, when the vehicle C1 and the peripheral vehicle C2 come into contact with each other, the user can use the vehicle C1 at the timing when the road condition of the road on which the vehicle C1 travels changes before the event occurs. By understanding the surrounding situation, it may be possible to identify the cause of the event.

Therefore, in the present embodiment, the above-mentioned road condition includes, for example, the state of the traffic lights E1 and E2 set on the road on which the vehicle C1 travels. Here, the predetermined timing according to the road condition is set to, for example, the timing when the road condition changes, in other words, the timing when the states of the traffic lights E1 and E2 change.

In the example of FIG. 1, the predetermined timing is the timing when the state of the traffic light E1 changes to a red light. Further, in the bird's-eye view image A shown in FIG. 1, the peripheral vehicle C2 makes a right turn at the intersection D, and the vehicle C1 is entering the intersection D even though the traffic light E1 is in a red light. Shown. The bird's-eye view image A will be described in detail later with reference to FIG. 8 and the like.

Next, the information processing device 50 outputs the generated bird's-eye view image A (step S4). For example, the information processing device 50 may output the bird's-eye view image A to the display unit 52 (described later) included in the information processing device 50 itself, or may output the bird's-eye view image A to the in-vehicle device 10 for display.

As a result, the user can easily grasp the situation around the vehicle C1 by checking the generated bird's-eye view image A. Further, the user can grasp the situation around the vehicle C1 such that the vehicle C1 enters the intersection D at the timing when the state of the traffic light E1 changes to a red light, so that the cause of the occurrence of the predetermined event is the vehicle. It is also possible to specify that it is on the C1 side.

<2. Configuration of information processing system including information processing device>
Next, the configuration of the information processing system 1 including the information processing device 50 according to the embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing a configuration example of the information processing system 1 including the information processing device 50. In the block diagram of FIG. 2 and the like, only the components necessary for explaining the features of the present embodiment are represented by functional blocks, and the description of general components is omitted.

In other words, each component shown in the block diagram such as FIG. 2 is a functional concept and does not necessarily have to be physically configured as shown in the figure. For example, the specific form of distribution / integration of each functional block is not limited to the one shown in the figure, and all or part of the functional blocks are functionally or physically distributed in arbitrary units according to various loads and usage conditions. -It is possible to integrate and configure.

As shown in FIG. 2, the information processing system 1 includes the above-mentioned in-vehicle device 10, the information processing device 50, and a surveillance camera 100, which are communicably connected via a communication network N such as an Internet network. To. The in-vehicle device 10, the information processing device 50, and the surveillance camera 100 are not limited to the above-mentioned communication network N, and may be communicably connected via other types of wireless communication or wired communication. Further, the in-vehicle devices 10 may be connected to each other so as to be able to communicate with each other via wireless communication such as vehicle-to-vehicle communication.

The surveillance camera 100 is a camera installed on the road on which the vehicle C1 and the peripheral vehicle C2 travel. The surveillance camera 100 can image the surroundings and record the captured image. The captured image captured by the surveillance camera 100 may include a vehicle C1 traveling on a road, a peripheral vehicle C2, traffic lights E1 and E2, stop lines F1 and F2 (see FIG. 1), and the like. Then, the surveillance camera 100 can transmit the vehicle information including the captured image to the information processing device 50. Further, the captured image of the surveillance camera 100 may include position information indicating the position where the captured image was captured (that is, the position of the surveillance camera 100), date and time information indicating the date and time when the captured image was captured, and the like. ..

Although the information processing system 1 described above includes the surveillance camera 100, the information processing system 1 is not limited to this, and the configuration may not include the surveillance camera 100. Further, although FIG. 2 shows an example in which the number of in-vehicle devices 10 is a plurality of devices, the present invention is not limited to this, and the number of in-vehicle devices 10 may be one.

Next, the configurations of the in-vehicle device 10 and the information processing device 50 described above will be specifically described.

<3. In-vehicle device configuration>
FIG. 3 is a block diagram showing a configuration example of the in-vehicle device 10. As shown in FIG. 3, the vehicle-mounted device 10 includes a communication unit 11, a vehicle-mounted camera 12, a vehicle-mounted sensor group 13, and a control device 20.

The communication unit 11 is a communication interface that connects to the communication network N so that bidirectional communication is possible, and transmits / receives information to / from the information processing device 50 or the like.

The in-vehicle camera 12 is, for example, a camera of a drive recorder, and images the periphery of the vehicle C1. The in-vehicle camera 12 outputs the captured image to the control device 20.

The in-vehicle sensor group 13 includes, for example, a vehicle speed sensor that detects the speed of the vehicle C1, a GPS receiver that detects position information indicating the position of the vehicle C1 based on a signal from a GPS (Global Positioning System) satellite, and the like. However, it is not limited to these. The in-vehicle sensor group 13 outputs the vehicle speed information indicating the speed of the obtained vehicle C1 and the position information of the vehicle C1 to the control device 20.

The control device 20 includes a control unit 30 and a storage unit 40. The control unit 30 is a microcomputer including a processing unit 31 and a transmission unit 32, and has a CPU (Central Processing Unit) and the like.

The storage unit 40 is a storage unit composed of storage devices such as a non-volatile memory, a data flash, and a hard disk drive. The storage unit 40 stores the captured image 41, the vehicle speed information 42, various programs, and the like.

The captured image 41 is a captured image captured by the vehicle-mounted camera 12. The vehicle speed information 42 is information on the vehicle speed obtained by the vehicle speed sensor of the vehicle-mounted sensor group 13. For example, in the vehicle speed information 42, since the vehicle speed information of the vehicle C1 is stored in time series, the vehicle speed information 42 includes the vehicle speed information when the captured image is captured.

The processing unit 31 of the control unit 30 acquires the captured image transmitted from the vehicle-mounted camera 12 and stores it in the storage unit 40 as the captured image 41. At this time, the processing unit 31 may store, for example, the position information transmitted from the vehicle-mounted sensor group 13 in association with the captured image as position information indicating the position where the captured image is captured. Further, the processing unit 31 may store date and time information indicating the date and time when the captured image was captured in association with the captured image.

Further, the processing unit 31 acquires the vehicle speed information transmitted from the vehicle-mounted sensor group 13 when the captured image is captured, and stores the vehicle speed information 42 in association with the captured image in the storage unit 40.

The transmission unit 32 can transmit the captured image 41 and the vehicle speed information 42 stored in the storage unit 40 to the information processing device 50 via the communication unit 11.

<4. Information processing device configuration>
Next, the information processing device 50 will be described with reference to FIG. FIG. 4 is a block diagram showing a configuration example of the information processing device 50. As shown in FIG. 4, the information processing device 50 includes a communication unit 51, a display unit 52, and a control device 60.

The communication unit 51 is a communication interface that connects to the communication network N so that bidirectional communication is possible, and transmits / receives information to / from the in-vehicle device 10 and the surveillance camera 100. The display unit 52 is a display capable of displaying the above-mentioned bird's-eye view image A and the like.

The control device 60 includes a control unit 70 and a storage unit 80. The control unit 70 includes an acquisition unit 71, a detection unit 72, a generation unit 73, and an output unit 74.

Here, the control device 60 includes, for example, a computer having a CPU, a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk drive, an input / output port, and various circuits.

The CPU of the computer functions as the acquisition unit 71, the detection unit 72, the generation unit 73, and the output unit 74 of the control unit 70 by reading and executing the program stored in the ROM, for example.

Further, at least a part or all of the acquisition unit 71, the detection unit 72, the generation unit 73, and the output unit 74 of the control unit 70 are used by hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). It can also be configured.

Further, the storage unit 80 is a storage unit composed of storage devices such as a non-volatile memory, a data flash, and a hard disk drive. The storage unit 80 stores vehicle information 81, a generation image 82, a bird's-eye view image 83, various programs, and the like.

The vehicle information 81 includes a captured image. For example, the vehicle information 81 includes a captured image captured by the vehicle C1, a peripheral vehicle captured image captured by the peripheral vehicle C2, a captured image captured by the surveillance camera 100, and the like.

Further, the vehicle information 81 may include vehicle speed information, position information, date and time information, and the like when the captured image and the captured image of the surrounding vehicle are captured. Since the captured image is used to generate the bird's-eye view image at the predetermined timing, the vehicle speed information when the captured image is captured is the vehicle speed information indicating the speed of the vehicle C1 and the peripheral vehicle C2 at the predetermined timing. It can be said that.

Here, the vehicle information 81 will be described with reference to FIG. FIG. 5 is a diagram showing an example of vehicle information 81. As shown in FIG. 5, the vehicle information 81 includes items such as "vehicle information ID", "information acquisition source", "image information", "position information", "date and time information", and "vehicle speed information". , The information of each of these items is associated with each other.

The "vehicle information ID" is identification information that identifies vehicle information. The "information acquisition source" is information indicating an acquisition source of various information such as image information and vehicle speed information. Here, the "information acquisition source" includes a vehicle C1, a peripheral vehicle C2, a surveillance camera 100, and the like.

"Image information" is information on a captured image. In the example shown in FIG. 5, for convenience, the "image information" is described as an abstract description such as "captured image M1", but specific information is stored in the "captured image M1". .. Hereinafter, other information may also be described abstractly.

The "position information" is information indicating the position where the captured image or the captured image of the surrounding vehicle is captured. The "date and time information" is information indicating the date and time when the captured image or the image captured by the surrounding vehicle was captured. The "vehicle speed information" is information indicating the vehicle speed of the vehicle C1 when the captured image is captured and the vehicle speed of the peripheral vehicle C2 when the peripheral vehicle captured image is captured.

In the example shown in FIG. 5, the data identified by the vehicle information ID "J01" has the information acquisition source "vehicle", the image information "captured image M1", the position information "position N1", and the date and time information "date and time". "P1", indicating that the vehicle speed information is "vehicle speed Q1". In addition, the data identified by the vehicle information ID "K01" has the information acquisition source "peripheral vehicle", the image information "captured image M2", the position information "position N2", the date and time information "date and time P2", and the vehicle speed. It shows that the information is "vehicle speed Q2". Further, the data identified by the vehicle information ID "L01" has an information acquisition source of "surveillance camera", image information of "captured image M3", position information of "position N3", and date and time information of "date and time P3". It is shown that.

Note that FIG. 5 shows an example in which vehicle speed information is not included when the information acquisition source is a “surveillance camera”, but the present invention is not limited to this, and the vehicle C1 is not limited to this, for example, from the analysis processing of the captured image M3. When the vehicle speed of the above and the vehicle speed of the peripheral vehicle C2 can be obtained, the obtained vehicle speed may be included as the vehicle speed information.

Returning to the description of FIG. 4, the generation image 82 is the information of the image used for generating the bird's-eye view image. For example, the generation image 82 is an image of the vehicle C1 when viewed from a predetermined virtual viewpoint (for example, above) and an image of peripheral objects (for example, peripheral vehicles C2, traffic lights E1, E2, stop lines F1, F2, etc.). including. Further, the generation image 82 is a schematic or simplified image of the vehicle C1 and surrounding objects, but is not limited thereto.

The above-mentioned generation image 82 is prepared in advance before the generation of the bird's-eye view image. For example, the generation image 82 is an image stored in advance in the storage unit 80, but is not limited to this, and may be an image acquired from an external server (not shown) before the generation of the bird's-eye view image. ..

Here, the generation image 82 will be described with reference to FIG. FIG. 6 is a diagram showing an example of information of the generation image 82. As shown in FIG. 6, the generation image 82 includes items such as “generation image ID” and “generation image information”, and the information of each of these items is associated with each other.

The "generated image ID" is identification information for identifying the generated image. "Generation image information" is information on the generation image. For example, the "generated image information" includes images of the vehicle C1 and surrounding objects when viewed from the above-mentioned predetermined virtual viewpoint.

In the "generated image information", there may be a plurality of types of vehicle images used as the peripheral vehicle C2 which is a peripheral object. That is, the peripheral vehicle C2 may be a vehicle of various types and sizes, such as a passenger car, a truck, a bus, a motorcycle, a small car, or a large car. Therefore, a plurality of vehicle images of various vehicle types and sizes are prepared in the "generated image information", and from among the prepared multiple types of vehicle images, peripheral vehicles are prepared according to the captured images and the like. The vehicle image of C2 may be appropriately selected.

In the example shown in FIG. 6, the data identified by the generation image ID "T01" indicates that the generation image information is "vehicle image U1". Further, the data identified by the generation image ID "T02" indicates that the generation image information is "vehicle image U2". Further, the data identified by the generation image ID "T11" indicates that the generation image information is "traffic light image V1". Further, the data identified by the generation image ID "T21" indicates that the generation image information is the "stop line image W1".

The image information for generation shown in FIG. 6 is merely an example and is not limited. That is, the generation image information may include information on various images related to the road on which the vehicle C1 travels, such as a road image, a lane image, a median strip image, a sidewalk image, and a pedestrian crossing image.

Returning to the description of FIG. 4, the bird's-eye view image 83 is information on the bird's-eye view image generated by the generation unit 73, as will be described later.

Here, the bird's-eye view image 83 will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of the information of the bird's-eye view image 83. As shown in FIG. 7, the bird's-eye view image 83 includes items such as “bird's-eye view image ID” and “bird's-eye view image”, and the information of each item is associated with each other.

The "overhead image ID" is identification information for identifying the bird's-eye view image. The "overhead image" is the information of the generated bird's-eye view image. In the example shown in FIG. 7, the data identified by the bird's-eye view image ID "X01" indicates that the bird's-eye view image is the "bird's-eye view image A1".

Returning to the description of FIG. 4, the acquisition unit 71 of the control unit 70 acquires various vehicle information from the vehicle-mounted device 10 of the vehicle C1 and the peripheral vehicle C2, the surveillance camera 100, and the like.

For example, the acquisition unit 71 acquires vehicle information including the captured image captured by the vehicle C1 from the vehicle-mounted device 10 mounted on the vehicle C1. For example, the acquisition unit 71 acquires vehicle information including a peripheral vehicle image captured by the peripheral vehicle C2 from the vehicle-mounted device 10 mounted on the peripheral vehicle C2. For example, the acquisition unit 71 acquires vehicle information including the captured image captured by the surveillance camera 100 from the surveillance camera 100. Then, the acquisition unit 71 stores the vehicle information including the acquired captured image in the storage unit 80 as the vehicle information 81 (see FIG. 5). Further, as described above, the captured image includes position information indicating the position where the captured image was captured, date and time information indicating the date and time when the captured image was captured, and the like, so that the acquisition unit 71 obtains the position information and the date and time information. It may be stored in the storage unit 80 in association with the corresponding captured image (see FIG. 5).

The acquisition unit 71 acquires vehicle speed information as vehicle information. For example, the acquisition unit 71 acquires vehicle speed information including vehicle speed information indicating the speeds of the vehicle C1 and peripheral vehicles C2 at a predetermined timing as vehicle information, and stores the vehicle speed information in the storage unit 80. At this time, the acquisition unit 71 may store the vehicle speed information in the storage unit 80 as the vehicle information 81 (see FIG. 5) in association with the corresponding captured image.

The detection unit 72 detects the positional relationship between the vehicle C1 and peripheral objects existing around the vehicle C1 (for example, peripheral vehicles C2, traffic lights E1, E2, stop lines F1, F2, etc., see FIG. 1).

For example, the detection unit 72 accesses the storage unit 80, reads the vehicle information 81, extracts the captured image and the peripheral vehicle captured image at the same date and time included in the vehicle information, and extracts the extracted captured image and the peripheral vehicle captured image. To analyze.

For example, the detection unit 72 analyzes the captured image and the captured image of the surrounding vehicle by various appropriate processes such as edge detection processing, filtering processing / connection processing of the detected edge, and image recognition processing using a model. , The position of the vehicle C1 and peripheral objects included in the captured image and the peripheral vehicle captured image can be detected. The processing used in the analysis in the detection unit 72 is not limited to the above.

Then, the detection unit 72 detects the relative position between the vehicle C1 and the peripheral object and the positional relationship such as the distance from the vehicle C1 to the peripheral object based on the analysis result of the captured image or the like. Then, the detection unit 72 outputs the information indicating the detected positional relationship to the generation unit 73.

The generation unit 73 generates the bird's-eye view image A. The output unit 74 outputs the bird's-eye view image A generated by the generation unit 73 to the display unit 52. The output unit 74 may output the bird's-eye view image A to the in-vehicle device 10 via the communication unit 51 and display it.

FIG. 8 is a diagram showing an example of the bird's-eye view image A. The bird's-eye view image A is a still image, but is not limited to this, and may be a moving image.

As shown in FIG. 8, for example, the generation unit 73 generates a bird's-eye view image A including images of the vehicle C1 and peripheral objects in a positional relationship detected by the detection unit 72 based on the vehicle information of the vehicle C1. The bird's-eye view image A is an image when the place where the captured image included in the vehicle information is captured (here, the intersection D) is viewed from a predetermined virtual viewpoint (for example, above (above)). Further, the bird's-eye view image A is an animation image prepared in advance and synthesizing images of the vehicle C1 and surrounding objects when viewed from a predetermined virtual viewpoint.

Hereinafter, the generation of the bird's-eye view image A will be described in detail. First, the generation unit 73 first stores the storage unit 80 based on the positional relationship between the vehicle C1 and the peripheral objects detected from the vehicle information (for example, the captured image) of the vehicle C1 which is the own vehicle. The image 82 for generation prepared in advance is read, and an image (image for generation) corresponding to the vehicle C1 and surrounding objects is acquired. For example, the generation unit 73 selects a vehicle image according to the vehicle type, size, etc. of the vehicle C1 included in the captured image and the peripheral vehicle C2 which is a peripheral object from among the plurality of types of vehicle images included in the generation image 82. Select and get. Further, when the captured image includes the state of the traffic light E1 which is a peripheral object and the stop lines F1 and F2, the generation unit 73 corresponds to the state of the traffic light E1 and the stop lines F1 and F2 from the generation image 82. Select and acquire an image (image for generation).

Here, when generating the bird's-eye view image A, the generation unit 73 obtains information that cannot be obtained from the vehicle information of the vehicle C1, in other words, information that is insufficient for generating the bird's-eye view image A, vehicle information of the peripheral vehicle C2, and the like. It may be obtained by using the vehicle information of the vehicle.

Specifically, for example, as shown in FIG. 8, information on the state (signal color, etc.) of the traffic light E2 in the oncoming lane of the vehicle C1 cannot be obtained from the captured image included in the vehicle information of the vehicle C1. Therefore, the generation unit 73 detects the state of the traffic light E2 based on, for example, the image captured by the peripheral vehicle included in the vehicle information of the peripheral vehicle C2. Then, the generation unit 73 may select and acquire an image (generation image) corresponding to the detected state of the traffic light E2 from the generation image 82.

Further, when the vehicle information of the peripheral vehicle C2 is used as described above, in the present embodiment, the vehicle information of the vehicle C1 and the vehicle information of the peripheral vehicle C2 are synchronized. For example, the generation unit 73 extracts the peripheral vehicle image captured by the peripheral vehicle C2 at the same date and time as the image captured by the vehicle C1 used to generate the bird's-eye view image A, and uses the extracted peripheral vehicle image as the extracted peripheral vehicle image. Based on this, the state of the traffic light E2 is detected. As described above, in the present embodiment, by using the date and time information, the vehicle information of the vehicle C1 and the vehicle information of the peripheral vehicle C2 can be synchronized, and therefore an accurate bird's-eye view image A can be generated. Becomes possible.

In the above, the generation unit 73 detects the state of the traffic light E2 using the vehicle information of the peripheral vehicle C2, but is not limited to this, and for example, the generation unit 73 uses the vehicle information of the surveillance camera 100 to detect the state of the traffic light E2. The condition may be detected.

Further, for the entire image of the intersection D including, for example, a road, a sidewalk, a pedestrian crossing, etc., the generation unit 73 selects a road from the generation images 82 prepared in advance based on the positional relationship between the vehicle C1 and the surrounding objects. , An image corresponding to a sidewalk or a pedestrian crossing (generation image) may be selected and acquired, or an image corresponding to a road or the like (generation image) from a map information server based on the position information included in the captured image. ) May be obtained.

Then, the generation unit 73 generates a bird's-eye view image A using the various acquired images for generation. For example, the generation unit 73 indicates the vehicle C1 and the peripheral vehicle C2 at the corresponding positions in the entire image of the intersection D including the road, the sidewalk, the pedestrian crossing, etc., based on the positional relationship between the vehicle C1 and the peripheral object. An image, an image showing the states of the traffic lights E1 and E2, an image showing the stop lines F1 and F2, and the like are fitted and combined to generate a bird's-eye view image A as shown in FIG. Therefore, the bird's-eye view image A does not include the captured image of the vehicle C1 or the peripheral vehicle captured image itself of the peripheral vehicle C2.

That is, the generation unit 73 according to the present embodiment does not use the captured image or the peripheral vehicle captured image itself, but is prepared in advance and shows the vehicle C1 and the peripheral vehicle C2 when viewed from a predetermined virtual viewpoint. An animation image obtained by synthesizing an image showing the state of E2, an image showing the stop lines F1 and F2, and the like at corresponding positions of the entire image of the intersection D is generated as a bird's-eye view image A.

In the above, the bird's-eye view image A does not include the captured image or the peripheral vehicle captured image itself, but is not limited to this. That is, for example, the generation unit 73 may generate a bird's-eye view image A by using an image (here, a road image) included in the captured image or the like for a flat peripheral object such as a road.

As described above, the generation unit 73 has, as images of the peripheral objects prepared in advance, an image of the peripheral vehicle C2 existing around the vehicle C1 and an image of the stop lines F1 and F2 of the road on which the vehicle C1 and the peripheral vehicle C2 travel. , And generate a bird's-eye view image A including an image of the state of the traffic lights E1 and E2 of the road.

The bird's-eye view image A generated as described above is output to the display unit 52 by the output unit 74. As a result, the user can grasp the positional relationship between the vehicle C1 and the peripheral objects (here, peripheral vehicles C2, traffic lights E1, E2, stop lines F1, F2, etc.) by checking the bird's-eye view image A, and thus the vehicle. The situation around C1 can be easily grasped.

The above-mentioned bird's-eye view image A includes, but is not limited to, an image of the peripheral vehicle C2, an image of the stop lines F1 and F2, and an image of the state of the traffic lights E1 and E2. .. For example, the bird's-eye view image A may include at least one of an image of the peripheral vehicle C2, an image of the stop lines F1 and F2, and an image of the states of the traffic lights E1 and E2.

Further, the generation unit 73 generates a bird's-eye view image A at a predetermined timing according to the road condition of the road on which the vehicle C1 and the peripheral vehicle C2 travel. For example, the generation unit 73 sets the timing when the road condition changes as a predetermined timing, and generates the bird's-eye view image A at the timing when the road condition changes.

Specifically, the generation unit 73 generates a bird's-eye view image A at the timing when the road condition regarding the stop of the vehicle C1 and the peripheral vehicle C2 and the road condition regarding the start of the vehicle C1 and the peripheral vehicle C2 change. Specifically, the generation unit 73 generates a bird's-eye view image A at the timing when the road conditions such as the timing when the state of the traffic light E1 and the traffic light E2 changes to a red signal, the timing when the signal changes to a green signal, and the timing when the road condition changes to a yellow signal. To do. The predetermined timing described above is specified by the user, but is not limited to this.

An example of FIG. 8 is a bird's-eye view image A at a predetermined timing in which the timing at which the state of the traffic light E1 changes to a red signal is designated by the user as a predetermined timing, as in FIG. Further, in the example of the bird's-eye view image A shown in FIG. 8, it is also shown that the state of the right turn signal of the traffic light E2 is a green light. The information on the state (signal color, etc.) of the traffic light E2 is information obtained based on the peripheral vehicle image captured in the vehicle information of the peripheral vehicle C2 as described above.

As a result, the user confirms the generated bird's-eye view image A so that, for example, the vehicle C1 enters the intersection D at the timing when the state of the traffic light E1 changes to a red light, and the peripheral vehicle C2 turns right. It is possible to easily grasp the situation around the vehicle C1 such as the state of the traffic light E2 at the time of the event being a green light, and thus it is also possible to identify that the cause of the occurrence of a predetermined event is on the vehicle C1 side. Become.

In the above, the timing at which the road condition changes, such as the state of the traffic light E1 changing to a red light, is set as a predetermined timing, but the timing is not limited to this. That is, for example, the predetermined timing includes the timing when the vehicle C1 approaches the stop line F1, the timing when the vehicle C1 crosses the stop line F1, the timing when the peripheral vehicle C2 approaches the stop line F2, the timing when the vehicle C2 crosses the stop line F2, and the like. Other timings may be used.

Further, the generation unit 73 may access the storage unit 80, read out the vehicle speed information stored in association with the captured image, and generate the bird's-eye view image A including the vehicle speed information. For example, the generation unit 73 may display the vehicle speed of the vehicle C1 in the display column Y1 and the vehicle speed of the peripheral vehicle C2 in the display column Y2 in the bird's-eye view image A. In the example of the bird's-eye view image A shown in FIG. 8, it is shown that the vehicle speed of the vehicle C1 is "70 km / h" and the vehicle speed of the peripheral vehicle C2 is "30 km / h".

As a result, the user can grasp the vehicle speeds of, for example, the vehicle C1 and the peripheral vehicle C2 by checking the generated bird's-eye view image A. Further, for example, when the speed limit of the intersection D shown in FIG. 8 is "50 km / h", the user can easily grasp that the vehicle C1 has entered the intersection D with the speed limit exceeded. Can be done.

In the above description, the generation unit 73 generates the bird's-eye view image A including the information indicating the vehicle speeds of the vehicle C1 and the peripheral vehicle C2, but the present invention is not limited to this. That is, the generation unit 73 may generate a bird's-eye view image A including information indicating the acceleration of the vehicle C1 and the peripheral vehicle C2 instead of or in addition to the vehicle speed.

<5. Control processing of the information processing device according to the embodiment>
Next, a specific processing procedure in the information processing apparatus 50 will be described with reference to FIG. FIG. 9 is a flowchart showing a processing procedure executed by the information processing apparatus 50.

As shown in FIG. 9, the control unit 70 of the information processing device 50 acquires vehicle information including an captured image captured by the vehicle C1 (step S10). Next, the control unit 70 detects the positional relationship between the vehicle C1 and peripheral objects (for example, peripheral vehicles C2, traffic lights E1, E2, stop lines F1, F2, etc.) based on the acquired vehicle information (step S11). ).

Next, the control unit 70 is a bird's-eye view image A including an image (generation image) of the vehicle C1 having a detected positional relationship and a peripheral object based on the vehicle information, and the vehicle C1 designated by the user is A bird's-eye view image A is generated at a predetermined timing according to the road condition of the traveling road (step S12). Subsequently, the control unit 70 outputs the generated bird's-eye view image to the display unit 52 or the like (step S13).

As described above, the information processing apparatus 50 according to the embodiment includes an acquisition unit 71, a detection unit 72, and a generation unit 73. The acquisition unit 71 acquires vehicle information including the captured image captured by the vehicle C1. The detection unit 72 detects the positional relationship between the vehicle C1 and peripheral objects existing around the vehicle based on the vehicle information acquired by the acquisition unit 71. The generation unit 73 is a bird's-eye view image including images of the vehicle C1 and surrounding objects in a positional relationship detected by the detection unit 72 based on the vehicle information, and is a predetermined image according to the road condition of the road on which the vehicle C1 travels. Generate a bird's-eye view image at the timing of. As a result, the user can easily grasp the situation around the vehicle C1.

The information processing device 50 according to the above-described embodiment is designed to generate a bird's-eye view image A at a predetermined timing, but the present invention is not limited to this, and for example, a bird's-eye view image A at an arbitrary timing is generated. May be good.

Further, in the example shown in FIG. 8 and the like, the bird's-eye view image A is an image when the intersection D is viewed from above, but is not limited to this, and is not limited to this, and is when viewed from another viewpoint such as from diagonally above. It may be an image.

Further, in FIG. 8 and the like, a traffic light image (generated image) in which the signal lights of the traffic lights E1 and E2 face upward is used so that the state of the traffic lights E1 and E2 can be easily recognized by the user. It is an example and is not limited. That is, for example, an image of the traffic lights E1 and E2 in which the signal light faces sideways, in other words, an image of the traffic lights E1 and E2 actually viewed from above, and the state of the traffic lights E1 and E2 (the color of the signal). ) May be used, and other types of traffic light images may be used.

Further effects and variations can be easily derived by those skilled in the art. For this reason, the broader aspects of the invention are not limited to the particular details and representative embodiments expressed and described as described above. Therefore, various modifications can be made without departing from the spirit or scope of the general concept of the invention as defined by the appended claims and their equivalents.

1 Information processing system 10 In-vehicle device 50 Information processing device 71 Acquisition unit 72 Detection unit 73 Generation unit C1 Vehicle C2 Peripheral vehicle

Claims (7)

An acquisition unit that acquires vehicle information including captured images captured in the vehicle,
Based on the vehicle information acquired by the acquisition unit, a detection unit that detects the positional relationship between the vehicle and peripheral objects existing around the vehicle, and a detection unit.
A bird's-eye view image including images of the vehicle and its peripheral objects in the positional relationship detected by the detection unit based on the vehicle information, and at a predetermined timing according to the road condition of the road on which the vehicle travels. An information processing apparatus including a generation unit for generating the bird's-eye view image in the above. The generator
Based on the positional relationship detected by the detection unit, an animation image obtained by synthesizing images of the vehicle and the peripheral objects prepared in advance and viewed from a predetermined virtual viewpoint is generated as the bird's-eye view image. The information processing apparatus according to claim 1. The acquisition unit
As the vehicle information, vehicle speed information indicating the speed of the vehicle at the predetermined timing is acquired, and the vehicle speed information is acquired.
The generator
The information processing device according to claim 1 or 2, wherein the bird's-eye view image including the vehicle speed information acquired by the acquisition unit is generated. The generator
The information processing apparatus according to any one of claims 1 to 3, wherein the timing at which the road condition changes is set as the predetermined timing, and the bird's-eye view image at the timing at which the road condition changes is generated. The peripheral object includes peripheral vehicles existing around the vehicle, and includes peripheral vehicles.
The acquisition unit
The information processing device according to any one of claims 1 to 4, wherein as the vehicle information, a peripheral vehicle image captured by the peripheral vehicle is acquired. The generator
As an image of the peripheral object prepared in advance, at least one of an image of a peripheral vehicle existing around the vehicle, an image of a stop line of the road on which the vehicle travels, and an image of the state of a traffic light on the road are used. The information processing apparatus according to any one of claims 1 to 5, wherein the bird's-eye view image including the above is generated. The acquisition process for acquiring vehicle information including captured images captured in the vehicle, and
Based on the vehicle information acquired by the acquisition step, a detection step of detecting the positional relationship between the vehicle and peripheral objects existing around the vehicle, and a detection step.
It is a bird's-eye view image including images of the vehicle and the peripheral objects in the positional relationship detected by the detection step based on the vehicle information, and at a predetermined timing according to the road condition of the road on which the vehicle travels. An information processing method including a generation step of generating a bird's-eye view image.

Images