JP2020021400A - Information processing device - Google Patents

Abstract

To provide an information processing device capable of obtaining more traffic signal information with less information.SOLUTION: In an information processing device 1, a control unit 11 acquires an image captured by a camera 3 mounted on a vehicle, and estimates a lighting state of a traffic light on a road intersecting with another road on which the vehicle is traveling, on the basis of a state of the other vehicle included in the acquired image.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an information processing device for estimating a lighting state of a traffic light.

  For example, it is not possible to generate information such as a lighting color of a traffic light corresponding to a traveling direction of a vehicle based on an image captured by a camera mounted on the vehicle, as in a signal information generating method described in Patent Document 1. Already known.

JP 2006-48624 A

  The signal information creating method described in Patent Document 1 can only create information on a traffic signal corresponding to the traveling direction of a vehicle. Therefore, for example, at an intersection, when trying to obtain not only the information of the traffic signal in the traveling direction but also the information of another traffic signal, the measurement vehicle or the like enters the intersection from another direction at the same timing, or the signal is visually observed from near the intersection. It is necessary to perform time-consuming operations such as observing the movement of the robot. In other words, the conventional method has a problem that much trouble and information are required to obtain information of another traffic signal at an intersection.

  An example of the problem to be solved by the present invention is to obtain more traffic signal information with less information.

  In order to solve the above problem, the invention according to claim 1 is included in a first acquisition unit that acquires an image captured by an imaging unit mounted on a vehicle, and the image acquired by the first acquisition unit. An estimating unit for estimating a lighting state of a traffic signal on a road intersecting with a road on which the vehicle travels based on a state of another vehicle.

  The invention according to claim 6 is an information processing method executed by an information processing device for estimating a lighting state of a traffic light, wherein a first acquisition step of acquiring an image captured by an imaging unit mounted on a vehicle; An estimating step of estimating a lighting state of a traffic light on a road intersecting with a road on which the vehicle travels based on a state of another vehicle included in the image acquired in the first acquiring step. And

  According to a seventh aspect of the present invention, a computer executes the information processing method according to the sixth aspect.

  An eighth aspect of the present invention is characterized by storing the information processing program according to the seventh aspect.

FIG. 1 is a schematic configuration diagram illustrating an information processing device and a server device according to an embodiment of the present invention. FIG. 2 is a schematic configuration diagram of the information processing apparatus shown in FIG. 1. FIG. 2 is a schematic configuration diagram of a server device shown in FIG. 1. 3 is a flowchart of an operation in the information processing apparatus shown in FIG. It is explanatory drawing which showed the range of the intersection. It is explanatory drawing of the method of estimating the lighting state of the traffic light installed in the intersection. It is explanatory drawing of the method of estimating the lighting state of the traffic light installed in the intersection. It is an example of traffic light lighting state estimation information. 4 is a flowchart of an operation in the information processing apparatus shown in FIG. It is an example of the cycle information of a traffic light.

  Hereinafter, an information processing apparatus according to an embodiment of the present invention will be described. An information processing apparatus according to an embodiment of the present invention acquires an image captured by an imaging unit mounted on a vehicle in a first acquisition unit, and obtains an image of another vehicle included in the image acquired by the first acquisition unit in the estimation unit. Based on the state, the lighting state of the traffic light on the road intersecting with the road on which the vehicle travels is estimated. In this way, it is possible to estimate the lighting state of the traffic light on the road intersecting the road on which the vehicle travels from the image captured by the specific vehicle. Therefore, it is possible to obtain information on traffic signals other than the traveling direction with less information.

  Further, the traffic light for which the estimating unit estimates the state does not have to include the lighting state in the image acquired by the first acquiring unit. In this way, for a traffic light whose lighting state is unknown from the image, the lighting state can be estimated based on the state of another vehicle.

  The image processing apparatus may further include an output unit that outputs a time at which the image is captured and a lighting state of the traffic signal estimated by the estimation unit from the image. By doing so, for example, the lighting state of the traffic signal estimated by the estimating unit can be uploaded to an external server or the like, and the cycle information of the traffic signal can be generated based on the estimation results from various vehicles. .

  The image processing apparatus further includes a second acquisition unit that acquires information about an imaging position of the image and information about a direction in which the imaging unit has taken an image, wherein the estimation unit includes an image acquired by the first acquisition unit and an image acquired by the second acquisition unit. The lighting state of the traffic light may be estimated based on the information on the imaging position and the information on the orientation of the image. By doing so, it is possible to know where and in what direction the image captured by the image capturing unit was captured, and to more accurately estimate the lighting state of the traffic light on the road that intersects the road on which the vehicle travels. it can.

  Further, the first acquisition unit may acquire a plurality of images, and the estimation unit may estimate a lighting state of the traffic light based on the plurality of images acquired by the first acquisition unit. In this manner, for example, the lighting state of the traffic light can be estimated based on a plurality of frames of images. Further, the lighting state of the traffic light can be estimated with reference to the image at the same time on another day. Alternatively, the lighting state of the traffic light can be estimated with reference to an image of another vehicle captured at the same time.

  Further, in the information processing method according to an embodiment of the present invention, the first obtaining step obtains an image captured by the imaging unit mounted on the vehicle, and the estimating step is included in the image obtained in the first obtaining step. A lighting state of a traffic light on a road intersecting with a road on which the vehicle travels is estimated based on the state of the vehicle. In this way, it is possible to estimate the lighting state of the traffic light on the road intersecting the road on which the vehicle travels from the image captured by the specific vehicle. Therefore, it is possible to obtain information on traffic signals other than the traveling direction with less information.

  Further, the information processing method described above may be an information processing program that is executed by a computer. In this manner, the lighting state of the traffic light on the road intersecting with the road on which the vehicle travels can be estimated from the image captured by the specific vehicle using the computer. Therefore, it is possible to obtain information on traffic signals other than the traveling direction with less information.

  Further, the above-described information processing program may be stored in a computer-readable recording medium. By doing so, the program can be distributed as a single unit in addition to being incorporated into the device, and the version can be easily upgraded.

  An information processing apparatus 1 according to an embodiment of the present invention will be described with reference to FIGS. The information processing apparatus 1 according to the present embodiment is mounted on a vehicle C as shown in FIG. The information processing device 1 is capable of communicating with the server device 2 via a network N such as the Internet.

  FIG. 2 shows a functional configuration of the information processing apparatus 1. The information processing device 1 includes a control unit 11, a communication unit 12, and a storage unit 13. Further, as shown in FIG. 2, the information processing apparatus 1 is connected to a camera 3 mounted on a vehicle C and a sensor 4 including a GPS receiver, a speed sensor, and the like.

  The control unit 11 is configured by, for example, a microcomputer having a CPU (Central Processing Unit), a memory, and the like, and controls the entire information processing apparatus 1. The control unit 11 estimates the lighting state of the traffic light (red, yellow, and blue lighting colors, the display direction of the arrow signal, and the like) based on the image captured by the camera 3, the detection result of the sensor 4, and the like.

  The communication unit 12 transmits the lighting state of the traffic light and the like estimated by the control unit 11 to the server device 2.

  The storage unit 13 stores programs operated by the control unit 11, various data, and the like. Further, the storage unit 13 stores map data for determining approach to an intersection described later.

  The camera 3 as an imaging unit is installed in the vehicle C as described above. Further, the camera 3 captures an image of, for example, the front (the traveling direction) of the vehicle C. The camera 3 outputs the captured image to the information processing device 1. The camera 3 may also be used for other purposes such as a drive recorder or an ADAS (Advanced Driver Assistance System). The image captured by the camera 3 may be a still image or a moving image.

  The sensor 4 is a sensor provided in the vehicle, such as a GPS receiver and a speed sensor, as described above. Note that a sensor included in the sensor 4, for example, a GPS receiver or the like may be shared with another vehicle-mounted device (such as a navigation device).

  The server device 2 includes a control unit 21, a communication unit 22, and a storage unit 23, as shown in FIG. The control unit 21 functions as a CPU of the server device 2 and controls the entire control of the server device 2. Further, the control unit 21 generates cycle information of the traffic light based on the lighting state of the traffic light transmitted from the information processing device 1.

  The communication unit 22 functions as a network interface of the server device 2 and receives the lighting state of the traffic signal transmitted by the information processing device 1.

  The storage unit 23 functions as a storage device such as a hard disk of the server device 2 and stores the cycle information and the like generated by the control unit 21. Further, the server device 2 distributes the cycle information stored in the storage unit 23 to the vehicle as needed.

  Next, an operation (information processing method) of the information processing apparatus 1 according to the above configuration will be described with reference to a flowchart of FIG. The flowchart illustrated in FIG. 4 is executed by the control unit 11. The flowchart shown in FIG. 4 is configured as a computer program (information processing program) operated by the CPU of the control unit 11. Further, the program is not limited to being stored in the memory or the like of the control unit 11, but may be stored in a storage medium such as a memory card or an optical disk.

  First, in step S11, the control unit 11 determines whether the vehicle C equipped with the information processing device 1 has approached an intersection. If it is determined that the vehicle C has approached an intersection, the process proceeds to step S12. Repeat this step.

  Whether the vehicle has approached the intersection may be determined based on, for example, the current position acquired from the GPS receiver included in the sensor 4 and the map data stored in the storage unit 13. In addition, the approach means, for example, approaching a predetermined distance such as a distance enough for the camera 3 to cover the range of the intersection. Here, the intersection is a portion where two or more roads intersect when two or more roads intersect in this embodiment. For example, in the case of a crossroad where the road R1 and the road R2 intersect as shown in FIG. 5, a range surrounded by a broken line B is an intersection.

  Next, in step S12, the control unit 11 acquires an image captured by the camera 3. That is, the control unit 11 functions as a first acquisition unit that acquires an image captured by the imaging unit mounted on the vehicle.

  Next, in step S13, the control unit 11 acquires the current position information from the GPS receiver included in the sensor 4. The current position indicated by the current position information is the imaging position. Further, the control unit 11 acquires from the camera 3 information on the direction in which the camera 3 has taken an image (the angle may be from a specific direction, such as forward or backward). The information of this direction may be obtained together with the image in step S12. When the imaging direction of the camera 3 is fixed, the direction may be set in the control unit 11 in advance. That is, the control unit 11 functions as a second acquisition unit that acquires information on the image capturing position of the image and information on the direction in which the camera 3 (image capturing unit) has captured the image.

  Next, in step S14, the control unit 11 estimates the lighting state of the traffic light at the intersection based on the image acquired in step S12, the imaging position acquired in step S13, and the imaging direction. An estimation method will be described with reference to FIG.

  FIG. 6 is a diagram when the vehicle C travels in the direction of the arrow and passes through an intersection. In FIG. 6, a traffic light a is a traffic light corresponding to the lane in the traveling direction of the vehicle C on the road R3, a traffic light b is a traffic light corresponding to the opposite lane of the lane on which the vehicle C is traveling on the road R3, and a traffic light c is the traveling direction of the vehicle C. A traffic light corresponding to a lane heading in one direction of the road R4 crossing the road R4, and a traffic light d corresponding to a lane heading in the other direction of the road R4.

  At the intersection shown in FIG. 6, the lighting state of the traffic light is estimated based on an image captured from the vehicle C and the like. The traffic light a may be estimated by a well-known method of recognizing the traffic light a from an image and estimating a lighting color or the like. According to the present embodiment, the lighting state can be further estimated for the traffic lights b, c, and d. For example, for the traffic lights c and d, it can be estimated whether or not there is a vehicle traveling on the road R4 and passing through the intersection to determine whether or not the traffic light is a red light. In the case of a moving image, the traffic lights c and d can be estimated to be blue if a vehicle passing through an intersection with images of a plurality of frames can be detected by a known image recognition technique.

  Further, even in the case of a still image, the traffic lights c and d can be estimated to be blue if a vehicle (symbol D in FIG. 7) that passes through the intersection by traveling straight on the road R4 on the image can be detected. That is, the control unit 11 functions as an estimation unit that estimates the lighting state of the traffic light on the road R4 intersecting with the road R3 on which the vehicle C travels based on the state of the vehicle D (other vehicle) included in the acquired image. I do.

  The lighting state of the traffic light b can be estimated based on whether the vehicle is stopped in the oncoming lane. For example, if a vehicle that travels straight through an intersection is detected from the image, the traffic light b can be estimated to be blue. Note that it is not necessary to estimate all of the traffic signals b, c, and d. Only the traffic light that can be estimated from the state of the other vehicle needs to be estimated.

  Further, not only images but also map data stored in the storage unit 13 may be obtained and used for estimation. For example, a process of acquiring map data to which information on the installation position and orientation of a traffic light is added, and identifying a corresponding traffic light by comparing which traffic light in the captured image corresponds to which traffic light in the map data. The accuracy of is improved. Further, by acquiring map data including the position of the lane, and collating the position of the lane included in the map data with the position of the vehicle in the image, the position of the vehicle can be estimated more accurately. Therefore, it is possible to improve the estimation accuracy of the lighting state of the traffic light. The map data may be stored in the storage unit 23 and downloaded to the information processing device 1 as needed.

  In this way, by estimating the lighting state of the traffic light based on the state of the other vehicle, it is possible to estimate the lighting state even for a traffic light whose lighting state is not reflected in the image.

  Returning to the description of FIG. 4, in step S15, the communication unit 12 associates the result estimated in step S14 with the image capturing time, image capturing position, image capturing direction, and the like of the image used for the estimation, and sets the communication unit 12 as traffic light lighting state estimation information. The data is transmitted (output) to the server device 2 via the server. Here, an example of the traffic light lighting state estimation information is shown in FIG. That is, the communication unit 12 functions as an output unit that outputs the time at which the image was captured and the lighting state of the traffic signal estimated by the control unit 11 (estimating unit) from the image.

  FIG. 8 shows the imaging time and the estimation result of the lighting state of each traffic light. In the estimation result of each traffic light, "front" means, for example, the lighting state of the traffic light a corresponding to the lane in the traveling direction of the vehicle C in FIG. 6 and the like, and "opposite" means that the vehicle C on the road R3 travels, for example. Traffic light b corresponding to the opposite lane of the traffic lane. Also, "intersection 1" corresponds to, for example, a traffic signal c corresponding to a lane in one direction of a road R4 intersecting with the traveling direction of the vehicle C, and "intersection 2" corresponds to, for example, a lane in the other direction of the road R4. Traffic light d. “Intersection 1” and “intersection 2” may have the traffic signals indicated by the respective intersections reversed.

  Then, by adding information of the image capturing position and the image capturing direction to the information of FIG. 8, the server device 2 recognizes information at which intersection, which traffic light indicates "front" or "opposite", and the like. be able to.

  When the position information of the traffic signal is included in the map data, the position information of each traffic signal may be used instead of the above “front”.

  As is clear from the above description, step S12 functions as a first acquisition step, and step S14 functions as an estimation step.

  Next, the operation of the server device 2 will be described with reference to the flowchart of FIG. The flowchart shown in FIG.

  First, in step S <b> 21, the control unit 21 acquires the traffic light lighting state estimation information transmitted from the information processing device 1 via the communication unit 22.

  Next, in step S22, the control unit 21 generates cycle information of the traffic light. FIG. 10 shows an example of the cycle information of the traffic light. FIG. 10 shows cycle information of the traffic light at the intersection shown in FIG. 6 and the like. The items shown in FIG. 10 are basically the same as those shown in FIG. 8, but the traffic signals are identified based on, for example, the installation positions. In addition, the lighting state is not only the lighting state at a specific time, but the lighting states at predetermined time intervals are accumulated, for example, in one day or several hours. In FIG. 10, the interval is 10 seconds, but the interval may not be the same, and the traffic light lighting state estimation information may be accumulated at the time when it was obtained. If the cycle information has already been generated, the cycle information may be updated based on the traffic light lighting state estimation information newly acquired from the information processing device 1.

  Next, in step S23, the control unit 21 stores the cycle information of the traffic signal generated in step S22 in the storage unit 23.

  According to the present embodiment, the information processing apparatus 1 causes the control unit 11 to acquire an image captured by the camera 3 mounted on the vehicle, and to control the vehicle based on the state of the other vehicle included in the acquired image. The lighting state of the traffic light on the road intersecting with the traveling road is estimated. In this way, it is possible to estimate the lighting state of the traffic light on the road intersecting the road on which the vehicle travels from the image captured by the specific vehicle. Therefore, it is possible to obtain information on traffic signals other than the traveling direction with less information.

  Further, the traffic light for which the control unit 11 estimates the state does not have to include the lighting state in the acquired image. In this way, for a traffic light whose lighting state is unknown from the image, the lighting state can be estimated based on the state of another vehicle.

  In addition, the information processing apparatus 1 includes a communication unit 12 that outputs a time at which an image is captured and a lighting state of a traffic signal estimated by the control unit 11 from the image. By doing so, for example, the lighting state of the traffic signal estimated by the control unit 11 can be uploaded to the server device 2, and cycle information including the lighting state of the traffic signal can be generated based on estimation results from various vehicles. It becomes possible.

  In addition, the control unit 11 acquires information on the image capturing position of the image and information on the direction in which the camera 3 captures the image, and based on the image, the information on the image capturing position of the image, and the information on the image direction, turns on the traffic light. Is estimated. By doing so, it is possible to know where and in what direction the image captured by the camera 3 was captured, and to more accurately estimate the lighting state of the traffic light on the road intersecting the road on which the vehicle travels. it can.

  In the above-described embodiment, the lighting state of the traffic light is estimated from an image captured at a specific date and time. However, the lighting of the traffic light is determined from a plurality of images of the image and an image at the same time on another day. The state may be estimated. By doing so, even when estimation is difficult at a specific date and time, estimation can be performed using an image captured on another day.

  In the above-described embodiment, the information processing apparatus 1 performs the estimation of the lighting state of the traffic light. By transmitting the signal, the server device 2 may estimate the lighting state of the traffic light. By doing so, the result of estimating the lighting state can be directly reflected as cycle information. Further, the reliability can be improved by reducing the processing load of the information processing device 1 or by combining the information with information from another vehicle or the like.

  Further, by estimating the lighting state of the traffic light by the server device 2, for example, not only an image captured by the vehicle C but also an image of a vehicle traveling ahead or behind the vehicle C at the same time, an oncoming vehicle, or the like. It is possible to statistically estimate using the obtained image. Therefore, it is possible to further improve the estimation accuracy of the lighting state of the traffic light.

  Note that the camera 3 does not need to constantly capture an image. For example, photographing may be started when approaching a traffic light such as an intersection based on map data or the like. In addition, when the color of the traffic light changes, when it is detected that the stopped vehicle has started to move, when the vehicle that was traveling has stopped, when the turning on or off of the brake lamp of the vehicle has been detected, etc. May be used for shooting.

  Further, the present invention is not limited to the above embodiment. That is, those skilled in the art can make various modifications in accordance with conventionally known knowledge without departing from the gist of the present invention. As long as the information processing apparatus of the present invention is still provided by such a modification, the modification is, of course, included in the scope of the present invention.

Reference Signs List 1 information processing device 2 server device 3 camera (imaging unit)
4 sensor 11 control unit (first acquisition unit, estimation unit, second acquisition unit)
12 Communication unit (output unit)
13 Storage unit

Claims (8)

A first acquisition unit that acquires an image captured by an imaging unit mounted on a vehicle,
An estimating unit that estimates a lighting state of a traffic light on a road intersecting with a road on which the vehicle travels, based on a state of another vehicle included in the image acquired by the first acquisition unit;
An information processing apparatus comprising:   2. The information processing apparatus according to claim 1, wherein the traffic light whose state is estimated by the estimating unit does not include a lighting state in an image acquired by the first acquiring unit. 3.   The information processing apparatus according to claim 1, further comprising an output unit configured to output a time at which the image is captured and a lighting state of the traffic signal estimated by the estimation unit from the image. A second acquisition unit configured to acquire information on an imaging position of the image and information on a direction in which the imaging unit captures the image,
The estimating unit sets a lighting state of the traffic light based on the image acquired by the first acquiring unit, information on an imaging position of the image acquired by the second acquiring unit, and information on an orientation of the image. presume,
The information processing apparatus according to claim 1, wherein: The first acquisition unit acquires a plurality of the images,
The information processing apparatus according to claim 1, wherein the estimating unit estimates a lighting state of the traffic light based on the plurality of images acquired by the first acquiring unit. . An information processing method executed by an information processing device that estimates a lighting state of a traffic light,
A first acquisition step of acquiring an image captured by an imaging unit mounted on the vehicle;
An estimation step of estimating a lighting state of a traffic light on a road intersecting with a road on which the vehicle travels, based on a state of another vehicle included in the image acquired in the first acquisition step;
An information processing method comprising:   An information processing program for causing a computer to execute the information processing method according to claim 6.   A computer-readable recording medium storing the information processing program according to claim 7.

Images