JP2008139320A - Notification apparatus and notification program of road environmental information - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a notification apparatus of road environmental information which can be built at a low price and can provide drivers on real time useful road environment information earlier than the drivers perceive by visual observations, and can improve the reliability of the result information of image recognition. <P>SOLUTION: An image recognition means 1102 extracts an object to be recognized from the image information captured by an imaging means 1101. A generation means of image recognition result information 1104 generates the image recognition result information based on the information of the result of image recognition and the absolute position of a self-vehicle from an vehicle position locating means 1103. A generation means of road environmental reproduction data integration 1108 generates the newest road environmental reproduction data integrating the history information of the image recognition result information memorized by a storage means of the image recognition result information 1107 with the road environmental reproduction data generated a short time before. An information notification means 1113 notifies the drivers of the information included in the road environmental reproduction data generated by the generation means of road environmental reproduction data integration 1108. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention provides information such as road signs, road signs, traffic lights, pedestrians, and surrounding vehicles that are not provided by the car navigation system or VICS information from images captured by the in-vehicle camera of the in-vehicle notification device. Recognize every moment by image recognition, and aggregate, integrate, or distribute information recognized by each vehicle via a network to provide real-time information that is useful and reliable for driver behavior determination The present invention relates to a road environment information notification device and a road environment information notification program.

  A car navigation system is widely used as a system that supports the behavior determination of a driver of a vehicle (automobile or the like). The car navigation system uses map information and VICS (Vehicle Information and Communication System) information to provide information on road location, road congestion, current location of the vehicle, route to the destination, etc. This is a system that supports the driver's behavior by notifying the driver by graphic display on the monitor or by voice.

  However, since the car navigation system cannot provide information on an object such as a road sign such as a stop sign, a signal state, or a vehicle or a pedestrian on the road (hereinafter referred to as road environment information), You must drive with care so as not to miss road signs and pedestrians. Therefore, as a system that assists the driver in acquiring road environment information, the road environment information such as road signs is recognized by a camera mounted on the driver's own vehicle, and the driver is notified by text, graphics, or voice. A real-time information providing system for notifying road environment information is disclosed in Patent Literature 1, Patent Literature 2, Patent Literature 3, and Non-Patent Literature 1.

  By these real-time information providing systems, it is possible to reduce the burden on the driver who has been focused on finding object information such as road signs and pedestrians. In addition, in order to realize these real-time information providing systems, it is only necessary to install a car navigation system with a camera in the vehicle. Therefore, it is easy to install necessary equipment and maintain the system, and the system can be constructed at low cost. .

  In addition, the Ministry of Land, Infrastructure, Transport and Tourism in Japan has proposed Intelligent Transport Systems (ITS) that aims to eliminate traffic accidents by connecting people, roads, and vehicles through a network. Advanced cruise-assist highway systems (AHS), one of the pillars of the concept of advanced traffic systems, are (1) pedestrians, vehicle location information, The vehicle surface information, (2) signal lighting information transmitted by the traffic light, and (3) other vehicle position information transmitted by the other vehicle are received by the receiving device installed in each vehicle, so that each vehicle can run fully automatically. This is a technique intended to be realized (see, for example, Non-Patent Document 2).

  Further, Non-Patent Document 3 discloses a traffic information providing system that provides a driver with an image captured by a camera installed beside a road via a network.

  Furthermore, a system that transmits and receives vehicle position information by GPS (Global Positioning System) between vehicles via a network and prevents a collision between vehicles and a vehicle from colliding with a road shoulder is disclosed in Non-Patent Document 4, Non-Patent Document. It is disclosed in Reference 5 and Non-Patent Reference 6.

JP 2000-67374 A (page 4-6, FIGS. 1 and 2) Japanese Patent Laid-Open No. 10-177699 (page 3-5, FIG. 1) JP 2001-84496 A (page 4-8, FIG. 1, FIG. 8, FIG. 10) US Pat. No. 6,311,123 Ministry of Land, Infrastructure, Transport and Tourism, “AHS”, [online], April 1, 2002, Ministry of Land, Infrastructure, Transport and Tourism, [September 9, 2002 search], Internet <URL: http: //www.nilim.go. jp / japanese / its / fields / ahs / ＞ US Pat. No. 5,396,429 US Pat. No. 6,405,132 US Pat. No. 6,370,475 US Pat. No. 6,275,773

  However, a conventional real-time information providing system that notifies road environment information by an in-vehicle camera of the own vehicle only provides the driver with information that can be perceived by the driver at almost the same time as the driver perceives. Absent. Therefore, the conventional real-time information providing system can reduce oversight of road environment information due to driver's carelessness, but obtain other useful road environment information that the driver cannot perceive himself at a specific time, It cannot be provided to the driver.

  For example, while driving on a road, a vehicle with a high vehicle height (for example, a trailer) blocks the front view of the vehicle, and the driver lights the front signal (for example, a red signal or a blue signal). For example, if you want to know as soon as possible the road conditions such as whether there is a shoulder parked vehicle on the road ahead, the conventional real-time information provision system uses There is a possibility that the driver cannot be notified of the presence / absence information.

  In addition, the performance of an object recognition technique generally called computer vision is low compared to human visual ability. Therefore, depending on the positional relationship between the camera installed in the host vehicle and the object to be recognized, the sunshine conditions, etc., the object to be recognized may not be recognized well. However, the conventional real-time information providing system uses such computer vision technology. There is no means to compensate for imperfections in object recognition.

  Further, in AHS, since infrastructure such as sensors is installed on the road without fail, it is possible to notify the driver of accurate road environment information faster than the driver visually perceives. However, the installation of infrastructure such as sensors on the road and the maintenance and management of the infrastructure require a large amount of money, which is problematic in terms of cost and feasibility. In addition, since the information communicated between the infrastructure installed on the road and the vehicle and the information communicated between the vehicles are always assumed to be correct information, inaccurate information was sent and received. In some cases, there is no mechanism to compensate for inaccuracy of information or a mechanism for integrating inaccurate information.

  Further, in a system that transmits and receives vehicle position information by GPS between vehicles via a network, a huge infrastructure investment is required to enable transmission of position information signals from guardrails, traffic lights, road signs, or the like. . In addition, the system can transmit and receive accurate position information, but does not have a mechanism for receiving and integrating image recognition result information including inaccurate elements.

  Therefore, the present invention can be constructed inexpensively without installing infrastructure on the road, and provides useful information such as road signs, signal states, and presence / absence of objects on the road earlier than the driver perceives visually. It is an object of the present invention to provide a road environment information notification device and a road environment information notification program that can be provided to a driver in real time and can improve the reliability of image recognition result information.

  A road environment information notification device according to the present invention is a road environment information notification device that is installed in a vehicle and notifies a driver of road environment information that is information related to the road environment, and outputs image information by imaging the periphery of the vehicle. Image recognition means for recognizing and extracting an image related to the road environment from the captured image information, outputting information on the extracted image recognition result, and image recognition result information including information on the result of image recognition Image recognition result information transmitting means for transmitting to other vehicles, image recognition result information receiving means for receiving image recognition result information from other vehicles, image recognition result information of the own vehicle, and other received Road environment reproduction data integration and generation means for generating road environment reproduction data that is information related to the integrated road environment by integrating vehicle image recognition result information, and information included in the road environment reproduction data Characterized in that an information notifying means for notifying the driver.

  According to such a configuration, it is possible to construct an inexpensive device without installing an infrastructure and an information center on the road, and it is faster than the driver can perceive by visual observation. Therefore, useful information such as the presence / absence of the object can be provided to the driver in real time, and the reliability of the image recognition result information can be improved.

  The road environment information notification device obtains image recognition result information based on vehicle position grasping means for grasping the absolute position of the vehicle, information on the result of image recognition output by the image recognition means, and information on the absolute position of the vehicle. Image recognition result information generating means for generating, image recognition result information storing means for storing image recognition result information of the own vehicle generated by the image recognition result information generating means, and history information of the received image recognition result information of other vehicles And a road environment reproduction data storage means for storing road environment reproduction data including the latest estimation result regarding the absolute position and attribute of the recognition target object at the current time and the overall certainty factor indicating the certainty of the estimation result. It is preferable.

  According to such a configuration, it is possible to generate image recognition result information including information on the absolute position of the recognition target object, the latest estimation result regarding the absolute position and attribute of the recognition target object at the current time, and the estimation result It is possible to generate road environment reproduction data including information on the overall certainty indicating the certainty.

  The road environment information notification device selects important information selection means from the road environment reproduction data and extracts the information judged to be important to the driver from the road environment reproduction data output by the road environment reproduction data integrated generation means. It is preferable that it is provided.

  According to such a configuration, it is possible to notify only important information to the driver among information included in the road reproduction data.

  In the road environment notification device installed in the vehicle, the image recognition means extracts the attribute of the recognition target object, the relative position with respect to the vehicle, and the recognition time by performing image recognition based on the captured image information, In addition, a recognition certainty indicating the certainty of the result obtained by image recognition is calculated, and information on the result of image recognition including the attribute, relative position, recognition time, and recognition certainty is output, and image recognition result information generation is performed. The means calculates the absolute position of the recognition target object based on the relative position of the recognition target object with respect to the vehicle included in the image recognition information and the absolute position of the vehicle acquired by the vehicle position grasping means. Information including position, attribute, recognition time, and recognition certainty is output as image recognition result information, and the road environment reproduction data integrated generation unit stores the image stored in the image recognition result information storage unit. The content of the information included in the history information of the image recognition result information and the road environment based on the value of the recognition certainty included in the history information of the recognition result information and the total certainty included in the road environment reproduction data one hour before By integrating the content of the information included in the reproduction data, calculating the absolute position and attribute estimation result of the recognition target object at the current time, and updating the content of the road environment reproduction data based on the estimation result, It is preferable that the latest road environment reproduction data is generated, and the information notification means notifies the driver of the information extracted by the important information selection means by display or sound by video.

  According to such a configuration, it is possible to generate road environment reproduction data in consideration of the recognition certainty indicating the certainty of the result of the image recognition recognized by the image recognition means, and to provide information to the driver by visual or voice. Can be notified.

  In the road environment notification device installed in the vehicle, the road environment reproduction data integration and generation unit generates an image when the image recognition result information storage unit stores the image recognition result information newly received by the image recognition result information reception unit. The total certainty factor may be calculated by calculating an average value of the certainty factors of recognition included in the history information of the recognition result information.

  According to such a configuration, the total certainty factor can be calculated in consideration of history information of past image recognition result information.

  In the road environment notification device installed in the vehicle, the road environment reproduction data integrated generation means includes the road environment reproduction data stored in the road environment reproduction data storage means and the image recognition result information stored in the image recognition result information storage means. If there is no information corresponding to the recognition target object of the image recognition result information received by the image recognition result information receiving means in any of the history information, the recognition certainty included in the received image recognition result information is recognized. The total certainty factor for the target object may be used.

  According to such a configuration, road environment reproduction data corresponding to a newly recognized recognition target object can be generated.

  When the attributes of the recognition target object change over time, the road environment reproduction data integrated generation unit has a difference between the recognition time and the current time in the history information of the image recognition result information stored in the image recognition result information storage unit. The road environment reproduction data may be generated based only on information within a predetermined time period.

  According to such a configuration, it is possible to notify the driver of highly reliable road environment information for a recognition target object whose attribute changes with time, such as a traffic light or a pedestrian.

  When the image recognition result information receiving means does not receive the image recognition result information including the information of the specific recognition target object for a predetermined time, the road environment reproduction data integration generating means Of the road environment reproduction data stored, the road environment reproduction data corresponding to the recognition target object may be deleted.

  According to such a configuration, it becomes possible to delete road environment reproduction data that has become unnecessary.

  The image recognition result information generating means generates image recognition result information including vehicle number information for specifying the vehicle, and the road environment reproduction data integrated generating means is the image recognition result information received by the image recognition result information receiving means. When the content of the vehicle environment information does not match the content of the road environment reproduction data, the vehicle number information included in the image recognition result information is stored for a predetermined time, and the image recognition result information received from the vehicle of the specific vehicle number information If the content does not match the content of the road environment reproduction data at a rate exceeding the predetermined frequency, the recognition reliability included in the image recognition information received from the vehicle is estimated to be low, and new road environment reproduction data is generated It may be.

  According to such a configuration, even when information with low reliability is included in the image recognition result information, road environment information with high reliability can be provided to the driver.

  The road environment information notification device includes an input operation unit that inputs an instruction relating to a planned travel route, and the important information selection unit may select and extract information related to a recognition target object existing on the planned travel route. Good.

  According to such a configuration, it is possible to select the road environment information related to the planned travel route instructed in advance and notify the driver.

  The input operation means may have a function of inputting designation of information to be acquired, and the important information selection means may select and extract information based on the designation input by the input operation means.

  According to such a configuration, it is possible to select road environment information related to a type of recognition target object designated in advance and notify the driver.

  The input operation means has a function of inputting designation of whether or not to transmit the image recognition result information to another vehicle, and the image recognition result information transmission means transmits the image recognition result information by an input of the input operation means. The image recognition result information may be transmitted only when the effect is designated.

  According to such a configuration, when the driver does not want other people to know the position of the host vehicle, it can be set not to transmit information from the host vehicle.

  The road environment information notification device has a problem when the content of the image recognition result information output by the image recognition result information generating means and the content of the road environment reproduction data do not coincide with each other beyond a predetermined frequency. And an operation state monitoring unit that monitors the operation state of the imaging unit or the image recognition unit, and the information notification unit provides information on the detected failure when the operation state monitoring unit detects the failure. May be notified.

  According to such a configuration, the driver can know the occurrence of a malfunction of the in-vehicle notification device mounted on the host vehicle.

  The road environment information notification device stores the video imaged by the imaging means for a predetermined time, stops recording a new video when it senses an impact on the vehicle, and retains the video just before sensing the impact. It may be provided with video storage means immediately before impact detection.

  According to such a configuration, when an accident occurs, the cause of the accident can be quickly investigated.

  The road environment information notification device includes vehicle speed measurement means for sensing the moving speed of the vehicle, and the important information selection means is for traffic environment reproduction data in which the attribute of the recognition target object is a traffic light or a stop sign. , A threshold for which the moving speed of the vehicle is predetermined with respect to the distance from the vehicle to the recognition target object calculated based on the absolute position of the recognition target object included in the road environment reproduction data and the absolute position of the vehicle at the current time The road environment reproduction data related to the recognition target object may be selected and extracted only when the number exceeds.

  According to such a configuration, the driver can decelerate the moving speed of the host vehicle with a margin before reaching the location of the red traffic light or the stop sign.

  The road environment information notifying device selects a recognition target object when the recognition target object corresponding to the received road environment reproduction data is in a position where it can be recognized from the host vehicle and the image recognition unit has not recognized the recognition target object. A non-extracted object determination means for determining, and the image recognition means extracts an unextracted object by a top-down image analysis process that performs image analysis based on information of a recognition target object determined by the unextracted object determination means. There may be.

  According to such a configuration, an image can be recognized even if the image recognition means of the host vehicle is a recognition target object that is difficult to detect.

  A road environment information notifying program according to the present invention is a road environment information notifying program installed in a vehicle for notifying a driver of road environment information that is information on the road environment, and the computer surroundings A process for recognizing and extracting an image related to the road environment from the image information of the image, outputting information of the extracted image recognition result, and a process of transmitting the image recognition result information including the information of the image recognition result to another vehicle And the processing for receiving image recognition result information from other vehicles, the image recognition result information of the own vehicle, and the received image recognition result information of other vehicles, and information relating to the integrated road environment. This is to execute processing for generating road environment reproduction data and processing for notifying the driver of information included in the road environment reproduction data.

  According to such a configuration, it is possible to construct an inexpensive device without installing an infrastructure and an information center on the road, and it is faster than the driver can perceive by visual observation. Therefore, useful information such as the presence / absence of the object can be provided to the driver in real time, and the reliability of the image recognition result information can be improved.

  According to the road environment information notification device according to the present invention, an image capturing unit that captures an image of the periphery of the vehicle and outputs image information, an image related to the road environment is recognized and extracted from the captured image information, and the extracted image recognition result Image recognition means for outputting the information, image recognition result information transmission means for sending image recognition result information including information on the result of image recognition to other vehicles, and image recognition result information from other vehicles A road environment that integrates the image recognition result information receiving means, the image recognition result information of the own vehicle, and the received image recognition result information of another vehicle, and generates road environment reproduction data that is information relating to the integrated road environment Since it is configured to include the integrated reproduction data generation means and the information notification means for notifying the driver of the information included in the road environment reproduction data, without installing infrastructure on the road It is possible to construct a road environment information notification system at a reasonable price, and provide useful information to the driver in real time, such as road signs, signal conditions, and the presence of objects on the road, faster than the driver can perceive visually. And the reliability of the image recognition result information can be improved.

  According to the road environment information notification program according to the present invention, the computer recognizes and extracts an image related to the road environment from the image information around the captured vehicle, and outputs the extracted image recognition result information; Processing for transmitting image recognition result information including information on the result of recognition to another vehicle, processing for receiving image recognition result information from another vehicle, image recognition result information of the own vehicle, and other received The image recognition result information of the vehicle is integrated, and the processing for generating road environment reproduction data that is information related to the integrated road environment and the processing for notifying the driver of information included in the road environment reproduction data are executed. It is possible to provide useful information to the driver in real time, such as road signs, signal conditions, and the presence or absence of objects on the road, faster than the driver can perceive visually. It is possible to improve the reliability of identification of the result information.

Embodiment 1 FIG.
Embodiments of a road environment information notification device according to the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of the road environment information notification device according to the first embodiment. The road environment information notification device shown in FIG. 1 is a system in which an information center collects information based on an image sent from a vehicle traveling on a road, and distributes information generated by integration based on the collected information to the vehicle. It is. The road environment information notification device includes an in-vehicle notification device 10 installed in each vehicle and an information center device 20 installed in the information center.

  As shown in FIG. 1, the in-vehicle notification device 10 includes an imaging unit 101 that captures the periphery of a vehicle, an image recognition unit 102 that recognizes an image, a vehicle position positioning unit 103 that measures the position of the host vehicle, and an image recognition result. Image recognition result information generating means 104 for generating information, image recognition result information transmitting means 105 for transmitting image recognition result information to the information center apparatus 20, and road environment reproduction data for receiving information from the information center apparatus 20 Information receiving means 111, important information selecting means 112 for selecting important information for the driver, and information notifying means 113 for notifying the driver of information are included.

  The imaging means is realized by a video camera, an ultrasonic sensor, an infrared sensor, or the like. The vehicle position positioning means 103 is realized by a GPS device, a gyroscope, or the like. The image recognition unit 102, the image recognition result information generation unit 103, and the important information selection unit 112 are realized by, for example, a control unit of a car navigation device that executes a program stored in a storage device of a car navigation device mounted on a vehicle. Is done. The image recognition result information transmitting unit 105 and the road environment reproduction data receiving unit 111 are realized by, for example, a transmission / reception unit (wireless LAN card, optical modem, etc.) of a car navigation apparatus and a control unit that controls the transmission / reception unit. The information notification unit 113 is realized by a display unit (a monitor or the like) of the car navigation apparatus and a control unit that controls the display unit.

  As shown in FIG. 1, the information center in-device 20 includes an image recognition result information receiving unit 106 for receiving information from the in-vehicle notification device 10, an image recognition result information storing unit 107 for storing the received information, Road environment reproduction data integrated generation means 108 that generates road environment reproduction data by integrating image recognition result information and the like, a road environment reproduction database 109 that stores road environment reproduction data, and a road that transmits information to the in-vehicle notification device 10 Environment reproduction data transmission means 110.

  The information center apparatus is an information processing apparatus such as a workstation. The image recognition result information receiving unit 106 and the road environment reproduction data transmitting unit 110 include, for example, a transmission / reception unit (wireless LAN card, optical modem, etc.) implemented in an information processing apparatus and a control unit that controls the transmission / reception unit. Realized. The image recognition result information storage unit 107 is realized by a storage device of an information processing device. The road environment reproduction data integrated generation unit 108 is realized by a control unit of the information processing apparatus that executes a program stored in the storage device of the information processing apparatus. The road environment reproduction database 109 is realized by a database device or the like connected to the information processing device.

  Although the in-vehicle notification device 10 is originally installed for each vehicle, the function of each unit of the in-vehicle notification device 10 is the same for all vehicles. Therefore, in FIG. 1, the configuration of the in-vehicle notification device 10 for only one vehicle. The description of the configuration of the in-vehicle notification device 10 for other vehicles is omitted. Further, the road environment reproduction data storage means is realized by the road environment reproduction database 109, and the vehicle position grasping means is realized by the vehicle position positioning means 103.

  The image pickup means 101 of the in-vehicle notification device 10 picks up images around the host vehicle every moment. Then, the imaging unit 101 outputs the captured video to the image recognition unit 102. The imaging means 101 includes a forward-facing camera for imaging the front of the host vehicle, and the forward-facing camera captures road signs, road signs, signals, store signs, and vehicles and pedestrians that precede the road. To do.

  Further, the imaging means 101 may include rear and side cameras as necessary. The rear-facing camera images a rear vehicle or the like, and the side-facing camera images a store signboard, a pedestrian on a sidewalk, a vehicle running on an alley, a pedestrian, or the like.

  In order to measure the distance to the object to be imaged, the imaging unit 101 may output a stereo image to the image recognition unit 102 by imaging with a plurality of cameras installed in each direction. The imaging unit 101 includes a distance measurement auxiliary sensor such as a millimeter wave radar or an ultrasonic sensor in combination with a camera in each direction, and the image recognition unit 102 captures an image captured by the camera and the distance measurement auxiliary sensor. Images may be output.

  The image recognition unit 102 recognizes a road sign, a road sign, a traffic signal, a store signboard, a vehicle, a pedestrian, or the like from a video captured by the imaging unit 101 based on a pattern of a recognition target object registered in the system in advance. The target object is extracted, and the attribute of the extracted recognition target object, the relative position with respect to the host vehicle, and the certainty of image recognition processing (hereinafter referred to as recognition certainty) are output to the image recognition result information generating unit 103. The recognition certainty is an index indicating the certainty of the result obtained by the image recognition, and the larger the value, the more accurate the recognition result of the image. The recognition confidence is generally calculated by a method called a simple similarity method or a composite similarity method. In addition, the relative position between the host vehicle and the recognition target object is generated based on distance information extracted from a stereo camera image or an ultrasonic sensor, and azimuth information of the host vehicle.

  Also, for example, the recognition certainty can be obtained using a neural network. In that case, the value of the output layer of the multilayer neural network is calculated, and the calculated value of the output layer is set as the recognition certainty.

  FIG. 2 is an explanatory diagram illustrating an example of an image captured by the imaging unit 101. In the example illustrated in FIG. 2, a road sign 201 indicating parking prohibition and an auxiliary sign 202 are included in the image output by the imaging unit 101. The image recognition unit 102 outputs image recognition result information based on the image shown in FIG.

  FIG. 3 is an explanatory diagram illustrating an example of image recognition result information output from the image recognition unit 102 based on the image illustrated in FIG. The information of the image recognition result includes code information 301 indicating that the output information relates to a road sign, relative position information 302 indicating a relative position of the sign with respect to the host vehicle, and a recognition time which is a time when the sign is recognized. 303, code information 304 indicating that the type of the sign is a parking prohibition sign as attribute information, information 305 indicating the date and time when the sign is effective, information 306 indicating the range where the sign is effective, image recognition And the recognition confidence 307 of the result. The attribute is the type of road sign (for example, stop sign or parking stop sign) when the recognition target object is a road sign, and when the recognition target object is a traffic signal, State (for example, red signal or blue signal).

  FIG. 4 is an explanatory diagram showing another example of the information of the image recognition result output by the image recognition unit 102. As an example of image recognition result information for another recognition target object, an example of image recognition result information when the imaging unit 101 images a target object whose attributes change with time, such as a traffic light, will be described. As shown in FIG. 4, the information of the image recognition result includes code information 401 indicating that the output information relates to a traffic signal, relative position information 402 indicating a relative position of the traffic signal with respect to the host vehicle, and a signal of the traffic signal. Code information 403 indicating the road range in which the signal is effective, recognition time 404 that is a recognition time of a traffic light, and code information 405 that indicates a lighting state of the traffic light (whether it is a green light or a red traffic light) as attribute information And the recognition certainty factor 406 of the image recognition result.

  The format of such image recognition result information is predetermined for each type of recognition target object. When a plurality of recognition target objects are recognized from a single image, the image recognition unit 102 sequentially outputs information about each recognition result to the image recognition result information generation unit 104.

  The vehicle position measuring means 103 acquires the absolute position of the host vehicle on the earth by GPS or the like and outputs it to the image recognition result information generating means 104. Based on the absolute position information of the host vehicle input from the vehicle position measurement unit 103, the image recognition result information generation unit 104 calculates the absolute position of the recognition target object included in the recognition result input from the image recognition unit 102. Convert to position. Then, the image recognition result information generation unit 104 encodes the converted recognition result, and outputs the encoded information to the image recognition result information transmission unit 105 and the important information selection unit 112 as image recognition result information.

  The image recognition result information transmitting unit 105 converts the image recognition result information input from the image recognition result information generating unit 104 into an optical signal, a radio wave signal, or the like, and transmits the optical signal to the information center apparatus 20 in the information center. The in-vehicle notification device 10 and the information center device 20 transmit and receive information using, for example, an FM multiplex broadcast or optical beacon system, which is a communication network of the VICS system. Moreover, the vehicle-mounted alerting | reporting apparatus 10 and the information center internal device 20 may transmit / receive information using other communication networks, such as wireless LAN.

  The image recognition result information receiving unit 106 of the information center apparatus 20 receives the image recognition result information transmitted from each in-vehicle notification device 10 every moment, and the received image recognition result information is stored in the image recognition result information storage unit 107. Output.

  The image recognition result information storage unit 107 stores the image recognition result information input from the image recognition result information receiving unit 106. Then, the image recognition result information storage unit 107 holds the stored image recognition result information as information indicating a history of the image recognition result information (hereinafter referred to as history information) for a predetermined time.

  The road environment reproduction database 109 stores road environment reproduction data that is information about each recognition target object on the road. The stored road environment reproduction data includes information on absolute position, attribute information, and information on total confidence that the recognition target object is estimated to exist at the current time. A method of calculating the total certainty factor will be described later. The road environment reproduction data is initialized immediately after the information center apparatus 20 is activated. After the initialization, the road environment reproduction data is generated as needed by the road environment reproduction data integrated generation unit 108, and the content is updated.

  The road environment reproduction data integrated generation means 108 has received new image recognition result information regarding the corresponding recognition target object from the in-vehicle notification device 10 for a certain period of time among road environment reproduction data stored in the road environment reproduction database 109. Delete no road environment reproduction data. Further, the road environment reproduction data integrated generation unit 108, when the road environment reproduction database 109 does not include road environment reproduction data corresponding to the recognition target object included in the received image recognition result information, the recognition target object. New road environment reproduction data corresponding to is generated.

  The road environment reproduction data integrated generation unit 108 is based on the image recognition result information held in the image recognition result information storage unit 107 and the road environment reproduction data before generation (one hour before) stored in the road environment reproduction database 109. Then, an absolute position and attribute estimation result at the current time of the recognition target object is generated. Further, the road environment reproduction data integrated generation unit 108 calculates a total certainty factor for the generated estimation result, and generates information on road environment reproduction data corresponding to the recognition target object.

  If the image recognition result information transmitted from the in-vehicle notification device 10 of each vehicle is always accurate, the road environment reproduction data integrated generation unit 108 converts the information included in the latest image recognition result information to the road environment. By writing the reproduction data, road environment reproduction data can be generated. However, since there is a possibility that the image recognition result information includes erroneous information, the road environment reproduction data integrated generation unit 108 performs the history information of the past image recognition result information (or the road environment reproduction one time ago). In consideration of the information contained in the data, an estimation result of the absolute position, attribute, etc. at the current time is generated.

  The procedure in which the road environment reproduction data integrated generation unit 108 generates road environment reproduction data will be specifically described. FIG. 5 is an explanatory diagram showing an example of the latest image recognition result information received by the image recognition result information receiving unit 106. As shown in FIG. 5, the information of the image recognition result information is that the stop sign (attribute = temporary stop) present at 35.00 north latitude and 136.00 east longitude is 50% at 18: 6 depending on the traveling vehicle. It has illustrated that it was detected by the recognition certainty of.

  FIG. 6 shows history information (stored in the image recognition result storage means 107) of past image recognition result information corresponding to the road sign that is a recognition target object and road environment reproduction data (stored in the road environment reproduction database 109). FIG. FIG. 6 (a) shows the history information of the image recognition result information at the past 18: 5 and 18: 3, and the road sign is recognized at a recognition confidence of 100% at 18: 5 and 18: 3. It has been illustrated that it was done. FIG. 6B illustrates that the total certainty factor included in the road environment reproduction data generated at 18:05 was 100%.

  FIG. 7 is an explanatory diagram showing an example of the latest road environment reproduction data generated by the road environment reproduction data integrated generation unit 108. The road environment reproduction data integrated generation means 108 shows the latest comprehensive confidence for the road sign, the history information of the past image recognition result information for two times shown in FIG. 6 (a), and the latest information shown in FIG. Calculation is performed by obtaining an average value of the recognition certainty included in the image recognition result information for three times including the image recognition result information. In this example, since the recognition certainty included in the image recognition result information for three times is 100%, 100%, and 50%, the road environment reproduction data integrated generation unit 108, as shown in FIG. The certainty factor is determined to be an average value of 84%.

  That is, when the image recognition result information receiving unit 106 receives the latest image recognition result information for the road sign, the road environment reproduction data integrated generation unit 108 stores the image recognition result stored in the image recognition result information storage unit 107. Image recognition result information regarding the same recognition target object is extracted from the history information of the information, an average value of the recognition certainty is calculated, and an overall certainty of the road environment reproduction data is generated. When there are a plurality of traveling vehicles and the image recognition result information is received from the plurality of in-vehicle notification devices 10, the image recognition result information storage unit 107 stores the history of the image recognition result information received from the plurality of in-vehicle notification devices 10. Information is stored, and the road environment reproduction data integrated generation means 108 can comprehensively generate road environment reproduction data based on the history information of image recognition result information from a plurality of vehicles.

  In addition, the road environment reproduction data integrated generation unit 108 does not determine the overall certainty by calculating the average value of the recognition certainty for three times, but calculates the average value of the recognition certainty for more times. For example, the total certainty factor may be determined based on other calculation criteria. Further, the road environment reproduction data integrated generation unit 108 calculates an average value of the recognition certainty factor included in the latest image recognition result information and the total certainty factor included in the road environment reproduction data one hour ago. The total certainty factor may be determined by considering the information.

  The situation in which the recognition certainty included in the newly acquired image recognition result information is deteriorated as compared with the recognition certainty included in the image recognition result information acquired in the past easily occurs due to a cause such as a poor front visibility of the vehicle. When the vehicle is in an environment where information cannot be transmitted / received to / from other vehicles or information centers via the network (that is, when the device installed in the vehicle provides information to the driver alone) The vehicle important information selection means may not notify the driver of information with low recognition confidence. According to the present invention, the image recognition result information recognized by each vehicle is collected by the information center in-center device 20 of the information center via the network. Since the information that has been aggregated and increased in reliability by the information center in-device 20 is distributed to each vehicle by means described later, road environment information that is effective even in vehicles and the like in which the front view is blocked is provided. Obtainable.

  Next, a case where the image recognition result information receiving unit 106 receives information related to a traffic light will be described as an example. Since the traffic signal changes with the passage of time, the traffic environment reproduction data may be generated based only on the latest image recognition result information, unlike a road sign. For example, with respect to a certain traffic light, image recognition result information including information that the attribute is a red signal at the same time is received from vehicles A, B, and C, and the recognition certainty included in the image recognition result information is 100%, Suppose that it was 90% and 20%. In this case, the road environment reproduction data integrated generation means 108 averages the total certainty of the road environment reproduction data at the current time for the traffic light with the recognition certainty included in the received three image recognition result information (100 % + 90% + 20%) / 3 = 70%.

  Next, a case where the image recognition result information receiving unit 106 receives image recognition result information having contradictory contents at the same time will be described as an example. For example, with respect to a certain traffic signal, information indicating that the traffic signal is a yellow signal from vehicle D at the same time as receiving image recognition result information including information that the attribute is a red signal at the same time from vehicles A, B, and C. Consider a case where image recognition result information is received. The recognition certainty included in the image recognition result information from vehicles A, B, and C was 100%, 90%, and 20%, respectively. The degree is assumed to be 10%. The road environment reproduction data integrated generation unit 108 calculates the average value of the certainty of recognition for each piece of information of the same attribute (lighting color), and the image recognition result information including the attribute having the larger calculated average value (in this example, Image recognition result information whose attribute is a red signal is selected, and information included in road environment reproduction data such as total confidence is generated. Note that the method in which the road environment reproduction data integrated generation unit 108 generates road environment reproduction data when image recognition result information having contradictory contents is received is not limited to the one shown in this example.

  Further, the road environment reproduction data integrated generation unit 108 includes both the road environment reproduction data stored in the road environment reproduction data storage unit and the history information of the image recognition result information stored in the image recognition result information storage unit. When there is no information corresponding to the recognition target object of the image recognition result information received by the image recognition result information receiving means, the recognition certainty included in the received image recognition result information is also used as the total certainty for the recognition target object. Good.

  When information about a recognition target object whose absolute position changes with time such as a pedestrian or a vehicle is received, the road environment reproduction data integrated generation unit 108 stores the image recognition result information storage unit. The estimation result of the absolute position of the recognition target object at the current time may be calculated from the history information of the past image recognition result information using a Kalman filter or the like. In this case, the total certainty factor is determined from the difference between the calculated absolute position estimation result and the absolute position included in the latest image recognition result information.

  Further, when information regarding a recognition target object whose absolute position changes with time such as a pedestrian or a vehicle is received, the road environment reproduction data integrated generation unit 108 stores the image recognition result information storage unit. The road environment reproduction data may be generated based only on information in which the difference between the recognition time and the current time is within a predetermined time in the history information of the image recognition result information.

  Next, a generation method in which the road environment reproduction data integrated generation unit 108 generates road environment reproduction data when no image recognition result information is received from any of the traveling vehicles will be described. The road environment reproduction data integrated generation unit 108 does not generate road environment reproduction data corresponding to a recognition target object whose absolute position and attribute (sign type, etc.) such as road signs and road surface signs do not change over time. The road environment reproduction data integrated generation unit 108 is configured to perform road environment reproduction data corresponding to a recognition target object whose absolute position and attributes (gender, age, lighting color, etc.) change over time, such as a pedestrian and a traffic light. The road environment reproduction data is generated by reflecting the influence of the passage of time.

  For example, when there is a possibility that the lighting state of the traffic light has changed, the road environment reproduction data integrated generation unit 108 reduces the total certainty factor at the current time at a predetermined rate from the total certainty factor one hour before. Generate by. For a recognition target object whose position changes, such as a pedestrian or a vehicle, the road environment reproduction data integrated generation unit 108 calculates the estimation result of the current time state from the history information of the past image recognition result information, and the total confidence Decrease the rate at a certain rate.

  The road environment reproduction data transmission means 110 converts the generated road environment reproduction data into an optical signal or a radio wave signal, and transmits it to the in-vehicle notification device 10 of each vehicle.

  The road environment reproduction data receiving unit 111 of the in-vehicle notification device 10 receives the road environment reproduction data transmitted from the road environment reproduction data transmission unit 110. Then, the road environment reproduction data reception unit 111 outputs the received road environment reproduction data to the important information selection unit 112. The road environment reproduction data receiving unit 111 ends the reception process immediately when the road environment reproduction data is not received within a predetermined time after the vehicle transmits the image recognition result information.

  The important information selection unit 112 is selected from the information included in the road environment reproduction data input from the road environment reproduction data reception unit 111 and the information included in the image recognition result information input from the image recognition result information generation unit 105. The information useful for the driver of the vehicle is selected and extracted. As an information selection method, the important information selection unit 112, for example, based on the information on the absolute position included in each information, information on the recognition target object existing in a place that enters the driver's field of view within a few seconds from the current time. select. Further, the important information selection unit 112 may select information having a high total certainty among road environment reproduction data. Further, the important information selection unit 112 determines that the information with the recognition certainty higher than the total certainty included in the road environment reproduction data is present in the image recognition result information input from the image recognition result information generating unit 105. The image recognition result information having a high recognition certainty may be selected. That is, the important information selection unit 112 selects information that is determined to be important to the driver from the road environment reproduction data or the image recognition result information.

  When the road environment reproduction data input from the road environment reproduction data receiving unit 111 and the information input from the image recognition result information generation unit 105 do not match, the important information selection unit 112 The information included in the reproduction data may be selected with priority, or the higher certainty is compared between the total certainty included in the road environment reproduction data and the recognition certainty included in the image recognition result information. Information may be selected.

  The important information selection unit 112 outputs the selected and extracted information to the information notification unit 113. The information notification means 113 notifies the driver of the information selected by the important information selection means 112 by video (monitor display or the like) or voice.

  Next, the operation of the present embodiment will be described. FIG. 8 is a flowchart illustrating an example of processing progress of processing executed by the in-vehicle notification device 10. FIG. 9 is a flowchart showing an example of processing progress of processing executed by the information center apparatus 20. The in-vehicle notification device 10 and the information center in-device 20 execute the processes shown in FIGS. 8 and 9 asynchronously with each other.

  First, the operation of the in-vehicle notification device 10 will be described. The imaging unit 101 captures an image around the vehicle (step S801). The image recognition unit 102 recognizes a predetermined recognition target object such as a road sign or a traffic light from the video input from the imaging unit 101, and outputs the recognized recognition result (step S802). The vehicle position measuring means 103 acquires the absolute position of the vehicle from GPS or the like (step S803). The image recognition result information generating unit 105 uses the own vehicle absolute position information acquired by the vehicle position measuring unit 103, and based on information included in the recognition result output by the image recognition unit 102, the absolute position of the recognition target object. The information such as the attribute and the recognition certainty is encoded, and the encoded image recognition result information is output (step S804). Then, the image recognition result information transmitting unit 105 transmits the output image recognition result information to the information center internal device 20 of the information center (step S805).

  After the road center reproduction data is generated and distributed to each vehicle by the information center internal device 20, the road environment reproduction data reception unit 111 receives the road environment reproduction data reception unit 110 of the information center internal unit 20 from the road environment reproduction data reception unit 110. The transmitted road environment reproduction data is received (step S806). If the road environment reproduction data cannot be received within a predetermined time after the host vehicle transmits the image recognition result information, the road environment reproduction data receiving unit 111 immediately ends the reception process. The important information selection means 112 is useful for the driver of the own vehicle from the road environment reproduction data received by the road environment reproduction data receiving means 111 and the image recognition result information output by the image recognition result information generation means 104 of the own vehicle. Information is selected and extracted (step S807). The information notification unit 113 notifies the driver of the selected information (step S808). When the notification is completed, the process returns to step S801 again, and the processes from step S801 to step S808 are repeatedly executed.

  Next, the operation of the information center apparatus 20 will be described. The road environment reproduction data integrated generation means 108 initializes road environment reproduction data stored in the road environment reproduction database 109 immediately after the information center apparatus 20 is activated (step S901). The image recognition result information receiving unit 106 receives the image recognition result information transmitted from the in-vehicle notification device 10 of each vehicle (step S902). Then, the image recognition result information receiving unit 106 confirms whether the image recognition result information has been received (step S903). If it can be received, the image recognition result information receiving means 106 outputs the received image recognition result information to the image recognition result information storage means 107, and the image recognition result information storage means 107 stores the image recognition result information. (Step S904). If the information cannot be received, step S904 is not executed, and step S905 is executed next.

  The road environment reproduction data integrated generation unit 108 generates road environment generation data based on the history information of the image recognition result information stored in the image recognition result information storage unit 107 and the road environment reproduction data one hour before ( Step S905). After the generation of the road environment reproduction data, the road environment reproduction data transmission unit 110 transmits the latest road environment reproduction data at the current time to the in-vehicle notification device 10 of each vehicle (step S906). Then, after the information is distributed to each vehicle, the process returns to step S902 again, and the processes from step S902 to step S906 are repeatedly executed.

  As described above, according to the present invention, information relating to an object that is not provided by map information of a car navigation system such as road signs, signals, vehicles, and pedestrians (ie, road environment information) can be stored on the road. Extraction is possible without installing infrastructure. Therefore, a road environment information notification device can be constructed at a lower cost than AHS or the like.

  In addition, in the conventional real-time information providing system using an in-vehicle camera, only the information recognized by the video of the camera mounted on the host vehicle is notified to the driver, so incorrect information is notified due to road conditions, sunlight conditions, etc. There is a problem that the information may be displayed, and there is a problem that the information cannot be notified sufficiently earlier than the driver notices visually. According to the present invention, the image recognition result information recognized by the in-vehicle notification device mounted on each vehicle is integrated based on the recognition reliability in the information center, so that highly reliable information can be provided to the driver. effective. In addition, since it is possible to obtain information on objects to be recognized that exist in places that cannot be captured by the camera of the vehicle at the current time, the driver can detect in advance situations and troubles that may occur in the future, and anticipate such troubles. The effect is that you can drive.

  Here, the effect obtained by acquiring in advance information on an object that cannot be captured by the own vehicle camera will be specifically described with reference to FIG. FIG. 10 is an explanatory diagram showing a specific example of a road environment. FIG. 10 shows a situation where the parked vehicle 1003 is stopped in the left lane of the one-side two-lane road. Since the own vehicle 1001 is obstructed by the large preceding vehicle 1002, the own vehicle 1001 is determined according to the conventional real-time information providing system that determines the road environment based only on the information of the image recognition device of the own vehicle. The driver cannot recognize the presence of the parked vehicle 1003 until the preceding vehicle 1002 changes lanes.

  According to the road environment information notification device according to the present invention, after the image recognition device mounted on the preceding vehicle 1002 and the preceding vehicle 1004 detects the parked vehicle 1003, road environment reproduction data regarding the parked vehicle 1003 is transferred from the information center to the host vehicle 1001. Since the vehicle 1001 is distributed, the driver of the own vehicle 1001 can notice the presence of the parked vehicle 1003 earlier than the preceding vehicle 1002 changes the lane, and can change the lane with a margin.

Embodiment 2. FIG.
Next, a second embodiment of the present invention will be described. FIG. 11 is a block diagram illustrating an example of the configuration of the road environment information notification device according to the second embodiment. As shown in FIG. 11, the road environment information notification device 11 includes an imaging unit 1101, an image recognition unit 1102, a vehicle position measurement unit 1103, an image recognition result information generation unit 1104, and a camera installed in the vehicle. Image information transmission means 1105, image recognition result information reception means 1106, image recognition result information storage means 1107, road environment reproduction data integrated generation means 1108, road environment reproduction database 1109, important information selection means 1112, information Notification means 1113.

  In the configuration of the present embodiment, the road environment information notification device 11 includes the image recognition result information receiving means included in the information center in-device 20 in addition to the components of the in-vehicle notification device 10 in the first embodiment. 106, an image recognition result information storage unit 107, a road environment reproduction data integrated generation unit 108, and a road environment reproduction database 109. In the first embodiment, information is transmitted and received between the in-vehicle notification device 10 and the information center device 20, whereas the present embodiment provides information between the road environment information notification devices 11 of a plurality of vehicles. Is different from the first embodiment in that transmission / reception is performed. Therefore, the road environment information notification system includes a plurality of road environment information notification devices 11 installed in each vehicle shown in FIG.

  Since each means of the road environment information notification device 11 has a common function in all vehicles, FIG. 11 illustrates only the configuration of the road environment information notification device 11 for one vehicle (vehicle A). This vehicle is omitted.

  Hereinafter, the function of each means included in the road environment information notification device 11 will be described. The imaging unit 1101, the image recognition unit 1102, the vehicle position measurement unit 1103, the image recognition result information generation unit 1104, the image recognition result information transmission unit 1105, the important information selection unit 1112, and the information notification unit 1113 are implemented by the first method. This is the same as the embodiment.

  The image recognition result information receiving unit 1106 is realized by, for example, a transmission / reception unit (wireless LAN card, optical modem, etc.) of a car navigation apparatus and a control unit that controls the transmission / reception unit. The image recognition result information storage unit 1107 is realized by a storage device of a car navigation device. The road environment reproduction data integrated generation unit 1108 is realized by, for example, a control unit of the car navigation device that executes a program stored in the storage device of the car navigation device. The road environment reproduction database 1109 is realized by, for example, a storage device incorporated in a car navigation device.

  The image recognition result information generation unit 1104 outputs the image recognition result information to the image recognition result information transmission unit 1105 and the image recognition result information storage unit 1107. The image recognition result information transmission unit 1105 transmits the image recognition result information input from the image recognition result information generation unit 1104 to the road environment information notification device 11 of another vehicle. The image recognition result information receiving unit 1106 receives image recognition result information transmitted from another vehicle.

  The image recognition result information storage unit 1107 stores the image recognition result information output from the image recognition result information reception unit 1106 and the image recognition result information output from the image recognition result information generation unit 1104 as history information of the image recognition result information. To do.

  The road environment reproduction data integrated generation means 1108 is based on the history information of the image recognition result information stored in the image recognition result information storage means 1107 and the road environment reproduction data one hour before stored in the road environment reproduction database 1109. An estimation result of the absolute position and attribute of the recognition target object at the time is generated. Further, the road environment reproduction data integrated generation unit 1108 calculates the overall certainty factor of the generated estimation result and generates road environment reproduction data. The processing procedure by which the road environment reproduction data integrated generation unit 1108 generates road environment reproduction data is the same as that in the first embodiment. Then, the important information selection unit 1112 selects and extracts information useful for the driver of the host vehicle from the road environment reproduction data 1108.

  The functions of the imaging unit 1101, the image recognition unit 1102, the vehicle position positioning unit 1103, and the information notification unit 1113 are the same as those in the first embodiment.

  Next, the operation of the second embodiment will be described. FIG. 2 is a flowchart illustrating an example of processing progress of processing executed by the road environment information notification device 11. First, immediately after the activation of the road environment information notification device 11, the road environment reproduction data integrated generation unit 1108 initializes the road environment reproduction data stored in the road environment reproduction database 109 (step S1201). The imaging unit 1101 captures an image around the vehicle (step S1202). The image recognition unit 1102 recognizes a preset recognition target object such as a road sign or a traffic light from the video input from the imaging unit 1101, and outputs the recognized recognition result (step S1203). The vehicle position positioning means 1103 acquires the absolute position of the host vehicle from GPS or the like (step S1204).

  The image recognition result information generation unit 1105 refers to the own vehicle absolute position information acquired by the vehicle position measurement unit 1103, and based on the image recognition result from the image recognition unit 1102, the absolute position, attribute, and recognition reliability of the recognition target object. Are encoded, and the encoded image recognition result information is output (step S1205). The image recognition result information storage unit 1107 stores the image recognition result information from the image recognition result information generation unit 1105 (step S1206), and the image recognition result information transmission unit transmits the image recognition result information to another vehicle. (Step S1207).

  After transmitting the image recognition result information, the image recognition result information receiving unit 1106 receives the image recognition result information transmitted from the other vehicle (step S1208). The image recognition result information receiving unit 1106 determines whether image recognition information has been received from the road environment information notification device 11 of another vehicle (step S1209). When the information is received, the image recognition result information storage unit 1107 stores the received image recognition result information (step S1210). If no information is received, the process of step S1210 is not executed, and step S1211 is executed next.

  The road environment reproduction data integrated generation unit 1108 refers to the history information of the image recognition result information stored in the image recognition result information storage unit 1107 and generates an estimation result of the position and attribute of the recognition target object at the current time. Then, road environment reproduction data stored in the road environment reproduction database 1109 is generated (step S1211). The important information selection unit 1112 selects and extracts information useful to the driver of the host vehicle from the information included in the road environment reproduction data (step S1212). The information notification unit 1113 notifies the driver of the extracted information by video (monitor display or the like) or voice (step S1213).

  As described above, in the second embodiment, unlike the first embodiment, the road environment information notification device 11 is installed in each vehicle without installing the information center in-device 20 in the information center. An environmental information notification system can be constructed.

  FIG. 13 is an explanatory diagram illustrating another example of information included in the image recognition result information. In the first embodiment and the second embodiment, the case where the image recognition result information includes the type of recognition target object, the attribute, the absolute position, the recognition time, and the recognition certainty has been described. As shown in FIG. 13, the image recognition result information may include vehicle number information 1301 that can identify the vehicle. By including the vehicle number information 1301, the reliability of road environment reproduction data can be further improved.

  Hereinafter, the effect that the image recognition result information includes the vehicle number information 1301 will be described. The road environment information reproduction data generation means 108, 1108 compares each image recognition result information received from each vehicle with road environment reproduction data generated by comprehensively determining the image recognition result information, thereby obtaining the road environment information. It is determined that there is a high possibility that the image recognition result information whose reproduction data and information content match is correct information. Further, the road environment information reproduction data generation means 108 and 1108 determine that the image recognition result information in which the content of the information is inconsistent with the road environment reproduction data is likely to be incorrect information.

  Then, the road environment information reproduction data generation means 108 and 1108 count the frequency at which the image recognition result information is determined to be correct information (hereinafter referred to as vehicle-specific correct information frequency) for each vehicle number, thereby correcting the vehicle-specific correct information. Generate an information frequency table. FIG. 14 is an explanatory diagram showing an example of the contents of the vehicle-specific correct information frequency table generated by the road environment information reproduction data generating means 108 and 1108. In this example, as shown in FIG. 14, the information information included in the image recognition result information received from the vehicle of the image recognition result information received from the vehicle of the image recognition result information received from the vehicle whose vehicle number is “0123456” is low. Is exemplified that there is a high possibility that erroneous information is notified for some reason.

  The road environment reproduction data integrated generation means 108, 1108 generates the road environment reproduction data by estimating the recognition reliability of the image recognition result information received from the vehicle having the low vehicle-specific correct information frequency, thereby generating the road environment reproduction data. Mixing erroneous information can be reduced, and more reliable road environment reproduction data can be output.

  For example, the road environment reproduction data integrated generation means 108, 1108 subtracts a predetermined value α (> 0) from the recognition certainty of the image recognition result information received from a vehicle with low vehicle-specific positive information frequency, and subtracts it. Then, road environment reproduction data is generated based on the recognition certainty obtained.

  Further, the road environment reproduction data integrated generation means 108, 1108 multiplies a certain value β (0 <β <1) determined in advance from the recognition reliability of the image recognition result information received from a vehicle having a low vehicle-specific positive information frequency. Then, road environment reproduction data may be generated based on the recognition certainty obtained by multiplication.

Embodiment 3 FIG.
Next, a third embodiment of the present invention will be described. FIG. 15 is a block diagram illustrating an example of a configuration of a road environment information notification device according to the third embodiment. As shown in FIG. 15, in the present embodiment, the in-vehicle notification device 12 includes an input operation means 1514 in addition to the components of the in-vehicle notification device 10 in the first embodiment.

  The input operation unit 1514 instructs the apparatus to select the type of information desired to be acquired from the information notification unit 1513, set the scheduled travel route, or whether to transmit information from the image recognition result information transmission unit 1505 according to the driver's operation. To do. Accordingly, the input operation means 1514 serves as a user interface between the road environment information notification device and the driver, and is realized by, for example, a remote controller of a car navigation device.

  The imaging unit 1501, the image recognition unit 1502, the vehicle position measurement unit 1503, the image recognition result information generation unit 1504, the image recognition result information transmission unit 1505, the road environment reproduction data reception unit 1511, and the important information included in the in-vehicle notification device 12 are included. The functions of the selection unit 1512 and the information notification unit 1513 are the same as those in the first embodiment. The image recognition result information receiving unit 1506, the image recognition result information storage unit 1507, the road environment reproduction data integrated generation unit 1508, the road environment reproduction database 1509, and the road environment reproduction data transmission unit 1510 included in the information center apparatus 22 are also included. Functions and implementation methods are the same as those in the first embodiment.

  In the present embodiment, the image recognition result information transmitting unit 1505 does not always transmit the image recognition result information to the information center internal device 22 of the information center, but is set to enable information transmission by the input operation unit 1514. Only the information is transmitted to the information center apparatus 22. By doing so, it is possible to prevent others from knowing the position of the vehicle if necessary.

  Further, the important information selection unit 1512 selects and extracts information on the recognition target object existing on the planned travel route specified by the input operation unit 1514, or the type of recognition target object specified by the input operation unit 1514. Select and extract information about. By doing so, the driver can efficiently acquire necessary information.

  The input operation means 1514 may be added to the in-vehicle notification device 10 in the first embodiment, or the input operation means may be added to the road environment information notification device 11 in the second embodiment.

Embodiment 4 FIG.
Next, a fourth embodiment of the present invention will be described. FIG. 16 is a block diagram illustrating an example of a configuration of a road environment information notification device according to the fourth embodiment. As shown in FIG. 16, in this embodiment, the in-vehicle notification device 13 includes an operation state monitoring unit 1615 in addition to the components of the in-vehicle notification device 10 in the first embodiment. The operation state monitoring unit 1615 is realized by, for example, a control unit of the car navigation device that executes a program stored in the storage device of the car navigation device.

  The operation state monitoring unit 1615 compares the information included in the image recognition result information input from the image recognition information encoding unit 1604 with the information included in the road environment reproduction data input from the road environment reproduction data receiving unit 1611. To do. When the frequency at which information whose contents do not match between the image recognition result information and the road environment reproduction data is included exceeds a predetermined value, the operation state monitoring unit 1615 It is determined that a malfunction has occurred in the imaging unit 1601 or the image recognition unit 1602. Then, the operation state monitoring unit 1615 outputs the caution information to the information notification unit 1613.

  For example, the road environment reproduction data includes information “There is a parking prohibition sign at position A”, whereas the image recognition result information includes information “There is a parking prohibition sign at position A”. If it is included, or the road environment reproduction data includes information “the signal at position B is a green signal”, the image recognition result information indicates that the signal at position B is a red signal. ”Is included, the operation state monitoring unit 1615 determines that the content does not match between the image recognition result information and the road environment reproduction data. Then, the operation state monitoring unit 1615 determines that a problem has occurred in the imaging unit 1601 or the image recognition unit 1602 when the frequency of including information whose contents do not match exceeds 50%, for example.

  By doing so, the driver can recognize the malfunction of the device mounted on the host vehicle.

  Note that the imaging unit 1601, the image recognition unit 1602, the vehicle position measurement unit 1603, the image recognition result information generation unit 1604, the image recognition result information transmission unit 1605, the road environment reproduction data reception unit 1611, and the important information included in the in-vehicle notification device 13 are included. The functions of the selection unit 1612 and the information notification unit 1613 are the same as those in the first embodiment. The image recognition result information receiving unit 1606, the image recognition result information storage unit 1607, the road environment reproduction data integrated generation unit 1608, the road environment reproduction database 1609, and the road environment reproduction data transmission unit 1610 included in the information center apparatus 23 are also included. Functions and implementation methods are the same as those in the first embodiment.

  The operation status monitoring means 1615 may be added to the in-vehicle notification device 10 in the first embodiment, or the operation status monitoring means 1615 is added to the road environment information notification device 11 in the second embodiment. Also good.

Embodiment 5. FIG.
Next, a fifth embodiment of the present invention will be described. FIG. 17 is a block diagram illustrating an example of a configuration of a road environment information notification device according to the fifth embodiment. As shown in FIG. 17, in this embodiment, the in-vehicle notification device 14 includes a video recording unit 1716 immediately before impact detection, in addition to the components of the in-vehicle notification device 10 in the first embodiment. The video recording means 1716 immediately before impact detection is realized by, for example, an impact detection sensor and a control unit of a car navigation device that controls the impact sensor.

  The video recording means 1716 immediately before impact detection records the video imaged by the imaging means 1701 for a predetermined time, and the video is recorded when a strong impact is detected on the vehicle (that is, when an accident occurs). The image recorded immediately before the impact is sensed is retained.

  By adding the video recording means 1716 immediately before the impact detection, it is possible to hold the video around the time when the host vehicle encounters an accident as information, and the cause of the accident can be investigated quickly.

  Note that the imaging means 1701, the image recognition means 1702, the vehicle position positioning means 1703, the image recognition result information generation means 1704, the image recognition result information transmission means 1705, the road environment reproduction data reception means 1711, and the important information included in the in-vehicle notification device 14 are included. The functions of the selection unit 1712 and the information notification unit 1713 are the same as those in the first embodiment. The image recognition result information receiving unit 1706, the image recognition result information storage unit 1707, the road environment reproduction data integrated generation unit 1708, the road environment reproduction database 1709, and the road environment reproduction data transmission unit 1710 included in the information center apparatus 24 are also included. Functions and implementation methods are the same as those in the first embodiment.

  The video recording means 1716 immediately before impact detection may be added to the in-vehicle notification device 10 in the first embodiment, or the video recording means immediately before impact detection is added to the road environment information notification device 11 in the second embodiment. 1716 may be added.

  Moreover, the vehicle-mounted alerting | reporting apparatus 14 may also contain a vehicle speed measurement means. The vehicle speed measuring means senses the moving speed of the vehicle. Then, for the road environment reproduction data in which the recognition target object is a red signal or a stop sign, the important information selection means includes the absolute position of the recognition target object included in the road environment reproduction data and the absolute position of the vehicle at the current time. The road environment reproduction data is selected only when the moving speed of the own vehicle sensed by the vehicle speed measuring means exceeds a predetermined threshold with respect to the distance from the own vehicle to the recognition target object calculated based on To extract. The vehicle speed measuring means is realized by a speedometer or the like.

Embodiment 6 FIG.
Next, a sixth embodiment of the present invention will be described. FIG. 18 is a block diagram showing an example of the configuration of the road environment information notification device according to the sixth embodiment. As shown in FIG. 18, in this embodiment, the in-vehicle notification device 15 includes an unextracted object determination unit 1817 in addition to the components of the in-vehicle notification device 10 in the first embodiment. The unextracted object determination unit 1817 is realized by, for example, a control unit of the car navigation device that executes a program stored in the storage device of the car navigation device.

  The unextracted object determination means 1817 is a recognition target object included in the road environment reproduction data as information and is present at a position within a range recognizable from the own vehicle and is not recognized by the image recognition means 1802 of the own vehicle. A target object is selected and extracted, and the extracted information is output to the image recognition unit 1802.

  Then, when performing the next recognition process, the image recognition unit 1802 performs a top-down recognition process that performs an image recognition process based on the recognition target object from the unextracted object determination unit 1817 in addition to the normal recognition process. Do.

  By doing so, the top-down recognition method can be applied even to a recognition target object that is difficult to detect by the bottom-up image recognition method in which the image recognition unit 1802 performs image recognition processing based only on the input image normally performed. Thus, the recognition target object can be recognized.

  Note that the imaging unit 1801, the image recognition unit 1802, the vehicle position measurement unit 1803, the image recognition result information generation unit 1804, the image recognition result information transmission unit 1805, the road environment reproduction data reception unit 1811, and the important information included in the in-vehicle notification device 15 are included. The functions of the selection unit 1812 and the information notification unit 1813 are the same as those in the first embodiment. Further, the image recognition result information receiving unit 1806, the image recognition result information storage unit 1807, the road environment reproduction data integrated generation unit 1808, the road environment reproduction database 1809, and the road environment reproduction data transmission unit 1810 included in the information center apparatus 25 are included. Functions and implementation methods are the same as those in the first embodiment.

  The unextracted object determination means 1817 may be added to the in-vehicle notification device 10 in the first embodiment, or the unextracted object determination means 1817 is added to the road environment information notification device 11 in the second embodiment. May be.

  As described above, according to the present invention, road environment information related to an object that is not provided by map information of a car navigation system such as a road sign, a signal, a vehicle, and a pedestrian can be driven without installing infrastructure on the road. Can be notified. Therefore, a system can be constructed at a lower cost than AHS or the like.

  In the conventional invention, only the information recognized by imaging with the on-vehicle camera of the own vehicle is used, so the problem of misrecognizing image information due to incompleteness of computer vision technology, or the driver perceives visually However, according to the present invention, the image recognition result information from each vehicle is integrated in the information center or each vehicle, so that the reliability is high. Since the road environment reproduction data can be generated and the highly reliable road environment reproduction data can be notified to the driver, the driver can determine the situation based on the highly reliable information. In addition, since it is also possible to acquire in advance information on objects that exist in places that cannot be imaged by the camera of the host vehicle at the current time, the driver can know in advance the situations and troubles that may occur in the future. .

  The configurations shown in FIGS. 1, 11, 15, 16, 16, and 18 can be realized by software. That is, in the present embodiment, an image recognition process for recognizing and extracting an image related to a road environment from image information around a captured vehicle, and outputting the extracted image recognition result information, and image recognition result information And image recognition result information generation processing for generating image recognition result information based on information on the absolute position of the vehicle, which is the position of the vehicle on the earth, and the image recognition result information, and an integrated road environment The road environment reproduction data integrated generation process that generates road environment reproduction data that is information, road environment reproduction data, and image recognition result information are compared, and information related to the recognition target object that is useful to the driver is selected and extracted. By the road environment information notification program for executing the important information selection processing to be performed and the information notification processing for notifying the driver of information regarding the extracted recognition target object, Road environment information notifying apparatus or road environment information notification method can be realized.

It is a block diagram which shows an example of a structure of the road environment information notification apparatus by this invention. It is explanatory drawing which shows an example of the image which an imaging means images. It is explanatory drawing which shows an example of the image recognition result which an image recognition means outputs. It is explanatory drawing which shows the other example of the image recognition result which an image recognition means outputs. It is explanatory drawing which shows an example of image recognition result information. It is explanatory drawing which shows an example of the historical information of image recognition result information, and road environment reproduction data memorize | stored in a road environment reproduction database. It is explanatory drawing which shows an example of the road environment reproduction data produced | generated by the road environment reproduction data integrated production | generation means. It is a flowchart which shows an example of the process progress of the process which a vehicle-mounted alerting device performs. It is a flowchart which shows an example of the process progress of the process which the apparatus in an information center performs. It is explanatory drawing which shows the specific example of a road environment. It is a block diagram which shows the other example of a structure of the road environment information notification apparatus by this invention. It is a flowchart which shows an example of the process progress of the process which the road environment information notification apparatus by this invention performs. It is explanatory drawing which shows the other example of image recognition result information. It is explanatory drawing which shows an example of the content of the positive information frequency table classified by vehicle. It is a block diagram which shows the further another example of a structure of the road environment information notification apparatus by this invention. It is a block diagram which shows the further another example of a structure of the road environment information notification apparatus by this invention. It is a block diagram which shows the further another example of a structure of the road environment information notification apparatus by this invention. It is a block diagram which shows the further another example of a structure of the road environment information notification apparatus by this invention.

Explanation of symbols

10, 12, 13, 14, 15 In-vehicle notification device 11 Road environment information notification device 20, 22, 23, 24, 25 Information center device 101, 1101, 1501, 1601, 1701, 1801 Imaging means 102, 1102, 1502, 1602, 1702, 1802 Image recognition means 103, 1103, 1503, 1603, 1703, 1803 Vehicle position positioning means 104, 1104, 1504, 1604, 1704, 1804 Image recognition result information generation means 105, 1105, 1505, 1605, 1705 1805 Image recognition result information transmission means 106, 1106, 1506, 1606, 1706, 1806 Image recognition result information reception means 107, 1107, 1507, 1607, 1707, 1807 Image recognition result information storage means 108, 1108, 1 08, 1608, 1708, 1808 Road environment reproduction data integrated generation means 109, 1109, 1509, 1609, 1709, 1809 Road environment reproduction database 110, 1510, 1610, 1710, 1810 Road environment reproduction data transmission means 111, 1511, 1611, 1711, 1811 Road environment reproduction data reception means 112, 1112, 1512, 1612, 1712, 1812 Important information selection means 113, 1113, 1513, 1613, 1713, 1813 Information notification means 1514 Input operation means 1615 Operation state monitoring means 1716 Impact detection Immediately preceding video storage means 1817 Unextracted object determination means

Claims (17)

A road environment information notification device that is installed in a vehicle and notifies a driver of road environment information, which is information about the road environment,
Imaging means for imaging the periphery of the vehicle and outputting image information;
Image recognition means for recognizing and extracting an image related to the road environment from the imaged image information, and outputting information of the extracted image recognition result;
Image recognition result information transmitting means for transmitting image recognition result information including information on the result of the image recognition to another vehicle;
Image recognition result information receiving means for receiving image recognition result information from another vehicle;
Road environment reproduction data integration generating means for integrating the image recognition result information of the own vehicle and the received image recognition result information of other vehicles and generating road environment reproduction data which is information related to the integrated road environment;
A road environment information notification device comprising: information notification means for notifying a driver of information included in the road environment reproduction data. Vehicle position grasping means for grasping the absolute position of the vehicle;
Image recognition result information generating means for generating image recognition result information based on information on the result of image recognition output by the image recognition means and information on the absolute position of the vehicle;
Image recognition result information storage means for storing the image recognition result information of the own vehicle generated by the image recognition result information generation means, and history information of the received image recognition result information of other vehicles;
A road environment reproduction data storage means for storing road environment reproduction data including the latest estimation result relating to the absolute position and attribute of the recognition target object at the current time and the overall certainty factor indicating the certainty of the estimation result. The road environment information notifying device according to Item 1. An important information selection unit that selects and extracts information determined to be important to the driver from road environment reproduction data output from the road environment reproduction data integrated generation unit is selected from the road environment reproduction data. The road environment information notification device described. The image recognition means extracts the attribute of the recognition target object, the relative position with respect to the vehicle, and the recognition time by performing image recognition based on the captured image information, and confirms the result obtained by the image recognition. A recognition certainty factor is calculated, and information on the result of image recognition including the attribute, relative position, recognition time, and recognition certainty factor is output,
The image recognition result information generating means calculates the absolute position of the recognition target object based on the relative position of the recognition target object included in the image recognition information with respect to the vehicle and the absolute position of the vehicle acquired by the vehicle position grasping means. And outputting information including the absolute position, attribute, recognition time, and recognition confidence of the recognition target object as image recognition result information,
The road environment reproduction data integrated generation means sets the value of the recognition certainty included in the history information of the image recognition result information stored in the image recognition result information storage means and the total certainty included in the road environment reproduction data one hour before. Based on the information content included in the history information of the image recognition result information and the information content included in the road environment reproduction data, the absolute position and attribute estimation result of the recognition target object at the current time To calculate the latest road environment reproduction data by updating the content of the road environment reproduction data based on the estimation result,
The road environment information notification device according to claim 3, wherein the information notification means notifies the driver of the information extracted by the important information selection means by video display or voice. When the image recognition result information storage means stores the image recognition result information newly received by the image recognition result information receiving means, the road environment reproduction data integrated generation means is a recognition certainty factor included in the history information of the image recognition result information. The road environment information notification device according to claim 4, wherein a total certainty factor is calculated by calculating an average value of the road environment information. The road environment reproduction data integrated generation means includes the image recognition result in any of the road environment reproduction data stored in the road environment reproduction data storage means and the history information of the image recognition result information stored in the image recognition result information storage means. When there is no information corresponding to the recognition target object of the image recognition result information received by the information receiving means, the recognition certainty included in the received image recognition result information is set as a total certainty for the recognition target object. The road environment information notification device according to claim 4 or 5. When the attributes of the recognition target object change over time, the road environment reproduction data integrated generation unit has a difference between the recognition time and the current time in the history information of the image recognition result information stored in the image recognition result information storage unit. The road environment information notification device according to any one of claims 3 to 6, wherein road environment reproduction data is generated based only on information within a predetermined time period. When the image recognition result information receiving means does not receive the image recognition result information including the information of the specific recognition target object for a predetermined time, the road environment reproduction data integration generating means The road environment information notification device according to any one of claims 3 to 7, wherein the road environment reproduction data corresponding to the recognition target object is deleted from the stored road environment reproduction data. The image recognition result information generating means generates image recognition result information including vehicle number information for specifying a vehicle,
The road environment reproduction data integrated generation unit preliminarily stores the vehicle number information included in the image recognition result information when the content of the image recognition result information received by the image recognition result information reception unit does not match the content of the road environment reproduction data. If the content of the image recognition result information stored for a certain fixed time and received from the vehicle of the specific vehicle number information does not match the content of the road environment reproduction data at a rate exceeding the predetermined frequency, The road environment information notification device according to any one of claims 3 to 8, wherein the recognition reliability included in the received image recognition information is estimated to be low to generate new road environment reproduction data. Input operation means for inputting instructions related to the planned travel route,
The road environment information notification device according to any one of claims 3 to 9, wherein the important information selection unit selects and extracts information related to the recognition target object existing on the planned travel route. The input operation means has a function of inputting designation of information to be acquired,
The road environment information notification device according to any one of claims 3 to 10, wherein the important information selection unit selects and extracts information based on the designation input by the input operation unit. The input operation means has a function of inputting designation as to whether or not to transmit image recognition result information to another vehicle,
The image recognition result information transmitting means transmits the image recognition result information only when it is designated to transmit the image recognition result information by an input of the input operation means. The road environment information notification device according to any one of the above items. Imaging by determining that a problem has occurred when the content of the image recognition result information output by the image recognition result information generation means and the content of the road environment reproduction data do not coincide with each other beyond a predetermined frequency An operation state monitoring means for monitoring the operation state of the means or the image recognition means,
The road environment information notification according to any one of claims 3 to 12, wherein the information notification unit notifies the driver of information regarding the detected fault when the operation state monitoring unit detects the fault. apparatus. An image storage means immediately before impact detection that stores an image captured by the imaging means for a predetermined period of time, stops recording of a new image when an impact on the vehicle is detected, and retains an image immediately before the impact is detected The road environment information notification device according to any one of claims 3 to 13. Vehicle speed measuring means for sensing the moving speed of the vehicle,
For the road environment reproduction data in which the recognition target object attribute is a red signal or a stop sign, the important information selection unit is configured to detect the absolute position of the recognition target object included in the road environment reproduction data and the vehicle at the current time. The road environment reproduction data relating to the recognition target object is selected only when the moving speed of the vehicle exceeds a predetermined threshold with respect to the distance from the vehicle calculated based on the absolute position to the recognition target object. The road environment information notification device according to any one of claims 3 to 14, which is extracted. An unextracted object that determines the recognition target object when the recognition target object corresponding to the received road environment reproduction data is in a position that can be recognized from the host vehicle and the image recognition means has not recognized the recognition target object. A determination means,
The image recognition unit extracts an unextracted object by top-down image analysis processing that performs image analysis based on information on the recognition target object determined by the unextracted object determination unit. Road environment information notification device. A road environment information notification program installed in a vehicle for notifying a driver of road environment information, which is information about the road environment,
On the computer,
Processing for recognizing and extracting an image related to the road environment from the image information around the imaged vehicle, and outputting information of the extracted image recognition result;
Processing for transmitting image recognition result information including information on the result of image recognition to another vehicle;
Processing for receiving image recognition result information from other vehicles;
A process of integrating the image recognition result information of the own vehicle and the received image recognition result information of another vehicle, and generating road environment reproduction data that is information relating to the integrated road environment;
A road environment information notification program for executing a process of notifying a driver of information included in the road environment reproduction data.

Images