JP2012009923A - Landscape image synthesis device, landscape image synthesis method and landscape image synthesis program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a driver with visual and useful information on the way of driving.SOLUTION: A landscape image synthesis device inputs a first landscape image photographed by an on-vehicle camera 11a of a first vehicle 10, receives a second landscape image photographed by an on-vehicle camera 21a of a second vehicle 20 via a wireless communication network 700, specifies an area with highest similarity by comparing an area of the first landscape image and an area of the second landscape image, superimposes the two landscape images at a specified area, and displays a superimposed synthesized landscape image.

Description

  The present invention relates to a technique for synthesizing two landscape images each shot by an in-vehicle camera of a different car.

  At present, communication networks are rapidly spread not only in locationally stationary environments such as homes and companies, but also in environments involving movement of people, cars, etc. as called mobile communications. In particular, the car is equipped with a high-performance navigation system. By using the latitude / longitude position information obtained from GPS and the ITS VICS service provided by the Road Bureau of the Ministry of Land, Infrastructure, Transport and Tourism, traffic congestion and Information on accidents is available in real time (see Non-Patent Documents 1 to 3).

“Car navigation”, [online], free encyclopedia “Wikipedia”, [searched on June 15, 2010], Internet <URL: http://en.wikipedia.org/wiki/%E3%82 % AB% E3% 83% BC% E3% 83% 8A% E3% 83% 93% E3% 82% B2% E3% 83% BC% E3% 82% B7% E3% 83% A7% E3% 83% B3 > “Global Positioning System”, [online], free encyclopedia “Wikipedia”, [searched on June 15, 2010], Internet <URL: http://en.wikipedia.org/wiki/% E3% 82% B0% E3% 83% AD% E3% 83% BC% E3% 83% 90% E3% 83% AB% E3% 83% BB% E3% 83% 9D% E3% 82% B8% E3% 82% B7% E3% 83% A7% E3% 83% 8B% E3% 83% B3% E3% 82% B0% E3% 83% BB% E3% 82% B7% E3% 82% B9% E3% 83% 86% E3% 83% A0> "Car Navigation & VICS Road Traffic Information Communication System", [online], Ministry of Land, Infrastructure, Transport and Tourism, [Search June 15, 2010], Internet <URL: http://www.mlit.go.jp/road/style/cont1 /index.html> "OpenCV-1.0 Reference Manual (Japanese translation)", [online], [Search June 15, 2010], Internet <URL: http://opencv.jp/opencv-1.0.0/document/>

  However, because the traffic volume observation section is rough and the places where traffic condition observation sensors are installed are limited to the main roads, the driver has useful information that can be used to reduce accidents at road intersections. Not fully provided. In addition, since the symbolized traffic jam information is mainly distributed, it is difficult to grasp the road conditions and traffic conditions.

  Currently, vehicles equipped with an in-vehicle camera are also on sale, but such an in-vehicle camera is only intended to provide visual assistance during parking. In addition, an omnidirectional vehicle-mounted camera only takes a landscape image to be added to a map provided on the web, and the landscape image is a still image and changes from moment to moment. It does not reflect the state of timely and appropriately.

  This invention is made | formed in view of the above, and makes it a subject to provide a driver | operator with the visual useful information in the middle of driving | running | working.

  The landscape image composition apparatus according to claim 1, an input unit that inputs a first landscape image captured by an in-vehicle camera of a first car, and a storage unit that stores the input first landscape image. Receiving means for receiving a second landscape image taken by an in-vehicle camera of a second car via a communication network; obtaining the first landscape image from the storage means; A composite unit that compares one region with one region of the second landscape image to identify a region having the highest similarity, and superimposes the two landscape images in the specific region, and a combined composite landscape image Display means for displaying.

  According to the present invention, the first landscape image captured by the in-vehicle camera of the first car is input, and the second landscape image captured by the in-vehicle camera of the second car is received via the communication network. Then, by comparing one region of the first landscape image and one region of the second landscape image, the region having the highest similarity is identified, and the two landscape images are overlapped and overlapped in the specific region. In order to display the synthesized composite image, the image information taken from different viewpoints is integrated, in other words, a landscape image that is not taken by the in-vehicle camera of the first car (taken only by the in-vehicle camera of the second car) Displayed scenery image), thereby providing the driver of the first car with visually useful information while traveling.

  The landscape image composition device according to claim 2 is the landscape image composition device according to claim 1, wherein a plurality of landmark images obtained by photographing the landmarks for each part and the positional information of each part are stored in association with each other. Storage means, a matching means for matching the combined landscape image and each landmark image together, and position information of a part corresponding to the landmark image having the highest matching degree is acquired from the storage means. Display means for displaying together with the composite landscape image; receiving means for receiving current position information of the first vehicle from a satellite positioning system; and correcting means for correcting the displayed position information of the part to the current position information. And further comprising.

  The landscape image composition device according to claim 3 is the landscape image composition device according to claim 1 or 2, wherein the composition means includes a plurality of time-series first landscape images taken during traveling, A plurality of the second landscape images photographed during traveling in the same time period are sequentially superimposed, and the display means continuously displays the plurality of superimposed composite landscape images in time series. .

  According to the present invention, a plurality of time-series first landscape images photographed during traveling and a plurality of second landscape images photographed during traveling in the same time zone are sequentially superimposed and superimposed respectively. Since a plurality of composite landscape images are continuously displayed in chronological order, the latest situation of unknown roads and environmental information that cannot be confirmed by the driver of the first car can be provided to the driver from time to time. Traffic accidents can be reduced.

  According to a fourth aspect of the present invention, there is provided the landscape image synthesizing apparatus according to any one of the first to third aspects, wherein the synthesizing unit is configured to generate the two images when the similarity is a predetermined threshold value or less. It is characterized by not superimposing the landscape images.

  According to the present invention, when the similarity is equal to or lower than the predetermined threshold value, the two landscape images are not overlapped, so that two different landscape images that have no overlapped portion are erroneously overlapped. It becomes possible to prevent this, and more appropriate useful information can be provided to the driver of the first car.

  According to a fifth aspect of the present invention, there is provided a landscape image synthesizing method in which a first step of inputting a first landscape image photographed by an in-vehicle camera of a first car by a computer and the input first landscape image. A second step of storing in the storage means; a third step of receiving a second landscape image taken by an in-vehicle camera of a second car via a communication network; and storing the first landscape image in the storage unit. A region having the highest similarity by comparing one region of the first landscape image and one region of the second landscape image, and acquiring the two landscape images in the specific region. It has the 4th step which superimposes, and the 5th step which displays the superimposed synthetic scenery image, It is characterized by the above-mentioned.

  According to the present invention, the first landscape image captured by the in-vehicle camera of the first car is input, and the second landscape image captured by the in-vehicle camera of the second car is received via the communication network. Then, by comparing one region of the first landscape image and one region of the second landscape image, the region having the highest similarity is identified, and the two landscape images are overlapped and overlapped in the specific region. In order to display the synthesized composite image, the image information taken from different viewpoints is integrated, in other words, a landscape image that is not taken by the in-vehicle camera of the first car (taken only by the in-vehicle camera of the second car) Displayed scenery image), thereby providing the driver of the first car with visually useful information while traveling.

  The landscape image composition method according to claim 6 is the landscape image composition method according to claim 5, wherein a plurality of landmark images obtained by photographing the landmarks for each part and the position information of each part are associated with each other and stored. The step of accumulating, the step of collating the combined landscape image with each of the landmark images, and the position information of the part corresponding to the landmark image having the highest degree of matching from the accumulating means Displaying with the composite landscape image; receiving current position information of the first vehicle from a satellite positioning system; correcting the displayed position information of the part to the current position information; It further has these.

  The landscape image composition method according to claim 7 is the landscape image composition method according to claim 5 or 6, wherein the fourth step includes a plurality of time-series first landscape images captured during traveling. And the plurality of second landscape images taken during traveling in the same time period are sequentially superimposed, and the fifth step continuously displays the plurality of superimposed composite landscape images in chronological order. It is characterized by.

  According to the present invention, a plurality of time-series first landscape images photographed during traveling and a plurality of second landscape images photographed during traveling in the same time zone are sequentially superimposed and superimposed respectively. Since a plurality of composite landscape images are continuously displayed in chronological order, the latest situation of unknown roads and environmental information that cannot be confirmed by the driver of the first car can be provided to the driver from time to time. Traffic accidents can be reduced.

  The landscape image composition method according to claim 8 is the landscape image composition method according to any one of claims 5 to 7, wherein the fourth step is performed when the similarity is equal to or less than a predetermined threshold. It is characterized by not superimposing two landscape images.

  According to the present invention, when the similarity is equal to or lower than the predetermined threshold value, the two landscape images are not overlapped, so that two different landscape images that have no overlapped portion are erroneously overlapped. It becomes possible to prevent this, and more appropriate useful information can be provided to the driver of the first car.

  According to a ninth aspect of the present invention, there is provided a landscape image composition program that causes a computer to execute each step in the landscape image composition method according to any one of the fifth to eighth aspects.

  According to the present invention, it is possible to provide the driver with visually useful information while traveling.

It is the figure which looked at the some vehicle which is drive | working the road intersection vicinity from upper direction. It is a figure which shows the functional block structure of a landscape image composition system. It is a figure explaining composition of a landscape image.

  Hereinafter, an embodiment for carrying out the present invention will be described with reference to the drawings. However, the present invention can be implemented in many different modes and should not be construed as being limited to the description of the present embodiment.

  FIG. 1 is a view of a plurality of vehicles traveling near a road intersection as viewed from above. Five roads intersect at this road intersection 500. The first car 10 is on the left road and the second car 20 is on the lower road as viewed from the center of FIG. The third cars 30 are traveling toward the road intersection 500 respectively.

  The first car 10 is equipped with four in-vehicle cameras 11a to 11d, each of which captures a landscape and landscape in four directions, front, rear, left and right while traveling, and each of a plurality of time-series landscape images captured at predetermined time intervals. Acquisition is possible for each in-vehicle camera or for each direction. The captured landscape images are continuously displayed in chronological order, and can be provided to the driver as environmental information.

  Similarly, each of the second car 20 and the third car 30 is equipped with four in-vehicle cameras, and all cars can communicate with each other via the wireless communication network 700. It has become. In addition, the vehicle-mounted camera mounted on the vehicle in this way may use a dedicated monitoring camera or web camera that captures the outside of the vehicle from the vehicle, or may use a vehicle-mounted camera equipped for parking purposes. Good.

  FIG. 2 is a diagram showing a functional block configuration of the landscape image composition system according to the present embodiment. The landscape image composition system 1 includes a landscape image composition device 100 and a landscape image transmission device 200. In operation, both devices are mounted on one vehicle. However, in order to facilitate understanding of the invention, here, the landscape image composition device 100 is mounted on the first vehicle 10 and the landscape image is displayed. It is assumed that the transmission device 200 is mounted on the second car 20.

  The landscape image transmitting apparatus 200 inputs a plurality of time-series landscape images (hereinafter referred to as second landscape images) respectively taken at predetermined time intervals by the in-vehicle cameras 21a to 21d while the second car 20 is traveling. An input unit 201, a storage unit 202 that stores a plurality of input second landscape images for each vehicle-mounted camera or for each direction in time series, and a plurality of time-series second landscape images from the storage unit 202. The transmission unit 203 is configured to acquire and transmit to the landscape image composition apparatus 100 via the wireless communication network 700.

  On the other hand, the landscape image composition device 100 includes an input unit 101, a storage unit 102, a first reception unit 103, a synthesis unit 104, a storage unit 105, a collation unit 106, a display unit 107, and a second unit. The receiving unit 108 and the correction unit 109 are configured.

  The input unit 101 is electrically connected to the vehicle-mounted cameras 11a to 11d, respectively, and a plurality of time-series landscape images captured by the vehicle-mounted cameras 11a to 11d at predetermined time intervals while the first car 10 is traveling. (Hereinafter, referred to as a first landscape image).

  The storage unit 102 has a function of storing a plurality of input first landscape images in time series for each in-vehicle camera or for each direction.

  The first receiving unit 103 has a function of receiving a plurality of time-series second landscape images transmitted from the landscape image transmitting apparatus 200 via the wireless communication network 700.

  The synthesizing unit 104 acquires a plurality of time-series first landscape images from the storage unit 102, and one region of the first landscape image and one of the second landscape images captured at the same time or the same time zone. A region having the highest similarity is identified by comparing the regions, and the two landscape images (the first landscape image and the second landscape image) are superimposed on the specific region. Note that the composition unit 104 sequentially performs this image superimposing process using a plurality of first landscape images in time series and a plurality of second landscape images taken during traveling in the same time period. Do. Thereby, a plurality of superimposed time-series composite landscape images are generated. Further, when the similarity is equal to or less than a predetermined threshold, the synthesis unit 104 determines that the image is a photograph of a landscape in a completely different direction (for example, an image photographed by the in-vehicle camera 21c and the in-vehicle camera 11c). In addition, it has a function of not superimposing these two landscape images.

  The accumulating unit 105 has a function of accumulating a plurality of landmark images obtained by photographing the landmarks for each part and the latitude / longitude information and landmark information of each part in association with each other. As a landmark image, for example, each image obtained by photographing the gate and corner of Tokyo Tower T, an image of a window of a famous building, and the like can be cited. In the storage unit 105, an image of the gate of the Tokyo Tower T and information indicating the latitude and longitude of the gate and the gate of the Tokyo Tower T are stored in association with each other.

  The matching unit 106 has a function of matching the combined landscape image with each landmark image and calculating the matching degree. Since a plurality of composite scenery images are generated in time series, the collation unit 106 sequentially performs this collation process using each composite scenery image.

  The display unit 107 has a function of acquiring latitude / longitude information of a part corresponding to the landmark image having the highest matching degree as a result of the matching in the matching unit 106 from the storage unit 105 and displaying it together with the composite landscape image. Yes. Since a plurality of composite scenery images are generated in time series, the display unit 107 continuously displays (updates display) the plurality of composite scenery images in time series.

  The second receiving unit 108 has a function of receiving current latitude / longitude information of the first vehicle 10 from a satellite positioning system such as GPS.

  The correcting unit 109 has a function of correcting the currently displayed position information to the current longitude / latitude information received from the satellite positioning system by the second receiving unit 108.

  Next, the processing operation of the landscape image composition system will be described. For ease of understanding, the first landscape image is an image of landscape A (see FIG. 1) taken by the in-vehicle camera 11a of the first car 10, and the second landscape image is the second car. It is assumed that the image is a landscape B photographed by 20 in-vehicle cameras 21a. Further, it is assumed that the first landscape image and the second landscape image are each one image taken at the same time.

  First, the second landscape image of the landscape B photographed by the in-vehicle camera 21a is input by the input unit 201 of the landscape image transmitting apparatus 200 and stored in the storage unit 202 (S1). Tokyo Tower T and the third car 30 are photographed in the second landscape image.

  Next, the transmission unit 203 transmits the second landscape image to the landscape image composition device 100 via the wireless communication network 700 (S2).

  Next, the first receiving unit 103 of the landscape image composition device 100 receives the second landscape image transmitted from the landscape image transmission device 200 via the wireless communication network 700 (S3).

  On the other hand, the first landscape image captured by the in-vehicle camera 11a is input by the input unit 101 of the landscape image composition device 100 and stored in the storage unit 102 (S4). Tokyo Tower T is photographed in the first landscape image. Note that S4 can be processed before S3.

  Next, the synthesizing unit 104 compares one area of the first landscape image and one area of the second landscape image to identify an area having the highest similarity, and the first landscape image and the first landscape image are identified in the specific area. The two landscape images are superimposed (S5).

  For example, a portion where the difference between the luminance value of each pixel or fixed region of the first landscape image and the luminance value of each pixel or fixed region of the second landscape image is specified is specified, and the specified portion is a margin part. As shown in FIG. 3, a composite landscape image is generated by superimposing two images. Specifically, an image area with Tokyo Tower T, which is taken in common with the two landscape images, becomes a marginal part, and a wide range of composite landscape images including Tokyo Tower T and the third car 30 are generated. Will be.

  Since there is a high possibility that there is a difference between the illumination level of the in-vehicle camera 11a and the illumination level of the in-vehicle camera 21a, the normalization process is performed on the two landscape images using the luminance distribution, and then the luminance value is changed. It is preferable to perform the difference calculation to detect the specific portion.

  Next, the composite landscape image superimposed in S5 is displayed on the screen by the display unit 107 (S6).

  Note that in S5, the combining unit 104, even in the region with the highest similarity, when the similarity is equal to or less than a certain threshold value (when the difference in luminance value between one region is equal to or greater than a certain threshold value). The two landscape images are not superimposed. As a result, it is possible to prevent two landscape images taken in completely different directions from being erroneously combined.

  As described above, since the third car 30 is captured in the composite landscape image, the presence of the third car 30 is indicated to the driver of the first car 10 driving toward the road intersection 500. It is possible to notify visually. In addition, by repeating S1 to S6 at predetermined time intervals and continuously displaying a plurality of composite scenery images in chronological order, the latest composite scenery images can be provided to the driver.

  The following processing can be further performed on the above S1 to S6.

  The matching unit 106 of the landscape image synthesizing apparatus 100 matches the combined landscape image generated in S5 with the plurality of landmark images stored in the storage unit 105, and the degree of matching of the combined landscape image with each landmark image. Is calculated (S7).

  Next, the latitude and longitude information and landmark information of the part corresponding to the landmark image having the highest matching degree as a result of matching in S7 are acquired from the storage unit 105 by the display unit 107 and displayed together with the composite landscape image ( S8).

  Next, the current latitude / longitude information of the first car 10 is received by the second receiving unit 108 from a satellite positioning system such as GPS (S9).

  Finally, the correction unit 109 corrects the position information currently displayed in S8 to the current longitude / latitude information received in S9 (S10).

  As described above, since the latitude / longitude information displayed together with the composite landscape image is corrected to the latitude / longitude information acquired from the GPS or the like, it is possible to provide the driver with an accurate position when the composite landscape image is generated. It becomes possible.

  As described above, it is possible to display a plurality of composite landscape images in a time-series order and display them as moving images. However, only a moving object such as a person or a bicycle can be specified and synthesized from the differential image processing. Is also possible. Thereby, the minimum necessary information can be provided to the driver. In addition, with respect to the difference image, in order to narrow down only to information having a position change, it is possible to make the displayed gradation level rough or use an edge image (see Non-Patent Document 4). Thus, it is possible to provide useful information to the driver while avoiding the privacy problem of the person, and to reduce traffic accidents.

  According to the present embodiment, the first landscape image captured by the in-vehicle camera 11a of the first car 10 is input, and the second landscape image captured by the in-vehicle camera 21a of the second car 20 is wirelessly transmitted. An area having the highest similarity is identified by comparing one area of the first landscape image and one area of the second landscape image received via the communication network 700, and the two areas are identified in the specific area. Since the combined landscape images are displayed by superimposing the landscape images, the image information captured from different viewpoints is integrated, in other words, a landscape image (first image not captured by the in-vehicle camera of the first car 10). Landscape image captured only by the in-vehicle camera of the second car 20) can be displayed, and thus visual useful information in the middle of traveling can be provided to the driver of the first car 10.

  According to the present embodiment, a plurality of time-series first landscape images photographed during traveling and a plurality of second landscape images photographed during traveling in the same time period are sequentially superimposed, Since the superimposed composite landscape images are continuously displayed in chronological order, the latest situation of unknown roads and environmental information that cannot be confirmed by the driver of the first car 10 can be provided to the driver from time to time, Traffic accidents can be reduced more reliably.

  According to the present embodiment, when the similarity is equal to or lower than a predetermined threshold, the two landscape images are not overlapped, so that two different landscape images that have no overlapping portions are erroneously overlapped. This makes it possible to prevent the occurrence of accidents and provide more appropriate useful information to the driver of the first car.

  Finally, the landscape image composition system 1 described in the present embodiment is applied to the industrial field related to the field of traffic engineering, multimedia field, encoding field, communication field, monitoring in the real environment, image sensing, video production, etc. Is possible.

  The landscape image composition system 1 described in the present embodiment is configured by a computer. That is, the storage unit 202, the storage unit 102, and the storage unit 108 are realized by a storage unit such as a memory. In addition, the input unit 201, the transmission unit 203, the input unit 101, the first reception unit 103, the synthesis unit 104, the collation unit 106, the display unit 107, the second reception unit 108, and the correction unit. 109 is realized by a calculation means such as a CPU and executed by a program.

  In addition, the landscape image composition system 1 described in the present embodiment is recorded as a program on a recording medium such as an optical storage device or a magnetic storage device so as to be readable, and this recording medium is incorporated into a computer or recorded on a recording medium. Can be downloaded to a computer via an arbitrary communication line, or installed from a recording medium, and the computer is operated by the program, thereby causing each processing operation described above to function as the landscape image composition system 1. Of course you can.

DESCRIPTION OF SYMBOLS 1 ... Scenery image composition system 10 ... 1st car 11a-11d ... Car-mounted camera,
20 ... 2nd car 21a-21d ... In-vehicle camera,
30 ... Third car 31a-31d ... In-vehicle camera,
DESCRIPTION OF SYMBOLS 100 ... Landscape image composition apparatus 101 ... Input part 102 ... Memory | storage part 103 ... 1st receiving part 104 ... Composition part 105 ... Accumulation part 106 ... Collation part 107 ... Display part 108 ... 2nd receiving part 109 ... Correction | amendment part 200 ... Landscape image transmission apparatus 201 ... input unit 202 ... storage unit 203 ... transmission unit 500 ... road intersection 700 ... wireless communication network A, B ... landscape S ... step T ... Tokyo Tower

Claims (9)

An input means for inputting a first landscape image taken by an in-vehicle camera of the first car;
Storage means for storing the input first landscape image;
Receiving means for receiving a second landscape image taken by an in-vehicle camera of a second car via a communication network;
The first landscape image is obtained from the storage means, and an area having the highest similarity is identified by comparing one area of the first landscape image and one area of the second landscape image, and the identification Combining means for superimposing the two landscape images in a region;
Display means for displaying the superimposed composite landscape image;
A landscape image composition apparatus characterized by comprising: Storage means for storing a plurality of landmark images obtained by photographing the landmarks for each part and position information of each part in association with each other;
Collating means for collating the composite landscape image and each landmark image together;
Display means for acquiring position information of a part corresponding to the landmark image having the highest matching degree from the storage means and displaying the position information together with the synthetic landscape image;
Receiving means for receiving current position information of the first vehicle from a satellite positioning system;
Correction means for correcting the position information of the displayed part to the current position information;
The landscape image composition apparatus according to claim 1, further comprising: The synthesis means includes
A plurality of time-sequential first landscape images captured during traveling and a plurality of second landscape images captured during traveling in the same time period are sequentially superimposed, respectively.
The display means includes
The landscape image composition apparatus according to claim 1 or 2, wherein a plurality of superimposed landscape images are continuously displayed in time series. The synthesis means includes
4. The landscape image composition apparatus according to claim 1, wherein the two landscape images are not superimposed when the similarity is equal to or less than a predetermined threshold. By computer
A first step of inputting a first landscape image taken by an in-vehicle camera of the first car;
A second step of storing the input first landscape image in a storage means;
A third step of receiving a second landscape image taken by the in-vehicle camera of the second car via a communication network;
The first landscape image is obtained from the storage means, and an area having the highest similarity is identified by comparing one area of the first landscape image and one area of the second landscape image, and the identification A fourth step of superimposing the two landscape images in a region;
A fifth step of displaying the superimposed composite landscape image;
A landscape image synthesizing method characterized by comprising: Storing a plurality of landmark images obtained by photographing the landmarks for each part and position information of each part in association with each other in the storage unit;
Matching the combined landscape image and each landmark image together,
Obtaining position information of a part corresponding to the landmark image having the highest matching degree from the storage means, and displaying the position information together with the synthetic scenery image;
Receiving current position information of the first vehicle from a satellite positioning system;
Correcting the position information of the displayed part to the current position information;
The landscape image synthesis method according to claim 5, further comprising: The fourth step includes
A plurality of time-sequential first landscape images captured during traveling and a plurality of second landscape images captured during traveling in the same time period are sequentially superimposed, respectively.
The fifth step includes
7. The landscape image composition method according to claim 5, wherein a plurality of superimposed composite landscape images are continuously displayed in chronological order. The fourth step includes
The landscape image synthesis method according to claim 5, wherein the two landscape images are not overlapped when the similarity is equal to or less than a predetermined threshold value.   9. A landscape image composition program that causes a computer to execute each step in the landscape image composition method according to claim 5.

Images