JP2019118103A - Information processing apparatus, information processing method, information processing program, and computer-readable recording medium recording information processing program - Google Patents

Abstract

To provide an information processing apparatus etc. which can update information included in a road image appropriately.SOLUTION: An information processing apparatus includes image acquisition means for acquiring a new road image including a thing related to a road, updating means for updating a missing portion in the road image of a predetermined thing included in the road image stored in storage means and the new road image on the basis of the new road image, image acquisition means for acquiring the new road image including the thing related to the road, and complementing means for complementing the missing portion in the new road image of the predetermined thing included in the road image stored in the storage means and the new road image on the basis of the road image.SELECTED DRAWING: Figure 7

Description

  The present application relates to an information processing apparatus or the like that updates a road image stored in a storage device.

  2. Description of the Related Art Conventionally, techniques for assisting a driver's driving using a captured image obtained by capturing an image of the traveling direction of a vehicle have become widespread. For example, there is a technology for assisting a driver in a parking operation by displaying on a display a captured image obtained by imaging the rear of a vehicle at the time of parking. In such parking operation support technology, the brightness of the captured image is low at the time of parking at night or in a dark place, and it is not possible to determine the parking lot line in the captured image, and there is a problem that the driver can not be sufficiently supported.

  On the other hand, Patent Document 1 detects the outline of a parking frame based on a captured image captured by a camera, and superimposes a figure based on the detected outline of the parking frame on the captured image. There is disclosed a technique for presenting a parking lot line to a driver in a pseudo manner by extending the frame line and assisting a parking operation. However, in the case where the parking frame has an incomplete shape such as being interrupted in the middle, the driver may not be able to recognize the correct frame line because the shape is not corrected. .

  Under such circumstances, in Patent Document 2, when the detected parking line is incomplete, a past image obtained by imaging a parking line having the same shape as the detected parking line is read out, and the image of the image currently being taken is parked There is disclosed a technique of aligning a parking division line of a past image with a division line and superimposing the image on a currently captured image. Thereby, the driver can be presented with a captured image in which the incomplete parking lot line is complemented by the past image.

JP 2003-63339 A Patent No. 4670463

  However, in the technique described in Patent Document 1, there is a problem that the parking lot line can not be complemented by the past image if the parking lot line is incomplete also in the past image.

  This application is made in view of the above-mentioned problem, and an example of the subject is providing an information processor etc. which can update suitably the information contained in a road picture.

  The invention according to claim 1 is an image acquisition means for acquiring a new road image including things relating to a road, and the road of the predetermined thing included in the road image stored in the storage means and the new road image. Updating means for updating a missing portion in the image based on the new road image.

  The invention according to claim 2 is an image acquisition means for acquiring a new road image including things relating to roads, the new one of the predetermined things included in the road image stored in the storage means and the new road image. And a complementing means for complementing a missing part in a road image based on the road image.

  The invention according to claim 3 is an information processing method used by an information processing apparatus, comprising: an image acquiring step of acquiring a new road image including things relating to a road; a road image stored in storage means; Updating the defective portion of the predetermined thing in the road image, which is included in the road image, based on the new road image.

  The invention according to claim 4 is an information processing method used by an information processing apparatus, comprising: an image acquiring step of acquiring a new road image including things related to a road; a road image stored in storage means; And D. a complementing step of complementing a missing part in the new road image of the predetermined thing included in the road image based on the road image.

  The invention according to claim 5 is characterized in that the computer is an image acquisition means for acquiring a new road image including things relating to a road, and the predetermined thing included in the road image stored in the storage means and the new road image. Function as updating means for updating the missing part in the road image of the above based on the new road image.

The invention described in claim 6 is a computer readable recording medium storing the information processing program according to claim 5.

It is a block diagram of information processor 1 in this embodiment. It is a figure which shows an example of the past road image in this embodiment. It is a figure which shows an example of the new road image in this embodiment. It is a figure which shows an example of the past road image updated by the new road image in this embodiment. It is a figure which shows an example of the new road image complemented by the past road image in this embodiment. It is a block diagram of navigation device NV in this embodiment. It is a flowchart which shows an example of the image update process by the control part 211 of the navigation apparatus NV of this embodiment. It is a figure which shows an example of the white line of the new road image in this embodiment. It is a figure which shows an example of the white line of the past road image in this embodiment. It is a figure which shows an example of the difference white line computed from the new road image and the past road image in this embodiment.

  A mode for carrying out the present invention will be described with reference to FIGS.

  As shown in FIG. 1, the information processing apparatus 1 includes an image acquisition unit 111A, a specification unit 111B, and an update unit 111C.

  In this configuration, the image acquisition unit 111A acquires a new road image including an object related to the road in the imaging range.

  The identifying unit 111B identifies, among the road images stored in the storage unit provided inside or outside the information processing apparatus 1, a road image obtained by imaging the same spot as the new road image acquired by the image acquiring unit 111A. .

  The update unit 111C updates a part of the information included in the road image identified by the identification unit 111B with the new road image acquired by the image acquisition unit 111A.

  As described above, according to the operation of the information processing apparatus 1 according to the embodiment, a part of the information on the road image stored in the storage unit is updated by the new road image, so the information included in the road image can be It can be the latest.

  For example, in the road image 500 shown in FIG. 2, when a part (an example of “part of the information of the road image”) on the white line 501L among the three white lines 501L, 501C, and 501R is covered with a puddle 502 In the above, a part of the white line 501L is updated by the new road image 600 obtained by photographing the same point as the road image 500 shown in FIG. Thereby, as shown in FIG. 4, the road image 500 is complemented by the white line portion 503 where the puddle 502 of the white line 501L was present. That is, the information of the road image 500 can be enhanced by the new road image 600. Further, as described later, since the road image is used when complementing a new road image, the accuracy at the time of complementation can be improved by enriching the information.

  On the other hand, when displaying the new road image 600 shown in FIG. 3 on the display, when a part of the three white lines 601L, 601C, and 601R is concealed by the thick fog 602, the white lines 601L, 601C, and 601R are displayed. A part of the image is complemented by the road image 500. Thereby, it can be set as the new road image 600 shown in FIG. As a result, the user of the information processing apparatus 1 can visually recognize the white line portions 603L, 603C, and 603R which are actually hidden by the thick fog 602 and can not be visually recognized. In addition, in the new road image 600 shown in FIG. 5, the user is able to distinguish which part is complemented by displaying the complemented white line parts 603L, 603C and 603R in a distinguishable manner with respect to the other parts. While being able to distinguish, it becomes easy to visually recognize the white line part.

  Next, specific examples corresponding to the above-described embodiment will be described.

  An embodiment will be described with reference to FIGS. The embodiment described below is an embodiment in which the present invention is applied to a navigation apparatus (an example of an "information processing apparatus").

[1. Configuration of Navigation Device NV]
First, the configuration of the navigation device NV according to the present embodiment will be described.

  As shown in FIG. 6, the navigation device NV includes a control unit 211, a storage device 212 such as an HDD, an input device 213 such as a keyboard or remote controller, and a touch panel, a display unit 214, and a bus line 215. An input / output interface unit 220, a vehicle speed sensor 221, an angular velocity sensor 222, an acceleration sensor 223, a steering angle sensor 224, a GPS (Global Positioning System) receiving unit 225 to which an antenna 226 is connected, a data transmitting / receiving unit 227 , And a camera 228.

  The vehicle speed sensor 221 detects the current speed of the vehicle using, for example, a speed detection process or the like using a vehicle speed pulse or the like acquired from a car equipped with the navigation device NV, and outputs speed data. The angular velocity sensor 222 detects, for example, the angular velocity of a change in direction of the vehicle, and outputs angular velocity data per unit time and relative orientation data. The acceleration sensor 223 detects, for example, the acceleration in the front-rear direction of the vehicle, and outputs acceleration data per unit time and the like. The steering angle sensor 224 detects the steering angle of the vehicle, and outputs steering angle data and the like. The GPS receiving unit 225 receives navigation radio waves from GPS satellites, and outputs latitude, longitude, altitude data, which is vehicle position information, absolute direction data of the traveling direction of the vehicle, GPS speed data, etc., as GPS positioning data. The data transmission / reception unit 227 performs processing related to transmission / reception of data with the server device via the network.

  The camera 228 is installed toward the outside of a car on which the navigation device NV is mounted, and captures a road image including an object related to the road on which the car travels in a shooting range. In the present embodiment, it is assumed that one camera 228 is installed in the car and photographs a landscape in front of the vehicle (traveling direction).

  The storage device 212 (an example of the “storage means”) stores map image data for displaying a map on the display unit 214, map information used when searching for a route, road link information, and the like. In addition, the storage device 212 stores a road image captured by the camera 228 or another camera as a past road image. The storage device 212 stores the past road image in association with the shooting position, the road ID (identification information allocated to all the roads), and the shooting condition information indicating the traveling direction. Furthermore, the storage device 212 stores various programs such as an operating system and application programs. The various programs and the past road image may be acquired from, for example, a server device or the like via a network, or read one recorded on a recording medium such as a CD, a DVD, or a USB memory. Also good.

  The input device 213 receives an input operation of the user, and transmits an operation signal indicating the content of the operation to the control unit 211.

  The display unit 214 displays various display data under the control of the control unit 211. The display unit 214 is configured to include a graphics controller 214a, a buffer memory 214b including a memory such as a VRAM (Video RAM), and a display 214c including a liquid crystal display or the like. In this configuration, the graphics controller 214 a controls the entire display unit 214 based on control data sent from the control unit 211 via the bus line 215. Also, the buffer memory 214 b temporarily stores image information that can be displayed immediately. Then, an image is displayed on the display 214 c based on the image data output from the graphics controller 214 a.

  The control unit 211 includes a CPU 211a that controls the entire control unit 211, a ROM 211b in which a control program for controlling the control unit 211 and the like is stored in advance, and a RAM 211c that temporarily stores various data. The control unit 211 is connected to the vehicle speed sensor 221, the angular velocity sensor 222, the acceleration sensor 223, the steering angle sensor 224 and the GPS receiving unit 225 via the bus line 215 and the input / output interface unit 220, and is output from each Control of the entire navigation device NV is performed based on velocity data, angular velocity data and relative heading data, steering angle data, GPS positioning data, absolute heading data of traveling direction of vehicle, acceleration data, etc. Control each operation in the component.

  The control unit 211 acquires the new road image captured by the camera 228 in real time, and among the past road images stored in the storage device 212, the past road image obtained by capturing the same spot as the new road image acquired from the camera 228 It identifies and updates a part (in this embodiment, a white line) of the information included in the identified past road image with the new road image acquired from the camera 228. Specifically, the control unit 211 collates the past road image with the new road image, and updates it when a white line not present in the past road image is detected. At this time, the control unit 211 associates and stores the position at the time of shooting, the road ID, the information indicating the traveling direction, and the like as the update record.

  At the same time, the control unit 211 complements a part of the information (white line in the present embodiment) included in the new road image acquired from the camera 228 with the past road image identified above, and causes the display unit 214 to perform real time. Display with. At this time, the control unit 211 displays the complemented part in a distinguishable manner with respect to the other parts. As a result, the display 214 c of the display unit 214 displays a real-time new road image in which a part of the information is complemented. The control unit 211 performs the update of the past road image or the complement of the new road image only when it is necessary to perform the update or the complement, respectively. That is, as described above, when a part of the white line is concealed for some reason, the concealed part is updated or complemented. In addition, the control unit 211 acquires a new road image as a moving image from the camera 228 (for example, 30 images per second) and causes the display unit 214 to display the road image. It will be displayed as

  In this configuration, the CPU 211 a of the control unit 211 is an example of “image acquisition unit”, “specific unit”, “update unit”, “complement unit”, or “output unit”.

[2. Image update processing by control unit 211 of navigation device NV]
Next, image update processing executed centering on the control unit 211 will be described using FIGS. 7 to 10. In addition, the white line part contained in a road image is made into update object or complementation object here.

  The control unit 211 acquires a new road image captured by the camera 228 (step S11). When acquiring a new road image, the control unit 211 acquires the shooting position (the vehicle position information output by the GPS receiving unit 225), the road ID during travel, and the traveling direction on the road (the GPS receiving unit 225 outputs Photographing condition information indicating absolute orientation data) is acquired.

  Next, the control unit 211 detects a white line portion in the new road image (step S12). As a detection method, a general known image processing is applied. For example, the white part is extracted with reference to the luminance value in the image, and the linear area in the white part is defined as the white line part.

Next, the control unit 211 specifies a past road image corresponding to the new road image from the past road images stored in the storage device 212 (step S13). Specifically, the control unit 211 searches the shooting condition information indicating the position at the time of shooting acquired in step S11, the road ID, and the traveling direction as a search key. However, depending on the accuracy of GPS, it may not be possible to identify an appropriate past road image. Therefore, the control unit 211 performs processing according to the following procedure as a method of specifying a desired past road image.
(1) The imaging condition information is used as a search key to narrow down a plurality of candidate past road images.
(2) Calculate the degree of similarity (not necessarily a numerical value) calculated from the features of the image of each past road image narrowed down in the above (1) and the new road image acquired in step S11; Identify high past road images.

  Here, the process (2) will be described in detail. First, the control unit 211 extracts image feature amounts for the past road image and the new road image narrowed down in the process of (1), and searches for corresponding points of the feature amounts between both images. Here, as the image feature quantity, SIFT (Scale-Invariant Feature Transform) or the like can be used. The small number of corresponding points means that there are few features that are similar between both images, and the appearance of the images is also significantly different. Therefore, the control unit 211 performs the search for the corresponding points on all past road images narrowed down in the process of (1), and the number of corresponding points is smaller than the threshold value T set empirically Exclude things from candidates for identification. Next, the control unit 211 identifies a final one past road image from among the remaining past road images, in consideration of the positional relationship of the corresponding points of the image feature amounts extracted earlier. Specifically, the control unit 211 integrates the difference in pixel position on the new road image and the past road image for all corresponding points. The integrated value becomes smaller as the appearance of the image is similar between the two images, and the integrated value becomes 0 when the appearance of the image is exactly the same. Therefore, the control unit 211 calculates the integrated value for each past road image that is a candidate for the identification target, and specifies the one that is the minimum value as the past road image. However, it is preferable that the control unit 211 also performs threshold processing on the integrated value and excludes a past road image whose appearance of an image is significantly different.

  Next, the control unit 211 compares the white line portion detected in the process at step S12 with the white line portion of each past road image stored in the storage device 212 (step S14), and determines whether there is a difference portion. It determines (step S15).

  Here, the process of step S15 will be described in detail with reference to FIGS. The control unit 211 collates the white line portions of the new road image acquired in the process of step S11 and the past road image identified in the process of step S13, and determines the presence or absence of the difference. Below, an example of the collation method is explained.

  Here, the case where the new road image 700 shown in FIG. 8 (A) and the past road image shown in FIG. 9 (A) are collated will be described. In the new road image 700, one white line 701 is detected, whereas in the past road image 800 shown in FIG. 9A, two white lines 801 and 802 are detected. As shown in FIGS. 8B and 9B, information is calculated from the angle on the image of the white line and the coordinates of the start point and the end point of the white line as the information of the white line, respectively.

  In the above example, based on the angle of the white line 701 of the new road image 700, the control unit 211 searches for a white line having a similar angle in the past road image 800. In this example, since the angle of the white line 701 of the new road image 700 is 35 ° and the angle of the white lines 801 and 802 in the past road image 800 is the same angle as the white line 701, the white line 801 and the white line 802 are searched. In this search, an error of ± ε may be allowed in the angle.

  Next, with respect to the white line 701 of the new road image 700 and the white line 801 and the white line 802 searched in the past road image 800, the control unit 211 calculates the difference of the white line area based on the viewpoint and the end point. In this example, the white line 701 of the new road image 700 is from (0, 480) to (500, 200), and the white line 801 and the white line 802 of the past road image 800 are lines divided by removing only a part of the white line 701 It has become. The control unit 211 obtains the difference white line 901 in the difference image 900 illustrated in FIG. 10 by calculating a range that is a difference from the start point and the end point.

  The control part 211 determines with "YES" by the process of step S15, when the difference white line 901 is obtained. In this example, there is an undetected portion of the white line in the past road image 800, and the case of obtaining the differential white line for supplementing that portion has been described. Conversely, there is an undetected portion of the white line in the new road image. Similarly, in the case of obtaining the differential white line also in the case, the control unit 211 determines “YES” in the process of step S <b> 15 also when the differential white line is obtained. That is, the control unit 211 compares the new road image with the past road image, and determines that there is a difference when there is an undetected portion in either one and a difference white line is obtained. When a differential white line is obtained, information on the differential white line is saved and used in the process of step S16 or the process of step S17 described later.

  When it is determined that there is no difference in the process of step S15 (step S15: NO), the control unit 211 causes the display unit 214 to display the new road image acquired in step S11 as it is (step S18). End the update process. On the other hand, when it is determined that there is a different part (step S15: YES), the control unit 211 then updates the past road image with the new road image (step S16). Specifically, when the different part determined in the process of step S15 is a white line part that can be detected in the new road image and can not be detected in the past road image, the control unit 211 can not detect the white line in the past road image. The part is updated by the white line part (the difference white line saved by the process of step S14) located at the same position in the new road image.

  On the other hand, when the different portion determined in the process of step S15 is a white line portion which can be detected in the past road image and which can not be detected in the new road image, the control unit 211 It complements by the white line part (difference white line retracted | saved by the process of step S14) in the same position in a road image (step S17). At this time, the control unit 211 complements the white line portion with, for example, green in order to make the complemented portion distinguishable with respect to other portions. After completing the process of step S17, the control unit 211 causes the display unit 214 to display the new road image subjected to the complementing process in the process of step S17 (step S18), and ends the image updating process.

  As described above, according to the operation of the navigation device NV according to the present embodiment, the control unit 211 acquires a new road image including an object related to a road in the imaging range from the camera 228 (step S11). Among the past road images stored in the image, a past road image obtained by imaging the same spot as the new road image acquired from the camera 228 is specified (step S13), and the white line portion included in the specified past road image is It updates with the new road image acquired from (step S16).

  Therefore, according to the navigation device NV, the white line portion of the past road image stored in the storage device 212 can be supplemented with the latest new road image.

  In addition, the control unit 211 complements the white line portion included in the new road image acquired from the camera 228 with the past road image identified in the process of step S13 (step S17). Thereby, the white line portion of the new road image acquired from the camera 228 can be displayed after being complemented by the past road image. The fact that the white line portion is not displayed in the new road image acquired from the camera 228 is highly likely that the user of the navigation device NV can not directly view the white line portion on the road. Therefore, by displaying a new road image in which the white line portion is complemented, the user can estimate the position of the white line and drive, thereby improving the safety.

[3. Modified example]
Next, a modification of the above embodiment will be described. In addition, the modification demonstrated below can be combined suitably.

[3.1. Modification 1]
In the embodiment described above, the past road image is stored in the storage device 212, but instead, the past road image is stored in the storage device of the server device outside the navigation device NV, and the network is The image updating process shown in FIG. 7 may be performed while accessing the storage device via the interface.

[3.2. Modification 2]
Although the case where the object of updating or complementation is a white line on a road was explained in the example mentioned above, not only a white line but all things contained in a road image become the object. For example, speed indication on the road, pedestrian crossing, stop position indication, signboards along the road, signs, traffic lights, buildings, etc. are targeted. Further, updating or complementing may be performed on a plurality of types of objects.

[3.3. Modification 3]
Although the above-mentioned embodiment explained the case where display was a display 214c, it may replace with this and may adopt other display. For example, a head-up display (HUD) may be used. According to the head-up display, since information is directly displayed in a human visual field, a new road image can be viewed as needed without moving the visual point on the display 214 c of the navigation device NV.

DESCRIPTION OF SYMBOLS 1 information processing apparatus 111A image acquisition means 111B identification means 111C update means NV navigation apparatus 211 control part 211a CPU
211b ROM
211c RAM
212 storage device 213 input device 214 display unit 214a graphics controller 214b buffer memory 214c display 215 bus line 220 input / output interface unit 221 vehicle speed sensor 222 angular velocity sensor 223 acceleration sensor 224 steering angle sensor 225 GPS receiver 226 antenna 227 data transceiver 228 camera

Claims (1)

An image acquisition means for acquiring a new road image including things related to the road;
Updating means for updating a missing portion in the road image of the predetermined thing included in the road image stored in the storage means and the new road image based on the new road image;
An information processing apparatus comprising: