JP2013073422A - Portable terminal device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a portable terminal device that can accurately recognizes external situations without depending on user's operation or traveling in a specific environment.SOLUTION: A portable terminal device 1 to be used for a driving support system includes: storage means 4 for storing feature part information for extracting a plurality of predetermined feature parts in a compartment on the basis of an image of the compartment from a rear camera 3, location information on each of the feature parts and predetermined reference location information relating to an installation location and installation angle of the portable terminal device 1 for each model; deriving means 5 for deriving a location of each of the feature parts on the basis of the stored feature part information and image information of the rear camera 3; comparison means 6 for comparing the derived location of each of the feature parts with the location information of each of the feature parts; calculation means 7 for calculating an actual installation location and an actual installation angle of the portable terminal device 1 from the comparison result; and correction means 8 for correcting an external recognition result by a front camera 2. The correction means 8 corrects the external recognition result from the front camera 2 in accordance with the deviation value of the actual installation location and the actual installation angle with respect to the reference location information.

Description

  The present invention relates to a portable terminal device used in a driving support system.

  In recent years, driving performance systems for vehicle drivers using this type of mobile terminal device have been proposed along with the improvement in performance and enhancement of applications of mobile terminal devices typified by smartphones (Patent Document 1). This type of system recognizes the position of the vehicle's traveling road, the distance from the vehicle ahead, the obstacles, etc. from the image acquired by the camera installed in the portable terminal device, and installs these recognition information and the vehicle. Based on the driver's driving operation information detected by various sensors, the driver's attention level is reduced and the possibility of departure from the lane is obtained, and an alarm is notified to the driver according to the situation at that time. .

JP-A-2006-119965 (see paragraphs 0013-0030, FIG. 1, etc.)

  By the way, in order to accurately recognize the position of the traveling road of the own vehicle and the distance between the vehicle and the vehicle ahead with the camera of the mobile terminal device, it is necessary to accurately know the installation position and the installation angle of the mobile terminal device. Therefore, in the prior art, the mobile terminal device is installed at a position where the mobile terminal device is installed in a predetermined place of the vehicle, and the mobile terminal device is fixed there, or the operation acquired by the mobile terminal device Before using the application related to the support system, the user inputs the installation height of the mobile terminal device and travels on a road with a white line, thereby estimating the installation position and installation angle of the mobile terminal device.

  However, many commercially available fixtures can be easily changed in direction and height as functions, and the fixing position and angle are easily changed by detachment and operation of the mobile terminal device. In addition, when the user inputs the installation height of the mobile terminal, an error in measuring the height from the ground or an input error is likely to occur, and as a result, the position recognition accuracy of the host vehicle decreases. Furthermore, since it is necessary to travel on a road with a white line for a certain time, the system operation cannot be started quickly.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a portable terminal device that can accurately recognize the external environment without running through a user operation or a specific environment.

  In order to achieve the above-described object, the mobile terminal device of the present invention has a front camera that recognizes the outside world situation of the vehicle and a rear camera that recognizes the inside situation of the vehicle interior, and the outside world recognized by the front camera. In a portable terminal device used in a driving support system that performs driving support based on image information, a plurality of predetermined feature portions in a vehicle interior that are determined in advance based on an image in the vehicle interior recognized by the rear camera are extracted. Characteristic part information, position information of each characteristic part in the vehicle interior, and information on an installation position and an installation angle of the mobile terminal device in the vehicle interior, and predetermined reference position information are stored for each vehicle type. Based on the storage means, the feature information stored by the storage means, and the vehicle interior image information recognized by the rear camera, the rear camera The derivation means for deriving the position of each feature in the recognized image, the position of each feature derived by the derivation means and the position information of each feature stored by the storage means are collated Correction means for calculating, an actual installation position and an actual installation angle of the portable terminal device in the vehicle interior based on a verification result of the verification means, and a correction for correcting an external environment recognition result recognized by the front camera The correction means corrects an external environment recognition result recognized by the front camera according to a deviation amount of the actual installation position and the actual installation angle with respect to the reference position information. 1).

  According to the first aspect of the present invention, feature part information for extracting a plurality of predetermined feature parts in the vehicle interior determined in advance based on the vehicle interior image recognized by the rear camera and the rear camera are recognized. Based on the image information in the passenger compartment, the position of each feature in the image recognized by the rear camera is derived by the deriving means.

  Then, the position of each feature in the rear camera image derived by the deriving unit and the position information of each feature in the vehicle interior stored in advance by the storage unit are collated by the collating unit, and collated by the collating unit. The actual installation position and actual installation angle of the mobile terminal device in the vehicle interior are calculated by the calculation means based on the position of each feature in the image in the vehicle interior and the corresponding position information.

  Therefore, as in the prior art, the mobile terminal device is fixed to a fixture so that the installation position and installation angle of the mobile terminal device are known in advance, or the installation height of the mobile terminal device is input at the initial setting of the application The actual actual installation position and actual installation angle of the mobile terminal device can be calculated without traveling on a road with a white line.

  In addition, in the correction means, the reference position information of the mobile terminal device stored in the storage means is reference position information that does not need to be corrected, and the actual position of the mobile terminal device calculated by the calculation means for the reference position information. In order to automatically correct the recognition result of the outside world recognized by the front camera according to the amount of deviation of the installation position and the actual installation angle, it is possible to accurately detect the road position where the host vehicle is traveling, the inter-vehicle distance from the preceding vehicle, etc. Can do.

It is a block diagram of the portable terminal device of one Embodiment of this invention. It is a figure which shows an example of the arrangement | positioning state in the vehicle interior of a portable terminal device. It is a figure which shows an example of the image image | photographed with the back camera. 3 is a flowchart for explaining operations of the mobile terminal device of FIG. 1.

  An embodiment of the present invention will be described with reference to FIGS. 1 is a block diagram of a portable terminal device according to an embodiment of the present invention, FIG. 2 is a diagram showing an example of setting a characteristic part in a vehicle interior, and FIG. 3 is an example of an image taken by the rear camera of FIG. FIG. 4 is a flowchart for explaining the operation of FIG.

(Constitution)
A configuration of a mobile terminal device 1 according to an embodiment of the present invention will be described with reference to FIG.

  As the mobile terminal device 1, for example, a smartphone has two cameras, a front camera and a rear camera, and has a function of connecting to various sites via the Internet and acquiring / saving various applications. It is desirable to use these functions in a driving support system for a vehicle driver.

  The front camera 2 is used for photographing the outside world ahead of the host vehicle and recognizing outside world situations such as other vehicles, obstacles, and traveling roads. In addition, the rear camera 3 is used for photographing the interior of the host vehicle and extracting a predetermined characteristic portion in a predetermined interior. In addition, the mobile terminal device 1 is installed such that all of a plurality of predetermined characteristic portions that are determined in advance are within the field of view of the rear camera 3.

  The storage unit 4 stores in advance feature portion information that serves as a base for a later-described derivation unit to derive a plurality of predetermined positions in the vehicle interior from a vehicle interior image recognized by the rear camera 3. Remember. For example, when the feature portion is derived by the pattern matching process, the storage unit 4 stores template image information that is the feature portion information in advance.

  Further, the storage means 4 stores position information in a vehicle interior of a predetermined characteristic portion in the vehicle interior that is determined in advance. At this time, the storage unit 4 sets an arbitrary point in the passenger compartment as the origin, and stores the position of each feature as three-dimensional coordinate data in advance with the origin as a reference.

  In addition, the storage unit 4 stores in advance position information (reference position information) that is information regarding the installation position and the installation angle of the mobile terminal device 1 in the vehicle interior and that is not required to be corrected. At this time, the storage unit 4 stores in advance information about the three-dimensional coordinate data and the installation angle of the reference installation position of the mobile terminal device 1 with the above-described origin as a reference.

  Note that the storage unit 4 stores the above-described feature information, feature position information, and reference position information for each vehicle type. These pieces of information do not need to be stored in the storage unit 4 in advance. For example, the information may be downloaded from an external server using the communication function of the mobile terminal device 1.

  The deriving unit 5 includes feature part information for extracting a plurality of predetermined feature parts in the vehicle interior stored in advance by the storage unit 4 based on the image in the vehicle interior recognized by the rear camera 3, and the rear camera 3. The position of each feature in the image recognized by the rear camera 3 is derived on the basis of the image information in the vehicle compartment recognized by.

  For example, as shown in FIG. 2, these characteristic portions include an upper side central portion 10 a of the left headrest, an upper side central portion 10 b of the right headrest, and an upper side central portion 10 c of the back door glass as viewed from the front of the vehicle. It is preferable to select at least three features having different distances and directions from the rear camera 3.

  In this embodiment, the deriving means 5 derives a predetermined feature using a pattern matching method based on image information in the vehicle compartment recognized by the rear camera 3. For example, when deriving the positions of the predetermined features 10a, 10b, and 10c described above, the deriving means 5 includes the features of the upper side central portion 10a of the left headrest, the features of the upper side central portion 10b of the right headrest, and the upper side of the back door glass. Template image information representing the features of the central portion 10c is acquired from the storage unit 4, and the template image is moved on the image f in the vehicle compartment recognized by the rear camera 3 shown in FIG. Search and derive a predetermined feature.

  The collating means 6 collates the position information of each feature 10a, 10b, 10c on the image f of the rear camera 3 derived by the deriving means 5 with the position information of each feature stored in the storage means 4.

  Predetermined features derived by the deriving unit 5 based on the image f recognized by the rear camera 3 are two-dimensionally represented on the rear camera image f. However, since the positions of these feature parts have three-dimensional coordinates, the three feature parts 10a, 10b, and 10c represented in two dimensions are changed by the change in the installation position and the installation angle of the mobile terminal device 1. The positional relationship changes. That is, the distance between the feature 10a and the feature 10b in the two-dimensional image f of the rear camera 3 or the features 10a, 10b, and 10c with lines due to changes in the installation position and installation angle of the mobile terminal device 1. The angles of the triangles formed when they are connected change. Further, if the installation position of the mobile terminal device 1 changes, the positions of these characteristic portions change on the image f of the rear camera 3.

  Therefore, the collation unit 6 projects the three-dimensional coordinate data of each of the characteristic portions 10a, 10b, and 10c stored in the storage unit 4 on the image f of the rear camera 3 and outputs the rear camera derived by the deriving unit 5. A viewpoint that has the same positional relationship as that of each of the characteristic portions 10a, 10b, and 10c on the image f is obtained.

  The calculation means 7 calculates the actual actual installation position and actual installation angle of the mobile terminal device 1 in the vehicle interior based on the verification result of the verification means 6. At this time, the calculation unit 7 calculates the position and angle of the viewpoint obtained by the collation unit 6 and sets the position and angle as the actual installation position and the actual installation angle of the mobile terminal device 1.

  The correction unit 8 corrects the external environment recognition result recognized by the front camera 2 based on the actual installation position and the actual installation angle of the mobile terminal device 1 calculated by the calculation unit 7. At this time, the correction means 8 is information related to the installation position and the installation angle of the portable terminal device 1 in the vehicle interior, and is calculated by the calculation means 7 for the reference position information, which is position information that is originally a reference that does not need to be corrected. The external environment recognition result recognized by the front camera 2 is corrected according to the shift amount of the actual installation position and the actual installation angle of the mobile terminal device 1.

  For example, when the front vehicle is reflected in the corner on the image of the front camera 2 due to the installation angle of the mobile terminal device 1 being obliquely deviated from the traveling direction of the host vehicle, the front vehicle originally exists in the driving support system. Despite this, the vehicle may be mistaken for a vehicle traveling in the adjacent lane. In such a case, the correction means 8 recognizes the vehicle as a forward vehicle traveling in the same lane by correcting the actual installation position and the actual installation angle of the mobile terminal device 1 according to the amount of deviation from the reference position information. To do. In the driving support system, the inter-vehicle distance between the vehicle and the host vehicle is recognized as the inter-vehicle distance from the preceding vehicle traveling in the same lane.

  Based on the information about the distance between the vehicle ahead and the vehicle traveling in the same lane corrected by the correction means 8 and the position of the traveling road of the vehicle, there is a risk of warning or lane departure when the vehicle distance is too short. An alarm is displayed on the liquid crystal screen of the display means 9 for notification. Further, in order to acquire various information stored for each vehicle type by the storage unit 4, the vehicle type is selected using the touch panel of the display unit 9 or the like.

  Further, since the distance between the host vehicle and the preceding vehicle, the position of the traveling road, the presence of obstacles, and the like are known from the image recognized by the front camera 2 of the mobile terminal device 1, the information and the information are installed in the vehicle. By obtaining driving operation information detected by various sensors, for example, in a car navigation system, a warning is given when a lane departs or the distance between vehicles is inappropriate, and driving assistance to the driver is performed.

  Next, operation | movement of the portable terminal device 1 concerning this embodiment is demonstrated with reference to the flowchart of FIG.

  First, the portable terminal device 1 according to the present invention is installed in an arbitrary place in the vehicle interior (step S1), and the vehicle type of the host vehicle is selected on the touch panel of the liquid crystal screen of the display means 9 (step S2).

  Next, the deriving means 5 acquires characteristic part information for extracting a predetermined plurality of predetermined characteristic parts stored in the storage means 4 (step S3), and the deriving means 5 And the position of each feature part in the image f recognized by the rear camera 3 are derived based on the vehicle interior image information recognized by the rear camera 3 (step S4).

  Next, the collation means 6 acquires the position information of each feature stored in the storage means 4 and collates the position information of each feature derived by the derivation means 5 with the position information. When the three-dimensional coordinate data of each feature stored in 4 is projected onto the image f of the rear camera 3, the same positional relationship as that of each feature on the image f of the rear camera 3 derived by the deriving means 5 is obtained. Such a viewpoint is obtained (step S6).

  Next, the calculation means 7 calculates the actual actual installation position and actual installation angle of the mobile terminal device 1 in the vehicle interior based on the collation result of the collation means 6 (step S7).

  The correction unit 8 is information regarding the installation position and the installation angle of the portable terminal device 1 in the vehicle interior, and is calculated by the calculation unit 7 for the reference position information that is reference information that does not need to be corrected. The external environment recognition result recognized by the front camera 2 is corrected according to the deviation of the actual installation position and the actual installation angle of the terminal device 1 (step S8).

  From the external recognition result corrected by the correction means 8, the distance between the host vehicle and the vehicle ahead traveling on the same lane road as the host vehicle and the position of the host vehicle traveling road are known. If there is a possibility of lane departure, the driver is assisted in driving by notifying an alarm on the liquid crystal screen of the display means 9 (step S9), and the operation is terminated.

  Therefore, according to the above-described embodiment, the feature portion information for extracting a plurality of predetermined feature portions 10a, 10b, and 10c in the vehicle interior determined in advance based on the vehicle interior image recognized by the rear camera 3, and Based on the image information in the passenger compartment recognized by the rear camera 3, the positions of the characteristic portions 10 a, 10 b, 10 c in the image recognized by the rear camera 3 are derived by the deriving means 5.

  And the position of each characteristic part 10a, 10b, 10c in the image f of the rear camera 3 derived | led-out by the deriving means 5, and the positional information in the vehicle interior of each characteristic part 10a, 10b, 10c previously memorize | stored by the memory | storage means 4 Are collated by the collating means 6, and the rear camera 3 derived by the deriving means 5 when the three-dimensional coordinate data of each feature stored in the storage means 4 is projected onto the image f of the rear camera 3. A viewpoint that has the same positional relationship as each feature on the image f is obtained.

  Further, the actual installation position and actual installation of the mobile terminal device 1 in the vehicle interior based on the positions of the feature portions 10a, 10b, 10c in the vehicle interior image f collated by the collating means 6 and the corresponding position information. The angle is calculated by the calculation means 7. Therefore, as in the prior art, the portable terminal device 1 is fixed to a fixture so that the installation position and the installation angle of the portable terminal device 1 are known in advance, or the installation height of the portable terminal device 1 is set by initial setting of the application. It is possible to calculate the actual actual installation position and the actual installation angle of the mobile terminal device 1 without traveling on a road with a white line after inputting.

  Further, in the correction means 8, the reference position information of the portable terminal device 1 stored by the storage means 4 is reference position information that does not need to be corrected originally, and the portable information calculated by the calculation means 7 for the reference position information. In order to automatically correct the recognition result of the outside world recognized by the front camera 2 in accordance with the deviation amount of the actual installation position and the actual installation angle of the terminal device 1, the road position where the host vehicle travels, the inter-vehicle distance from the preceding vehicle, etc. Can be accurately detected. Further, based on the external recognition result recognized by the front camera 2 corrected by the correcting means 8, the distance between the front vehicle and the host vehicle traveling in the same lane as the host vehicle and the position of the traveling road of the host vehicle are accurately determined. Since it understands, this information can be utilized for a driver's driving support system.

  The present invention is not limited to the above-described embodiment, and various modifications other than those described above can be made without departing from the spirit of the present invention.

  For example, in the above-described embodiment, a plurality of predetermined characteristic portions 10a, 10b, and 10c in the vehicle interior that are determined in advance are the upper-side central portion 10a of the left headrest, the upper-side central portion 10b of the right-side headrest, and the upper-side central portion 10c of the back door glass. However, in addition to each of these features, it may be set at the corners at both ends of the upper side of the back door glass, the corners at both ends of the upper side of the side door glass, a room lamp, or the like. Further, as a feature portion, for example, a plurality of blue stars may be provided at an arbitrary location in the passenger compartment. In this case, the template image information relating to the blue star indicating the feature part may be stored in the storage unit 4, and the feature unit may be derived by the pattern matching method in the derivation unit 5.

  In the above-described embodiment, the derivation means 5 derives each feature by the pattern matching method. For example, as described above, when a blue star is provided in the vehicle interior as each feature, As feature part information for extracting the feature part, blue information and information on the positions of the five vertices of the star are stored in the storage means 4, and the blue star in the passenger compartment is stored based on these information. Can be derived by the deriving means 5.

DESCRIPTION OF SYMBOLS 1 ... Portable terminal device 2 ... Front camera 3 ... Back camera 4 ... Memory | storage means 5 ... Derivation means 6 ... Collation means 7 ... Calculation means 8 ... Correction means

Claims (1)

A portable terminal for use in a driving support system having a front camera for recognizing an external situation of a vehicle and a rear camera for recognizing a situation in a vehicle interior and performing driving assistance based on external image information recognized by the front camera In the device
Feature information for extracting a plurality of predetermined feature portions in the vehicle interior determined in advance based on an image in the vehicle compartment recognized by the rear camera, position information in the vehicle compartment of each feature portion, and the portable terminal Storage means for storing, for each vehicle type, predetermined reference position information that is information related to the installation position and the installation angle in the vehicle interior of the apparatus;
Derivation means for deriving the position of each feature in the image recognized by the rear camera based on the feature information stored by the storage means and the image information in the passenger compartment recognized by the rear camera; ,
Collation means for collating the position of each feature portion derived by the derivation means with the position information of each feature portion stored by the storage means;
Calculating means for calculating an actual installation position and an actual installation angle of the portable terminal device in a vehicle interior based on a verification result of the verification means;
Correction means for correcting an external recognition result recognized by the front camera,
The mobile terminal device, wherein the correction unit corrects an external environment recognition result recognized by the front camera in accordance with a deviation amount of the actual installation position and the actual installation angle with respect to the reference position information.

Images