JP2013055416A - Exterior video provision system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an exterior video provision system capable of realizing AR display with a high quality in a vehicle compartment using exterior video.SOLUTION: An on-vehicle machine 3 comprises: terminal information acquisition means 11 for acquiring terminal information including information relating to a direction of a camera 5 from a mobile terminal 2; on-vehicle camera/photographing region portion specification means 12 for specifying an on-vehicle camera 4 and the portion in the photographing region corresponding to the terminal information; video portion cut-out means 14 for cutting the video portion corresponding to the portion in the specified photographing region from the photographing video of the specified camera 4; image processing means 15 for executing image processing required for generating exterior video to be provided to the cut-out video portion; and provision video creation means 16 for creating AR display video not including a photographing obstacle on the vehicle.

Description

  The present invention relates to a vehicle exterior video providing system, and in particular, provides a user with a desired vehicle exterior video by cooperation of a mobile terminal equipped with a camera and an augmented reality function, a plurality of vehicle-mounted cameras, and a vehicle-mounted device. The present invention relates to a system for providing images outside the vehicle.

  2. Description of the Related Art Conventionally, mobile terminals such as mobile phones, smartphones, and e-books are generally equipped with a camera, and it has been possible to take still pictures and videos using this camera.

  Recently, application software that realizes a function called AR (Augmented Reality) running on a predetermined type of mobile terminal has been developed and provided. In the AR function, a balloon in which the name and description of a feature (building, land, etc.) in a captured image called an air tag is described on a captured image (image) captured by a camera mounted on a mobile terminal. It is possible to display the additional information such as etc. for use as a city guide.

JP 2010-98720 A JP 2003-283976 A JP 2005-184142 A

  By the way, as one use mode of a mobile terminal equipped with the above-described camera, there is a case where a mobile terminal is brought into a vehicle (automobile) cabin and a scenery outside the vehicle is photographed from the vehicle cabin.

  However, in such a case, conventionally, a vehicle part such as a pillar, dashboard or seat of a vehicle, a passenger, etc. are reflected as an obstacle, or reflection of light on a window glass, etc. There were times when it was not possible to shoot properly.

  In particular, when such a situation occurs while the vehicle is running, the subject has passed away even when attempting to take a picture again, so that the chance of taking a picture is often lost.

  With the widespread use of the AR function described above, it is considered that in the future, for example, there will be an increased chance of guiding the surrounding area from the passenger compartment while traveling at a travel destination. The problem of being unable to shoot is expected to become more serious. For example, FIG. 12 is a display image (schematic diagram) in which the scenery outside the vehicle is photographed from the rear seat of the vehicle interior and AR display is performed on the photographed image. As shown in FIG. 12, the display image shows only a part of the scenery outside the vehicle due to shooting obstacles (vehicle parts and sunlight reflections) on the vehicle, and the air tag is superimposed and displayed. Since the features outside the vehicle that should be directed are hidden by shooting obstacles, the content is difficult to grasp.

  In Patent Document 1, a subject that the user wants to see in a stadium or the like is detected based on the direction in which the user points the mobile terminal, and the detected subject is a projector installed in the stadium. A technique for photographing and providing it to a portable terminal is proposed.

  Japanese Patent Application Laid-Open No. 2004-228561 proposes a technique for requesting a subject desired by the user to be photographed by a third party around the user.

  However, since the technologies described in Patent Document 1 and Patent Document 2 are based on the premise that infrastructure and personnel such as cameras and servers other than the user-owned cameras are provided, it is possible to move from the moving passenger compartment. It is difficult to realistically cope with the case where the location of the subject covers an unspecified number (wide area) as in the case of shooting a scene outside the vehicle.

  Further, Patent Document 3 proposes a technique for cutting out the vehicle surroundings from a captured image of an in-vehicle camera corresponding to the orientation of the image display surface of a display arranged in the vehicle.

  The technology described in Patent Document 3 can self-contain a photographing operation only by a user-owned display and on-vehicle equipment, and can cope with acquisition of a photographed image of a subject outside the vehicle existing in an unspecified number of positions. It seems that the AR function cannot be supported because the camera-equipped mobile terminal is not used.

  Therefore, the present invention has been made in view of the above-described points, and can realize high-quality AR display in the vehicle interior using a vehicle exterior image. As a result, a guide function for features outside the vehicle in the vehicle interior can be realized. An object of the present invention is to provide a system for providing images outside the vehicle that can properly demonstrate the above.

  In order to achieve the above-described object, the outside-vehicle image providing system according to the present invention is equipped with a camera and can display a photographed image of the camera on a display unit. A mobile terminal capable of executing an augmented reality function that superimposes and displays additional information related to an object, and a predetermined mounting position that is different from each other at a predetermined mounting position of the vehicle, and each of the attachments around the vehicle A plurality of in-vehicle cameras that shoot for each predetermined imaging area corresponding to each orientation in position, and an in-vehicle device to which the portable terminal can be connected and to which the plurality of in-vehicle cameras are connected, A vehicle outside video providing system for providing a user with a desired vehicle outside video by cooperation of a terminal, the vehicle mounted camera and the vehicle mounted device, wherein the vehicle mounted device is connected Further, terminal information acquisition means for acquiring terminal information necessary for generating an external video to be provided including information on the camera orientation of the mobile terminal from the mobile terminal, and the terminal information acquired by the terminal information acquisition means From the in-vehicle camera corresponding to the in-vehicle camera and the in-vehicle camera / imaging area part specifying means for specifying the part in the imaging area of the in-vehicle camera, and the captured image of the in-vehicle camera specified by the in-vehicle camera / imaging area part specifying means, The video part cutout unit that cuts out the video part corresponding to the part in the imaging region specified by the in-vehicle camera / imaging region part specifying unit, and the video part cut out by the video part cutout unit provide the video part Image processing means for performing predetermined image processing necessary for generation of a vehicle exterior video, and the vehicle exterior video to be provided Then, a subject outside the vehicle corresponding to the orientation of the camera reflected in the terminal information by the camera of the portable terminal in the vehicle interior of the vehicle is photographed under a photographing environment assuming that the photographing obstacle on the vehicle has been removed. In order to provide the second outside image, the second outside image in which the additional information based on the augmented reality function is superimposed can be provided on the first outside image to be obtained in this case, The in-vehicle camera / imaging area part specifying unit specifies the in-vehicle camera in the direction closest to the camera direction of the mobile terminal reflected in the terminal information, and the image processing unit is configured to perform the predetermined image processing according to the predetermined image processing. A first outside video is generated, and at least one of the in-vehicle device and the mobile terminal superimposes the additional information on the first outside video generated by the image processing means. Accordingly, provided video generation means for generating the second outside video is provided.

  According to such a configuration, a high-quality second outside image (AR display image) based on the first outside image that does not reflect the shooting obstacle on the vehicle can be provided. The feature on the image outside the vehicle 2 can be accurately grasped based on the additional information.

  Further, when it is assumed that the display part of the mobile terminal facing the opposite direction to the camera of the mobile terminal is a mirror as the outside image to be provided, the camera information reflected in the terminal information in the vehicle interior It is possible to provide a third outside image corresponding to a mirror image outside the vehicle to be imaged in an environment assuming that the obstacle on the vehicle has been removed to the mirror opposite to the direction, and the third outside image The vehicle-mounted camera / imaging area part specifying unit specifies the vehicle-mounted camera closest to the direction of the camera of the mobile terminal reflected in the terminal information, and the image processing unit The third image outside the vehicle may be generated by performing a process including an inversion process of inverting the left and right of the video part as the predetermined image process.

  And according to such a structure, the obstruction on a vehicle is not reflected as a 3rd vehicle exterior image (vehicle exterior mirror image image) imitating an electronic mirror image | video, without requiring the camera for exclusive use of an electronic mirror. Since a high-quality video can be provided, convenience can be further improved.

  Further, as the outside image to be provided, the captured image of the subject outside the vehicle actually photographed in the actual photographing environment in the vehicle interior by the camera of the mobile terminal is made translucent to the first outside image. In addition, in order to provide the fourth vehicle-exposed image superimposed on the vehicle-mounted image, the vehicle-mounted camera / imaging region part specifying means reflects the portable information reflected in the terminal information. The in-vehicle camera in the direction closest to the direction of the camera of the terminal is specified, and the image processing means generates the first outside video by the predetermined image processing, and at least one of the in-vehicle device and the portable terminal is The fourth vehicle exterior projection is made by translucently superimposing the actually captured image of the object outside the vehicle on the first vehicle exterior image generated by the image processing means. It may comprise a second providing image generating means for generating.

  And according to such a structure, since the high quality 4th image outside a vehicle by which the imaging | photography obstacle on the vehicle was made translucent can also be provided, the convenience can further be improved.

  Furthermore, the in-vehicle camera is a camera capable of photographing a wide viewing angle range having a wide-angle lens, and the predetermined image processing is a distortion correction process for correcting distortion of the video portion caused by the wide-angle lens. May be included.

  According to such a configuration, it is possible to provide a higher quality external image with reduced distortion.

  The terminal information may include information related to the focal length of the camera of the mobile terminal.

  According to such a configuration, since the vehicle-mounted device can accurately detect the object outside the vehicle aimed by the camera of the mobile terminal, it is possible to provide an image outside the vehicle that reflects the user's intention more faithfully. it can.

  Further, the generated outside video to be provided may be provided by displaying on the display unit of the in-vehicle device.

  And according to such a structure, since a video outside a vehicle can be provided on the large screen of a vehicle-mounted device, a guide function with good visibility can be realized.

  Furthermore, the generated outside video to be provided may be provided by displaying on the display unit of the mobile terminal.

  And according to such a structure, high quality AR display which was not realizable only with the portable terminal is realizable on a portable terminal.

  In addition, at least one of the in-vehicle device and the mobile terminal may include a storage unit that stores the generated second outside image.

  And according to such a structure, the record of the trip with the description by an air tag can be made into a high quality video work by preserve | saving the 2nd vehicle outside image | video.

  Further, the plurality of in-vehicle cameras may be for displaying a monitoring image for monitoring the periphery of the vehicle on a display unit of the in-vehicle device.

  And according to such a structure, since the existing vehicle equipment can be utilized, the increase in the number of parts can be suppressed, and the guide function of the feature outside the vehicle in the vehicle interior can be realized economically. Can do.

  ADVANTAGE OF THE INVENTION According to this invention, the high quality AR display in a vehicle interior using the image outside a vehicle can be implement | achieved, and by extension, the guide function of the feature outside a vehicle in a vehicle interior can be exhibited appropriately.

1 is a schematic configuration diagram showing a first embodiment of an outside-vehicle video providing system according to the present invention. Explanatory drawing for demonstrating the example of arrangement | positioning of a vehicle-mounted camera in 1st Embodiment. The figure which shows an example of the 1st exterior image in 1st Embodiment. The figure which shows an example of the 2nd vehicle exterior image in 1st Embodiment. Flow chart showing the first embodiment Schematic configuration diagram showing a modification of the first embodiment The schematic block diagram which shows 2nd Embodiment of the vehicle external image provision system which concerns on this invention. Explanatory drawing for demonstrating specification of the site | part in a vehicle-mounted camera and imaging | photography area | region in 2nd Embodiment. Explanatory drawing for demonstrating the production | generation of the 3rd vehicle exterior image by inversion processing in 2nd Embodiment. A figure showing typically an example of the 4th outside picture in a 2nd embodiment. Flowchart showing the second embodiment Explanatory drawing for demonstrating the problem of the conventional AR display

(First embodiment)
Hereinafter, a first embodiment of an outside-vehicle image providing system according to the present invention will be described with reference to FIGS. 1 to 6.

As shown in FIG. 1, the outside-vehicle image providing system 1 in this embodiment is roughly classified as follows:
The mobile terminal 2 includes a mobile phone, a smartphone, an electronic book, or the like, an in-vehicle device 3, and a plurality of (5 in FIG. 1) in-vehicle cameras 4. The mobile terminal 2 can be connected to the in-vehicle device 3 by wireless communication means such as Bluetooth (registered trademark) or wired communication means such as USB. Each on-vehicle camera 4 is electrically connected to the on-vehicle device 3 by a connection cable.

  More specifically, as shown in FIG. 1, the portable terminal 2 is equipped with a digital camera 5 having a lens arranged in a state where external light can be captured on a predetermined surface side thereof. Yes. In the activated state, the camera 5 forms an image of an object (object light) captured from the outside through a lens as a captured image on an imaging surface of a solid-state imaging device such as a CCD or CMOS. Yes. The captured image formed on the imaging surface is confirmed in real time (live view display) on the display unit 6 provided on the back side with respect to the lens of the camera 5 even before the shutter is released. It is possible. As shown in FIG. 1, the mobile terminal 2 is equipped with various sensing devices 7 such as a geomagnetic sensor (digital compass), a three-axis acceleration sensor (gyroscope), and a zoom level / focus level control unit. The sensing device 7 can detect the direction (east, west, north and south), angle (roll, pitch, yaw) and focal length of the mobile terminal 2 (camera 5). Further, as shown in FIG. 1, the mobile terminal 2 is equipped with a position detection device 8 composed of a GPS receiver or the like, and the position detection device 8 detects the position (current location) of the mobile terminal 2 as absolute coordinates. It is supposed to be. Furthermore, the AR function is installed as the application software 19 in the mobile terminal 2, and when the AR function is activated by a user operation, the captured video displayed on the display unit 6 is displayed on the captured video. The air tag (additional information) corresponding to the feature can be displayed in a superimposed manner. Here, the air tag is information shared among users through the Internet, and the location of the user and the location of the camera 5 using the position information, the direction information, the angle information, and the like detected by the position detection device 8 and the sensing device 7 are used. Since it is associated with the direction, the air tag corresponding to the position of the mobile terminal 2 and the direction of the camera 5 during activation of the AR function can be acquired and displayed via the Internet. As shown in FIG. 1, the mobile terminal 2 includes a control unit 20 composed of a CPU or the like, and the control unit 20 performs various processes of the mobile terminal 2.

  On the other hand, as shown in FIG. 2, each in-vehicle camera 4 is mounted at a different mounting position of the vehicle 9 in a predetermined direction different from each other, and the periphery of the vehicle 9 is determined according to each direction at each mounting position. It is possible to shoot for each shooting area. Furthermore, each in-vehicle camera 4 is a camera capable of photographing a wide viewing angle range in which a wide-angle lens such as a fisheye lens is mounted. Furthermore, each in-vehicle camera 4 is a digital camera that captures each imaging region at a predetermined frame rate and forms an image on the imaging surface of a solid-state imaging device such as a CCD or CMOS. In the example of FIG. 2, one of the in-vehicle cameras 4 has a direction (posture) such that the left front portion of the vehicle 9 (for example, the left headlamp) looks down the road surface in front of the vehicle 9 diagonally from above. The left front camera 4a is capable of shooting a shooting area centered on the road surface in front of the left side of the vehicle 9. In addition, one of the other in-vehicle cameras 4 is mounted on the right front portion of the vehicle 9 (for example, the right headlight) in a right direction so that the road surface in front of the right side of the vehicle 9 is looked down obliquely from above. The right front camera 4b is capable of shooting a shooting area centered on the road surface on the right front side of the vehicle 9. Furthermore, one of these other in-vehicle cameras 4 is mounted on the left side of the vehicle 9 (for example, the left door mirror) with the left side mounted on the left side of the vehicle 9 so as to look down obliquely from above. The left side camera 4c is capable of shooting a shooting area centered on the road surface on the left side of the vehicle 9. Furthermore, one of these other in-vehicle cameras 4 is mounted on the right side of the vehicle 9 (for example, the right door mirror) in such an orientation that the road surface on the right side of the vehicle 9 is looked down obliquely from above. The right side camera 4d is capable of shooting a shooting area centered on the road surface on the right side of the vehicle 9. The remaining one in-vehicle camera 4e is a back camera 4e attached to the rear portion of the vehicle 9 (for example, a rear license garnish portion) so as to look down on the road surface behind the vehicle 9 from diagonally above. The back camera 4 e is capable of shooting a shooting area centered on the road surface behind the vehicle 9.

  And the vehicle exterior image provision system 1 in this embodiment can provide a user with the 2nd vehicle exterior image | video as a desired vehicle exterior image | video by cooperation (interlocking) of the portable terminal 2, each vehicle-mounted camera 4, and vehicle equipment 3. It is possible. However, the second outside-of-vehicle image is a shooting environment assuming that the shooting obstacle on the vehicle 9 is removed from the outside subject corresponding to the orientation of the camera 5 of the portable terminal 2 in the vehicle interior of the vehicle 9. This is an image in which an air tag (additional information) based on the AR function is superimposed on a first outside-vehicle image that should be obtained when shooting.

  Specifically, as shown in FIG. 1, the in-vehicle device 3 has an ECU (Electrical Control Unit) 10 as its main control unit, and this ECU 10 is stored in a storage unit (ROM or the like) (not shown). By executing the stored program, various processes for generating a video to be provided are performed.

  That is, as shown in FIG. 1, the ECU 10 has a terminal information acquisition unit 11 as terminal information acquisition means. The terminal information acquisition unit 11 receives a second vehicle exterior image (provided) including information on the orientation of the camera 5 of the mobile terminal 2 from the mobile terminal 2 in which the connection state with the in-vehicle device 3 is established (communicable). Terminal information necessary for generating a video outside the vehicle) is acquired by communication. This terminal information includes the detection result of the sensing device 7 described above. In addition to this, the terminal information may include a detection result of the position detection device 8, information on the angle of view of the camera 5, a request signal for the second vehicle exterior image, and the like.

  Further, as shown in FIG. 1, the ECU 10 includes an in-vehicle camera / imaging area part specifying unit 12 as an in-vehicle camera / imaging area part specifying unit. The in-vehicle camera / imaging area part specifying unit 12 specifies the in-vehicle camera 4 corresponding to the terminal information acquired by the terminal information acquiring unit 11 and the part in the imaging area. Specifically, in the present embodiment, the in-vehicle camera / imaging region region specifying unit 12 specifies the in-vehicle camera 4 that is closest to the direction (direction / angle) of the camera 5 of the mobile terminal 2 reflected in the terminal information. In addition, for example, a part corresponding to the photographing region of the camera 5 of the mobile terminal 2 detected based on the terminal information is identified as a part corresponding to the subject from the identified photographing region of the in-vehicle camera 4. ing. At this time, as information regarding the orientation of the in-vehicle camera 4, information obtained by storing a known external parameter of the in-vehicle camera 4 in a storage unit (not shown) may be used. At this time, for the angle information in the terminal information indicating the orientation of the camera 5 of the portable terminal 2, the acceleration value of the vehicle 9 is subtracted by subtracting the detection value of the acceleration sensor for the navigation device mounted on the vehicle 9. It is desirable to handle the information after correcting it to pure gravity acceleration (inclination of the terminal 2) from which the influence has been removed. Further, at this time, as described above, when a part corresponding to the imaging region of the camera 5 of the mobile terminal 2 is specified from the imaging region of the in-vehicle camera 4, the terminal information or the known information of the in-vehicle camera 4 is used. Based on the difference between the directions of the two cameras 4 and 5, the position, the angle of view, and the like, the corresponding part may be specified using the calculated difference. In this case, as the position of the in-vehicle camera 4, a position stored in the storage unit as a relative position with respect to a predetermined reference position of the vehicle 9 may be used. On the other hand, as for the position of the camera 5 of the portable terminal 2, means for detecting the relative position of the camera 5 with respect to the reference position of the vehicle 9 (for example, which seat of the vehicle 9 the operator of the camera 5 is sitting on To the in-vehicle device 3 as terminal information, and on the in-vehicle device 3 side, a predetermined fixed position associated with the notified seat may be determined as a relative position). Also good. Alternatively, when such a means is not provided, a predetermined position (for example, a position at a predetermined height in the center of the vehicle body) in the vehicle interior that is fixed in advance with respect to the reference position of the vehicle 9 is used as the camera 5 of the mobile terminal 2. It may be regarded as the position. Alternatively, the detection accuracy of the subject may be sacrificed to some extent, but the positions of both cameras 4 and 5 may be regarded as the same (positional difference is 0).

  Furthermore, as shown in FIG. 1, the ECU 10 includes a video part cutout unit 14 as a video part cutout unit. The video part cutout unit 14 determines whether the in-vehicle camera 4 or the imaging region part specifying unit 12 identifies the in-vehicle camera 4 or the imaging region part specifying unit 12 from the imaging image of the in-vehicle camera 4 or the imaging region part specifying unit 12. The video part corresponding to the part is cut out. At this time, as a captured image of the in-vehicle camera 4, as shown in FIG. 1, an image acquired and held in the image capture unit 13 in the ECU 10 is used. Here, the broken-line frame F in FIG. 2 indicates the left front camera specified by the in-vehicle camera / imaging region region specifying unit 12 from the captured image I of the left front camera 4a specified by the in-vehicle camera / imaging region region specifying unit 12. The range cut out as an image | video part applicable to the site | part in the imaging | photography area | region of 4a is illustrated.

  Returning to FIG. 1, the ECU 10 has an image processing unit 15 as image processing means. The image processing unit 15 performs predetermined image processing necessary for generating the second outside-vehicle image on the video part cut out by the video part cutting-out unit 14. In the present embodiment, the predetermined image processing is configured by viewpoint conversion processing and distortion correction processing for correcting image distortion caused by the tilt of the image and the distortion of the wide-angle lens caused by the mounting angle of the in-vehicle camera 4. Yes. Then, the image processing unit 15 generates the first outside video by performing such predetermined image processing.

  Here, FIG. 3 shows a photographed image corresponding to the scenery outside the vehicle as in FIG. 12 as the first outside image generated in this way. This first vehicle exterior image is a high quality image in which no shooting obstacles are reflected in the vehicle interior.

  Returning to FIG. 1, the outside-vehicle image providing system 1 includes a provided image generating unit 16 as provided image generating means. The provided video generation unit 16 generates a second external video by superimposing an air tag based on the AR function mounted on the mobile terminal 2 on the first external video generated by the image processing unit 15. It has become. Such a provided video generation unit 16 may be provided in the in-vehicle device 3 (ECU 10), may be provided in the mobile terminal 2 (control unit 20), or may be provided in the in-vehicle device 3 and the mobile terminal. 2 may be provided. Here, when the in-vehicle device 3 includes the provided video generation unit 16, the terminal information includes air tag information (including position information associated with the air tag) corresponding to the imaging region of the camera 5 of the mobile terminal 2. The provided video generation unit 16 may generate the second outside video by acquiring this information and pasting it on the corresponding position in the first outside video. On the other hand, when the provided video generation unit 16 is provided in the mobile terminal 2, the provided video generation unit 16 acquires the first vehicle exterior video generated by the image processing unit 15 from the in-vehicle device 3 through communication. The second outside image may be generated by pasting the air tag acquired by the mobile terminal 2 to the first outside image. The provided video generation unit 16 displays the second external video generated in this way on at least one of the display unit 17 of the in-vehicle device 3 and the display unit 6 of the mobile terminal 2 to present to the user. It is supposed to be.

  Here, FIG. 4 shows an AR display image corresponding to FIG. 3 as the second vehicle outside image displayed in this way. As can be seen by comparing this AR display image with FIG. 12, according to the present embodiment, it is possible to display a high-quality second outside image that does not reflect any shooting obstacles in the vehicle interior. A user who has viewed the video can easily and accurately grasp the features in the video based on the air tag.

  Next, the operation of the present embodiment will be described with reference to FIG.

  For the sake of convenience, in the initial state of FIG. 5, it is assumed that the connection state between the mobile terminal 2 and the in-vehicle device 3 is established via both communication modules (not shown). At this time, the mobile terminal 2 is triggered by a user operation, and the camera 5 is activated by the control unit 20, and an image captured of the scenery outside the vehicle by the camera 5 (a reality in which a shooting obstacle in the vehicle interior is reflected) (Photographed image under the shooting environment) is displayed on the display unit 6 in real time by the finder function. Further, at this time, the mobile terminal 2 corresponds to the feature in the captured image of the camera 5 because the application software 19 of the AR function stored in its storage unit is activated by the control unit 20 in response to a user operation. It is assumed that the air tag to be obtained is acquired through the Internet and superimposed on the corresponding position on the captured video. Furthermore, at this time, an operation button that enables an operation to instruct execution of AR display using the captured video of the in-vehicle camera 4 is displayed on the display unit 6 of the mobile terminal 2 by the application software of the mobile terminal 2. It is assumed that The display of the operation button may be accompanied by a message such as “The video from the in-vehicle camera is available”, for example. At this time, it is assumed that each vehicle-mounted camera 4 has started shooting in each shooting area. The start of shooting by the in-vehicle camera 4 may be triggered by establishment of a connection between the mobile terminal 2 and the in-vehicle device 3, or may be triggered by turning on the Acc power source.

  From such an initial state, first, in step 1 (ST1) of FIG. 5, the control unit 20 of the mobile terminal 2 performs an AR display using a captured image of the in-vehicle camera 4 as a user operation on the mobile terminal 2. It is determined whether or not an operation for instructing execution has been performed. If a positive determination result is obtained in step 1 (ST1), the process proceeds to step 2 (ST2). If a negative determination result is obtained, step 1 (ST1) is repeated.

  Next, in step 2 (ST2), terminal information is transmitted to the in-vehicle device 3 by the communication function of the portable terminal 2.

  Next, in step 3 (ST3), the terminal information acquisition unit 11 receives and acquires the terminal information transmitted in step 2 (ST2).

  Next, in step 4 (ST4), the in-vehicle camera / imaging region region specifying unit 12 identifies the corresponding in-vehicle camera 4 and the region in the imaging region based on the terminal information acquired in step 3 (ST3). To do.

  Next, in step 5 (ST5), the video part cutout unit 14 determines from the captured image of the in-vehicle camera 4 specified in step 4 (ST4) in the imaging region of the in-vehicle camera 4 specified in step 4 (ST4). The video part corresponding to the part of is cut out.

  Next, in step 6 (ST6), the image processing unit 15 performs viewpoint conversion processing on the video part cut out in step 5 (ST5).

  Next, in step 7 (ST7), the image processing unit 15 performs a distortion correction process on the video part subjected to the viewpoint conversion process in step 6 (ST6), thereby generating a first outside-vehicle image.

  Next, in step 8 (ST8), the provided video generation unit 16 superimposes an air tag on the first external video generated in step 7 (ST7), thereby generating a second external video.

Next, in step 9 (ST9), the provided video generation unit 16 displays the second outside-vehicle video generated in step 8 (ST8) on the display units 6 and 17, and the process ends.
(Modification)
Next, FIG. 6 shows a modification of the present embodiment.

  In addition to the basic configuration shown in FIG. 1, the outside-vehicle image providing system 1 according to the present modification further includes a provided image storage processing unit 18 as storage means. The provided video storage processing unit 18 stores the second vehicle outside video generated by the provided video generating unit 16 by recording it in its storage area triggered by a user operation. The provided video storage processing unit 18 is provided in one or both of the in-vehicle device 3 and the portable terminal 2 provided with the provided video generation unit 16. Further, the second outside video stored by the provided video storage processing unit 18 can be read and reproduced by the playback display function of the devices 3 and 2 including the provided video storage processing unit 18. .

  According to such a configuration, by storing the second outside-of-vehicle video, it is possible to finish a travel record with explanation by the air tag into a high-quality video work.

(Second Embodiment)
Next, a second embodiment of the outside-vehicle image providing system according to the present invention will be described with reference to FIGS. 7 to 11 with a focus on differences from the first embodiment.

  For convenience, portions having the same or similar basic configuration as the first embodiment will be described using the same reference numerals.

  The outside-vehicle image providing system 21 in the present embodiment shown in FIG. 7 is the same as the first embodiment in that a second outside-image can be provided as an outside-vehicle image to be provided.

  However, unlike the first embodiment, the outside-vehicle image providing system 21 in the present embodiment can selectively provide the third outside-image and the fourth outside-image in addition to the second outside-image. It is said that. Here, the third exterior video is the camera reflected in the terminal information in the vehicle interior when it is assumed that the display unit 6 of the mobile terminal 2 facing the opposite direction to the camera 5 of the mobile terminal 2 is a mirror. 5 is an image corresponding to a mirror image outside the vehicle to be imaged under an environment in which it is assumed that an obstacle on the vehicle has been removed by the mirror in the direction opposite to the direction 5. In addition, the fourth vehicle outside image is a semi-transparent image obtained by photographing the subject outside the vehicle actually captured under the actual shooting environment in the vehicle interior by the camera 5 of the portable terminal 2 in the first vehicle outside image. It is the image superimposed above.

  As to whether to provide the second outside video, the third outside video, or the fourth outside video, a menu screen for selecting one of these images is displayed on the display unit 6 of the mobile terminal 2. The selection may be made according to the result of the user operation on this menu screen.

  Here, since the configuration for providing the second outside video is as already described in the first embodiment, in this embodiment, the third outside video and the fourth outside video are provided. The explanation will focus on the specific composition of the.

(Configuration for providing a third vehicle exterior image)
First, when providing a 3rd vehicle outside image | video, the vehicle-mounted camera / imaging | photography region site | part specific | specification part 12 selects the vehicle-mounted camera 4 of the direction nearest to the direction of the camera 5 of the portable terminal 2 reflected in terminal information. It has come to identify. At this time, the in-vehicle camera / imaging area part specifying unit 12 specifies the imaging area of the camera 5 when it is assumed that the camera 5 is set in the opposite direction from the current direction for specifying the part in the imaging area of the in-vehicle camera 4. In addition, the direction information and the angle information in the terminal information are obtained by reversing 180 degrees, and the part corresponding to the obtained imaging area is specified from the imaging area of the in-vehicle camera 4. Good.

  For example, when the user points the camera 5 forward, as shown in FIG. 8, the in-vehicle camera / imaging region region specifying unit 12 causes the corresponding region in the imaging region of the back camera 4e (the broken line frame F in FIG. 8). Reference) will be specified.

  In addition, when providing the third outside image, the image processing unit 15 performs a reversing process for reversing the left and right of the video part after the viewpoint conversion process and the distortion correction process described above as the predetermined image process. It has become.

  FIG. 9 shows an example of such inversion processing using a video (a) before inversion and a video (b) after inversion. This inversion process is due to the fact that the video to be provided is a mirror image.

  Then, by such inversion processing, a third outside-vehicle image as shown in FIG. 9B is generated.

  Further, the image processing unit 15 displays the third outside video generated in this way on the display unit 17 of the in-vehicle device 3 and / or on the display unit 6 of the portable terminal 2 through communication. Provided to users.

(Configuration for providing the fourth outside image)
Next, in the case of providing the fourth outside video, the outside video providing system 21 generates the first outside video by the same method as in the first embodiment.

  And as shown in FIG. 7, as a structure for providing a 4th vehicle outside image | video, the vehicle outside image provision system 21 has the 2nd provision image | video production | generation part 22 as a 2nd provision image | video production | generation means. Yes. The second provided video generation unit 22 uses the first outside video generated by the image processing unit 15 as a subject outside the vehicle actually captured by the camera 5 of the mobile terminal 2 under the actual shooting environment in the vehicle interior. A fourth outside-of-vehicle image is generated by superimposing the captured image after translucent it. Note that the captured image of the subject outside the vehicle is an image having the same size as the first exterior image, and both images are based on images captured by the cameras 4 and 5 at the same time. It is desirable for the two images to have the same portions other than the shooting obstacle on the vehicle.

  Here, FIG. 10 schematically shows an example of such a fourth outside image.

  Such a second provided video generation unit 22 may be provided in the in-vehicle device 3 (ECU 10), may be provided in the mobile terminal 2 (control unit 20), or may be provided in the in-vehicle device 3. And the portable terminal 2 may be provided. Here, in the case where the on-vehicle device 3 includes the second provided video generation unit 22, the terminal information is configured to include the captured video of the camera 5 of the mobile terminal 2, and the provided video generation unit 22 What is necessary is just to produce | generate a 4th vehicle exterior image | video by superimposing on a 1st vehicle exterior image | video after acquiring a video | video and making it semi-transparent. On the other hand, when the second provided video generation unit 22 is provided in the mobile terminal 2, the second provided video generation unit 22 uses the first on-vehicle video generated by the image processing unit 15 to communicate with the in-vehicle device 3 by communication. And the fourth outside image may be generated by superimposing the captured image of the camera 5 of the mobile terminal 2 on the obtained first outside image after making it semitransparent. Then, the second provided video generation unit 22 displays the fourth external video generated in this manner on at least one of the display unit 17 of the in-vehicle device 3 and the display unit 6 of the mobile terminal 2, thereby Presented to the user.

  Next, the operation of this embodiment will be described with reference to FIG.

  For convenience, it is assumed that the connection state between the mobile terminal 2 and the vehicle-mounted device 3 is established in the initial state of FIG. At this time, in the mobile terminal 2, the camera 5 is activated by the control unit 20 in response to a user operation, and a captured image of the scenery outside the vehicle by the camera 5 is displayed on the display unit 6 in real time by the finder function. It is assumed that Further, at this time, it is assumed that an operation button capable of selecting display of any of the second to fourth outside images is displayed on the display unit 6 of the mobile terminal 2 in response to a user operation.

  Then, from such an initial state, first, in step 11 (ST11) of FIG. 11, any one of the second to fourth outside-vehicle images is performed as a user operation on the portable terminal 2 by the control unit 20 of the portable terminal 2. It is determined whether or not an operation for selecting a display has been performed. If a positive determination result is obtained in step 11 (ST11), the process proceeds to step 12 (ST12). If a negative determination result is obtained, step 11 (ST11) is repeated.

  Next, in step 12 (ST12), it is determined by the control unit 20 whether or not it is the second outside-vehicle image that is selected in step 11 (ST11). If a positive determination result is obtained in step 12 (ST12), the process proceeds to step 13 (ST13). If a negative determination result is obtained, the process proceeds to step 14 (ST14).

  Here, when it progresses to step 13 (ST13), the control part 20 starts the AR function of the portable terminal 2, and acquires the air tag applicable to the picked-up image of the camera 5. FIG. Then, after step 13 (ST13), the second exterior video is displayed on the display units 6 and 17 by sequentially executing the processes after step 2 (ST2) in FIG. 5 of the first embodiment.

  On the other hand, when the process proceeds to step 14 (ST14), it is determined by the control unit 20 whether or not the image selected in step 11 (ST11) is the third outside image.

  Here, a series of subsequent steps when a positive determination result is obtained in step 14 (ST14) will be described first.

  That is, when a positive determination result is obtained in step 14 (ST14), after sequentially executing a series of steps from step 2 (ST2) to step 7 (ST7) in FIG. 5 of the first embodiment, Proceed to step 15 (ST15). However, in this embodiment, when specifying the vehicle-mounted camera 4 in step 4, the vehicle-mounted camera 4 closest to the direction opposite to the direction of the camera 5 is specified.

  Next, in step 15 (ST15), the image processing unit 15 performs a reversal process on the image portion after the distortion correction process in step 7 (ST7), thereby generating a third vehicle exterior image.

  Next, in step 16 (ST16), the image processing unit 15 displays the third vehicle exterior image generated in step 15 (ST15) on the display units 6 and 17, and the process is terminated.

  Next, a subsequent series of steps when a negative determination result is obtained in step 14 (ST14) will be described.

  That is, when a negative determination result is obtained in step 14 (ST14), after sequentially executing a series of steps from step 2 (ST2) to step 7 (ST7) in FIG. 5 of the first embodiment, Proceed to step 17 (ST17).

  Next, in step 17 (ST17), the second provided video generation unit 22 makes the first video outside the vehicle generated in step 7 (ST7) semi-transparent to the video captured by the camera 2 of the mobile terminal 2. A fourth outside image is generated by superimposing at.

  Next, in step 18 (ST18), the second provided video generation unit 22 displays the fourth vehicle outside video generated in step 17 (ST17) on the display units 6 and 17, and the process is terminated.

  As described above, according to the present invention, by providing a high-quality second outside image based on the first outside image on which the shooting obstacle on the vehicle is not reflected, the second outside image is displayed. Since the feature can be accurately grasped based on the air tag, the guide function of the feature outside the vehicle in the vehicle interior can be properly exhibited.

  In addition, this invention is not limited to embodiment mentioned above, A various change can be made in the limit which does not impair the characteristic of this invention.

  For example, the in-vehicle device 3 and the in-vehicle camera 4 may be configured to cooperate with the plurality of mobile terminals 2 and provide a desired outside video for each terminal 2 individually to each terminal 2.

DESCRIPTION OF SYMBOLS 1 Outside-vehicle image provision system 2 Portable terminal 3 In-vehicle device 4 In-vehicle camera 5 Camera 11 Terminal information acquisition part 12 In-vehicle camera / imaging area part specifying part 14 Video part cutout part 15 Image processing part 16 Provided image generation part

Claims (9)

A camera is mounted, and the captured video of the camera can be displayed on the display unit, and an augmented reality function that superimposes and displays additional information related to the feature in the video can be executed on the captured video. Mobile devices,
A plurality of in-vehicle cameras that are attached to different predetermined mounting positions of the vehicle in different predetermined directions, and that image the periphery of the vehicle for each predetermined imaging region according to each direction at each of the mounting positions;
An in-vehicle device to which the portable terminal is connectable and the plurality of in-vehicle cameras are connected,
A vehicle outside video providing system for providing a user with a desired vehicle outside video by cooperation of the mobile terminal, the vehicle mounted camera and the vehicle mounted device,
The in-vehicle device is
Terminal information acquisition means for acquiring terminal information necessary for generation of a video outside the vehicle to be provided including information related to the orientation of the camera of the mobile terminal from the connected mobile terminal;
The in-vehicle camera corresponding to the terminal information acquired by the terminal information acquisition means and the in-vehicle camera / imaging area part specifying means for specifying the part in the imaging area thereof;
Video part cutout that cuts out a video part corresponding to a part in the imaging area specified by the in-vehicle camera / imaging area part specifying unit from the video image of the in-vehicle camera specified by the in-vehicle camera / imaging area part specifying part Means,
Image processing means for performing predetermined image processing necessary for generation of the video outside the vehicle to be provided on the video part cut out by the video part cutting-out means;
Assuming that the shooting obstacle on the vehicle has been removed from the vehicle corresponding to the orientation of the camera reflected in the terminal information by the camera of the mobile terminal in the vehicle interior of the vehicle as the vehicle outside image to be provided A second outside image in which the additional information based on the augmented reality function is superimposed on the first outside image to be obtained when the image is taken under the shooting environment.
In order to provide the second outside image,
The in-vehicle camera / imaging region part specifying unit specifies the in-vehicle camera in the direction closest to the direction of the camera of the mobile terminal reflected in the terminal information;
The image processing means generates the first vehicle exterior video by the predetermined image processing,
At least one of the in-vehicle device and the portable terminal includes provided video generation means for generating the second external video by superimposing the additional information on the first external video generated by the image processing means. An outside-vehicle video providing system characterized by having Assuming that the display of the mobile terminal facing the opposite side of the camera of the mobile terminal is a mirror as the video outside the vehicle to be provided, the direction of the camera reflected in the terminal information in the vehicle interior It is possible to provide a third outside image corresponding to a mirror image outside the vehicle to be imaged in an environment assuming that the obstacle on the vehicle has been removed to the mirror in the opposite direction,
In order to provide the third outside image,
The in-vehicle camera / imaging area part specifying unit specifies the in-vehicle camera in the closest direction opposite to the camera direction of the mobile terminal reflected in the terminal information;
The said image processing means produces | generates a said 3rd vehicle exterior image by performing the process including the reversal process which reverses right and left of the said image | video part as said predetermined | prescribed image process. Car outside video providing system. As a video outside the vehicle to be provided, the first video outside the vehicle is made semi-transparent from a video image of a subject outside the vehicle actually captured in the actual imaging environment in the vehicle interior by the camera of the mobile terminal. It is possible to provide a fourth outside image superimposed on the above,
In order to provide the fourth outside image,
The in-vehicle camera / imaging region part specifying unit specifies the in-vehicle camera in the direction closest to the direction of the camera of the mobile terminal reflected in the terminal information;
The image processing means generates the first vehicle exterior video by the predetermined image processing,
At least one of the in-vehicle device and the mobile terminal translucently superimposes a photographed image of a subject outside the vehicle that is actually photographed on the first image outside the vehicle generated by the image processing unit. The outside video providing system according to claim 1 or 2, further comprising second provided video generating means for generating the fourth outside video. The in-vehicle camera is a camera capable of photographing a wide viewing angle range having a wide-angle lens,
The outside image providing system according to any one of claims 1 to 3, wherein the predetermined image processing includes distortion correction processing for correcting distortion of the video portion caused by the wide-angle lens. . The outside terminal image providing system according to any one of claims 1 to 4, wherein the terminal information includes information related to a focal length of a camera of the portable terminal. The vehicle exterior image according to any one of claims 1 to 5, wherein the generated vehicle exterior image to be provided can be provided by being displayed on a display unit of the vehicle-mounted device. Offer system. The vehicle exterior image according to any one of claims 1 to 6, wherein the generated vehicle exterior image to be provided can be provided by being displayed on a display unit of the portable terminal. Offer system. 8. The vehicle exterior according to claim 1, wherein at least one of the in-vehicle device and the mobile terminal includes a storage unit that stores the generated second vehicle external image. Video providing system. The plurality of in-vehicle cameras are for displaying a monitoring image for monitoring the periphery of the vehicle on a display unit of the in-vehicle device. The outside-vehicle image providing system according to item.