JP6725734B2 - Image display device, image display system, image display method, and program - Google Patents

Description

The present invention relates to a technique of displaying an image showing the vicinity of a vehicle.

2. Description of the Related Art Conventionally, there has been known a technique of synthesizing images obtained by photographing the periphery of a vehicle such as an automobile and displaying the appearance of the periphery of the vehicle from a virtual viewpoint. With such a technique, a user (typically a driver) can confirm the situation around the vehicle while staying in the vehicle.

Further, in recent years, a technique of transmitting such an image to a mobile terminal carried by a user has been proposed (for example, Patent Document 1). By visually recognizing such an image, the user can understand the state of the vehicle and the risk of theft even if the vehicle is located remotely.

JP2012-121384A

However, in the proposed technique, the viewpoint of seeing the periphery of the vehicle is limited, and the peripheral image cannot always be visually recognized from the angle desired by the user. Therefore, for example, even if the image of a suspicious person approaching the vehicle is confirmed on the mobile terminal, it is not clear clearly what kind of person the suspicious person is and what kind of action the suspicious person is taking. On the contrary, there is a fear that the user's anxiety is promoted.

In view of the above problems, it is an object of the present invention to provide a technique capable of changing a virtual viewpoint to a viewpoint desired by a user when displaying an image showing a vehicle periphery from a virtual viewpoint.

In order to solve the above problems, a first aspect of the present invention is an image display device that displays an image, and a generation unit that generates a composite image viewed from a virtual viewpoint using an image captured by a camera, An operation detection unit that detects a user operation for changing the viewpoint of the composite image, the operation detection unit corresponding to an image region in which the composite image is displayed and the predetermined viewpoint. A button area in which a plurality of buttons for selecting the display of the composite image are displayed, and the image area displays the composite image corresponding to the viewpoint selected by the button, When a touch operation is performed in the image area, the composite image is displayed in the image area from the viewpoint corresponding to the touch operation, and the touch operation is a flick operation or a pinch-in operation.

A second aspect of the present invention is an image display device that displays an image, wherein the image capturing device uses a image captured by a camera to generate a composite image viewed from a virtual viewpoint; Operation detecting means for detecting a user operation for changing the viewpoint, wherein the operation detecting means displays an image area in which the combined image is displayed, and displays the combined image corresponding to the predetermined viewpoint. A button area in which a plurality of buttons for selecting are displayed, and the buttons are arranged in the up, down, left, and right directions of the button area corresponding to the front, rear, left, and right directions of the vehicle. , A button for selecting the display from the viewpoint of the initial display in a wide area around the vehicle is further displayed, and the composite image corresponding to the viewpoint selected by the button is displayed in the image area. It

Further, a third aspect of the present invention uses a captured image captured by a camera, obtains image information about a composite image generated in a form viewed from a virtual viewpoint, and an image based on the obtained image information. Display operation means for displaying, and operation detection means for giving a user instruction to change the viewpoint of the composite image displayed on the display means, wherein the operation detection means is a user's touch operation on the display means. A touch panel for detecting the, and the display means displays an image area in which the composite image is displayed, and a button area in which a plurality of buttons for instructing the display of the composite image corresponding to the predetermined viewpoint are displayed. And, in the image area, the composite image corresponding to the viewpoint selected by the button is displayed, and when a touch operation is performed in the image area, the touch operation is performed. The combined image of the viewpoint corresponding to is displayed in the image area, and the touch operation is a flick operation or a pinch-in operation.

Further, a fourth aspect of the present invention uses a captured image captured by a camera, obtains image information about a composite image generated in a form viewed from a virtual viewpoint, and an image based on the obtained image information. Display operation means for displaying, and operation detection means for giving a user instruction to change the viewpoint of the composite image displayed on the display means, wherein the operation detection means is a user's touch operation on the display means. A touch panel for detecting the, and the display means displays an image area in which the composite image is displayed, and a button area in which a plurality of buttons for instructing the display of the composite image corresponding to the predetermined viewpoint are displayed. And, the buttons correspond to the front, rear, left, and right directions of the vehicle, and the respective buttons are arranged in the up, down, left, and right directions of the button area, and in the button area, the initial display in a wide area around the vehicle is provided. The terminal device is characterized in that a button for selecting display from a viewpoint is further arranged, and the composite image corresponding to the viewpoint selected by the button is displayed in the image area.

In addition, in a fifth aspect based on the third or fourth aspect, the communication means receives a notification when the security device detects an abnormality.

According to the first to fifth aspects, the position of the virtual viewpoint of the composite image is changed based on the user's operation, so that the subject can be easily displayed from the viewpoint desired by the user.

Further, since the direction of the virtual viewpoint is determined toward the specific point set at the specific position of the vehicle and the position of the virtual viewpoint is changed based on the world coordinate system and the viewpoint coordinate system, the subject can be displayed from various directions and positions. ..

According to the fifth aspect of the invention, the user of the image display device can recognize that a prior phenomenon that may lead to the theft of the vehicle has occurred, even if the user is remotely located from the vehicle, and promptly displays a screen around the vehicle. You can check it at.

FIG. 1 is a diagram showing an outline of an image display system. FIG. 2 is a diagram showing an outline of the image display system. FIG. 3 is a block diagram showing the configuration of the image acquisition device according to the first embodiment. FIG. 4 is a diagram illustrating the arrangement of cameras. FIG. 5 is a block diagram showing the configuration of the mobile terminal according to the first embodiment. FIG. 6 is a diagram illustrating a method of generating a peripheral image and a composite image. FIG. 7 is a diagram showing the position of the virtual viewpoint. FIG. 8 is a diagram showing the position of the virtual viewpoint. FIG. 9 is a diagram for explaining the movement of the virtual viewpoint. FIG. 10 is a diagram for explaining the movement of the virtual viewpoint. FIG. 11 is a diagram for explaining the movement of the virtual viewpoint. FIG. 12 is a diagram for explaining the movement of the virtual viewpoint. FIG. 13 is a diagram illustrating a display image. FIG. 14 is a diagram illustrating a display image. FIG. 15 is a diagram illustrating a display image. FIG. 16 is a diagram illustrating a display image. FIG. 17 is a diagram illustrating a display image. FIG. 18 is a diagram illustrating a display image. FIG. 19 is a diagram illustrating a display image. FIG. 20 is a diagram illustrating a display image. FIG. 21 is a diagram illustrating a display image. FIG. 22 is a diagram illustrating a display image. FIG. 23 is a diagram illustrating a display image. FIG. 24 is a diagram illustrating a display image. FIG. 25 is a flowchart showing the processing procedure of the first embodiment. FIG. 26 is a flowchart showing the processing procedure of the first embodiment. FIG. 27 is a block diagram showing the configuration of the mobile terminal according to the first embodiment. FIG. 28 is a diagram for explaining the movement of the virtual viewpoint. FIG. 29 is a diagram for explaining the movement of the virtual viewpoint. FIG. 30 is a diagram illustrating a display image. FIG. 31 is a flowchart showing the processing procedure of the second embodiment. FIG. 32 is a flowchart showing the processing procedure of the second embodiment. FIG. 33 is a diagram showing a modified example of the image display system.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<1. First Embodiment>
<1-1. Overview>
FIG. 1 shows an outline of an image display system 1 according to an embodiment of the present invention. In the image display system 1, an image around the vehicle 2 acquired by the image acquisition device 3 provided in the vehicle 2 from the cameras 4 (4F, 4B, 4L, 4R) arranged in the vehicle 2 is located remotely from the vehicle 2. This is a system for displaying on the mobile terminal 8 of the user.

In such a system, the vehicle 2 is provided with a security device 5 having a monitoring sensor. The monitoring sensor is activated when the suspicious person SI approaches the vehicle 2 abnormally or physically damages the vehicle 2, and the security device 5 notifies the user that the monitoring sensor has been activated. To do. The user who receives the notification from the security device 5 requests the image acquisition device 3 to transmit the image, and causes the mobile terminal 8 to display the image around the vehicle 2. At this time, the user freely changes the display range of the image and refers to the situation around the vehicle 2 from a desired angle.

FIG. 2 is an example of the display 83 in which the image of the vicinity of the vehicle 2 is displayed on the mobile terminal 8. By operating the touch panel 83a of the mobile terminal 8, the user can change the display range of the image and inspect the concerned area in detail. The user can refer to the image around the vehicle 2 and remotely activate the alarm of the vehicle 2 or notify the security company if necessary. By using the image display system 1 as described above, the user can always maintain the vehicle 2 owned by the user safely even when the user is located remote from the vehicle 2.

<1-2. Composition>
Next, the configuration of each device included in the vehicle 2 used in the image display system 1 will be described. FIG. 3 shows a configuration of each device included in the vehicle 2. The vehicle 2 includes an image acquisition device 3, a camera 4, a security device 5, and an alarm device 6.

The image acquisition device 3 is an electronic control device that receives image data from the camera 4 installed in the vehicle and transmits the received image data to the mobile terminal 8. The image acquisition device 3 includes a control unit 31, a communication unit 32, and a storage unit 33.

The control unit 31 is a microcomputer including a CPU, RAM, and ROM. The control unit 31 is connected to another configuration included in the image acquisition device 3 and controls the entire device. The control unit 31 includes a request reception unit 31a, a camera control unit 31b, an image acquisition unit 31c, an image transmission unit 31d, and an antitheft unit 31e.

The request receiving unit 31a receives a request signal from the user who possesses the mobile terminal 8 to activate the camera 4 and start photographing.

When the request receiving unit 31a receives a request signal to start shooting from the user, the camera control unit 31b transmits an operation signal to the camera 4 and causes the camera 4 to start shooting.

The image acquisition unit 31c acquires the image data transmitted from the camera 4 and converts the acquired image data into a format that can be processed by the control unit 31.

The image transmission unit 31d transmits the image data acquired by the image acquisition unit 31c from the camera 4 to the mobile terminal 8 via the communication unit 32.

The anti-theft unit 31e transmits an operation signal to the alarm device 6 to cause the alarm device 6 to issue an alarm. This prevents the vehicle 2 from being stolen. The anti-theft unit 31e may cause the alarm device 6 to issue an alarm and control an engine control device (not shown) to prevent the engine from operating. Moreover, you may notify to a security company via the communication part 32. In short, the anti-theft unit 31e may use a device provided in the vehicle 2 and function to prevent the vehicle 2 from being stolen.

The communication unit 32 has a communication function using wireless communication, and performs information communication with the mobile terminal 8 via the network 7. For example, it is information communication using a wireless communication technology such as WiMAX (Worldwide Interoperability for Microwave Access) and LTE (Long Term Evolution).

The storage unit 33 is a memory that stores data. For example, it is a non-volatile storage medium such as an EEPROM (Electrical Erasable Programmable Read-Only memory), a flash memory, or a hard disk drive provided with a magnetic disk. The storage unit 33 also stores a program 33a.

The program 33 a is firmware that is read by the control unit 31 and is executed by the control unit 31 to control the image acquisition device 3.

Next, the camera 4 will be described. The camera 4 includes a front camera 4F, a rear camera 4B, a left side camera 4L, and a right side camera 4R. Each camera (4F, 4B, 4L, 4R) includes a lens and an image sensor, and is arranged at different positions of the vehicle 2.

FIG. 4 shows the positions where the cameras 4 (4F, 4B, 4L, 4R) are arranged in the vehicle 2 and the directions in which the optical axes of the cameras are directed. The front camera 4F is installed at the front end of the vehicle 2, and the optical axis 4Fa is directed in the straight traveling direction of the vehicle 2. The rear camera 4B is installed at the rear end of the vehicle 2, and the optical axis 4Ba is directed in the direction opposite to the straight traveling direction of the vehicle 2, that is, in the reverse traveling direction. The left side camera 4L is installed on the left side mirror ML, and the optical axis 4La is directed to the left direction of the vehicle 2 (direction orthogonal to the straight traveling direction). Further, the right side camera 4R is installed on the right side mirror MR, and its optical axis 4Ra is directed to the right direction of the vehicle 2 (direction orthogonal to the straight traveling direction).

Each camera (4F, 4B, 4L, 4R) of the camera 4 captures images in different directions around the vehicle 2 and acquires captured images electronically. Further, the lens included in the camera 4 has a relatively shorter focal length than a standard lens and has an angle of view θ of 180 degrees or more. Therefore, by using the four cameras 4, the entire circumference of the vehicle 2 can be photographed.

Referring back to FIG. The security device 5 is a device that detects a prior phenomenon leading to theft of the vehicle 2 or an article in the vehicle 2 and notifies the mobile terminal 8 of the user of the vehicle 2 of the possibility of theft by e-mail. .. The security device 5 includes a monitoring sensor 5a and an email notification unit 5b.

The monitoring sensor 5a is a sensor that detects a prior phenomenon that may lead to theft of the vehicle 2 or articles in the vehicle 2. For example, a vibration sensor that detects vibration generated in the vehicle 2, a glass breakage sensor that detects breakage of the glass of the vehicle 2, an inclination sensor that detects the inclination of the vehicle 2 due to a jack or a crane, an intruder inside the vehicle 2 For example, an intrusion sensor for detecting.

When the monitoring sensor 5a detects a prior phenomenon that leads to theft, the email notification unit 5b notifies the user that the fear of theft has occurred. Specifically, the email notification unit 5b generates an email having the content of the fear of theft, and sends the email to the mobile terminal 8 carried by the user. Note that the notification may be made by voice information instead of the text information using the electronic mail. In this case, the mobile terminal 8 may read the voice information. Further, when the security device 5 is provided with a plurality of monitoring sensors, the mail notification unit 5b preferably includes in the e-mail which monitoring sensor has detected a prior phenomenon leading to theft. This is because it is easy for the user to understand the situation of the vehicle 2.

The alarm device 6 is a device that gives a sound to the surroundings to give a warning. It is a horn provided for the speaker and the vehicle 2. The alarm device 6 may be one that emits light as well as one that emits sound. For example, it is a warning light or a light provided in the vehicle 2. In short, it is sufficient that it calls attention to the surroundings and gives a warning.

Next, the mobile terminal 8 will be described. The mobile terminal 8 is an information terminal that is carried by a user and has a function of displaying an image, a function of connecting to an information network, and the like. For example, mobile phones and smartphones. FIG. 5 shows the configuration of the mobile terminal 8. The mobile terminal 8 includes a control unit 81, a communication unit 82, a display 83, and a storage unit 84.

The control unit 81 is a microcomputer including a CPU, RAM, and ROM. The control unit 81 is connected to another configuration included in the mobile terminal 8 and controls the entire terminal. Each function of the control unit 81 will be described later.

The communication unit 82 has a communication function using wireless communication, and performs information communication with the image acquisition device 3 and the security device 5 via the network 7. For example, it is information communication using a wireless communication technology such as WiMAX (Worldwide Interoperability for Microwave Access) and LTE (Long Term Evolution).

The display 83 displays various information such as characters and figures, and visually presents the information to the user. For example, it is a display device such as a liquid crystal display, a plasma display, or an organic EL display. The display 83 includes a touch panel 83a.

The touch panel 83a senses a user's touch on the button area displayed on the display 83, and transmits the sensed position information to the control unit 81.

The storage unit 84 is a memory that stores data. For example, it is a non-volatile storage medium such as an EEPROM (Electrical Erasable Programmable Read-Only memory), a flash memory, or a hard disk drive provided with a magnetic disk. The storage unit 84 also stores vehicle body image data 84a and a program 84b.

The vehicle body image data 84a is image data showing the appearance of the vehicle 2. The vehicle body image data 84a includes image data when the vehicle 2 is viewed from any angle outside. The vehicle body image data 84a may be acquired from an external server via the network 7 after the image acquisition device 3 is attached to the vehicle 2 without being stored in the mobile terminal 8 in advance. In this case, the vehicle body image data 84a that matches the appearance of the vehicle 2 to which the image acquisition device 3 is attached can be acquired. The user may transmit the vehicle type name of the vehicle 2 from the mobile terminal 8 to the external server and request the vehicle body image data 84a.

The program 84b is firmware read by the control unit 81 and executed by the control unit 81 to control the mobile terminal 8.

Each function of the control unit 81 will be described. The control unit 81 includes an image acquisition unit 81a, an image generation unit 81b, a display control unit 81c, and an operation detection unit 81d.

The image acquisition unit 81a acquires the image data transmitted from the image acquisition device 3 and converts the acquired image data into a format that can be processed by the control unit 81. That is, the image acquisition unit 81a acquires a plurality of images obtained by shooting the subject with each of the plurality of cameras 4, and the image generation unit 81b combines the plurality of captured images acquired by the camera 4 and the vehicle 2 viewed from the virtual viewpoint. A peripheral image showing the surroundings is generated. The virtual viewpoint is a viewpoint overlooking the vehicle 2 from a position outside the vehicle 2. In addition, the image generation unit 81b superimposes the vehicle body image showing the vehicle 2 that can be seen from the virtual viewpoint on the peripheral image. The peripheral image generation method and the virtual viewpoint setting method by the image generation unit 81b will be described later.

The display control unit 81c displays data such as images and characters on the display 83. Further, the display control unit 81c changes the image or the like displayed on the display 83 based on the touch position input on the touch panel 83a.

The operation detector 81d detects an operation on the touch panel 83a by the user. Specifically, the operation detection unit 81d detects in which direction on the touch panel 83a the user's fingertip has moved after touching the touch panel 83a based on the touch position information transmitted from the touch panel 83a.

The operation performed on the touch panel 83a by the user is, for example, a so-called flick operation in which the fingertip is slid while being in contact with the touch panel 83a. Further, this is a so-called pinch-in operation in which the distance between the fingertips is reduced while the two fingertips are in contact with the touch panel 83a. Further, this is a so-called pinch-out operation in which the distance between the fingertips is widened while the two fingertips are in contact with the touch panel 83a.

The notification unit 81e transmits a predetermined signal to a device external to the mobile terminal 8 via the communication unit 82. The signal transmitted by the notification unit 81e is, for example, a signal requesting the image acquisition apparatus to transmit image data.

<1-3. Generation of peripheral image and composite image>
Next, a method will be described in which the image generation unit 81b generates the peripheral image AP showing the peripheral region of the vehicle 2 and the composite image CP in which the vehicle body image 10 is superimposed on the peripheral image AP. FIG. 6 shows a method in which the image generation unit 81b generates the peripheral image AP.

First, when each of the front camera 4F, the rear camera 4B, the left side camera 4L, and the right side camera 4R takes a picture around the vehicle 2, four images AP showing the front, the rear, the left side, and the right side of the vehicle 2 are shown. (F), AP(B), AP(L), AP(R) are acquired. These four images include image data showing the entire circumference of the vehicle 2.

The image generation unit 81b converts the data (value of each pixel) included in these four images AP(F), AP(B), AP(L), and AP(R) into a three-dimensional curved surface in a virtual three-dimensional space. Is projected onto the projection surface TS. The projection surface TS has, for example, a substantially hemispherical shape (bowl shape). The center of the projection plane TS (bottom of the bowl) is the position where the vehicle 2 exists. A portion other than the central portion of the projection surface TS is associated with any one of the images AP(F), AP(B), AP(L), and AP(R).

The image generation unit 81b projects the peripheral images AP(F), AP(B), AP(L), and AP(R) onto a portion other than the central portion of the projection surface TS. The image generation unit 81b projects the image AP(F) of the front camera 4F on an area corresponding to the front of the vehicle 2 on the projection surface TS, and projects the image AP(B) of the rear camera 4B on the area corresponding to the rear of the vehicle 2. To do. Further, the image generation unit 81b projects the image AP(L) of the left side camera 4L on the area corresponding to the left side of the vehicle 2 on the projection surface TS, and the image of the right side camera 4R on the area corresponding to the right side of the vehicle 2. Project AP(R). As a result, a peripheral image AP showing the entire area around the vehicle 2 is generated.

Next, the image generation unit 81b sets a virtual viewpoint VP from an arbitrary viewpoint position in this three-dimensional space to an arbitrary viewpoint direction. Then, the image data projected on the area included in the viewing angle viewed from the set virtual viewpoint VP on the projection surface TS is cut out. A peripheral image AP showing an area around the vehicle 2 viewed from the virtual viewpoint VP is generated from the cut-out image data.

The image generation unit 81b reads the vehicle body image data from the storage unit 84 and generates the vehicle body image 10 of the vehicle 2 viewed from the virtual viewpoint VP. Then, the vehicle body image 10 is combined with the peripheral image AP showing the area around the vehicle 2 as viewed from the virtual viewpoint VP to generate a combined image CP.

When the image generation unit 81b sets a virtual viewpoint VPt in which the viewpoint position is directly above the vehicle 2 and the view direction is directly below, the peripheral images AP(F), AP(B), AP(L), AP(R) and The body image 10 is used to generate a synthetic image CPt that looks down at the vehicle 2 and the surrounding area of the vehicle 2. The composite image CPt is an image that looks down on the vehicle 2 from directly above, and shows a region around the vehicle 2.

Further, when the image generation unit 81b sets a virtual viewpoint VPb in which the viewpoint position is the upper front of the vehicle 2 and the view direction is the lower rear of the vehicle 2, the peripheral images AP(B), AP(L), AP(R). ) And the vehicle body image 10 are used to generate a synthetic image CPb that looks down at the vehicle 2 and the surrounding area of the vehicle 2. The composite image CPb is an image of the vehicle 2 looking down from the upper front to the rear, and shows a region behind the vehicle 2.

Further, when the image generation unit 81b sets a virtual viewpoint VPl in which the viewpoint position is diagonally rearward and upward of the vehicle 2 and the visual field direction is forward and downward of the vehicle 2, the peripheral images AP(F), AP(L), Using AP(R) and the vehicle body image 10, a synthetic image CPl that looks down at the vehicle 2 and the surrounding area of the vehicle 2 is generated. The composite image CPl is an image in which the vehicle 2 is viewed diagonally to the left rearward and from above to the front, and shows a region on the left side of the vehicle 2.

<1-4. Setting virtual viewpoint>
Next, a method for the image generation unit 81b to set the virtual viewpoint VP will be described in detail. 7 to 12 show a method in which the image generation unit 81b sets the virtual viewpoint VP. In addition, in each drawing, a three-dimensional XYZ orthogonal coordinate system cc is used to indicate an appropriate direction and direction. Each coordinate axis of the orthogonal coordinate system cc is fixed relative to the vehicle 2. That is, the X-axis direction is the left-right direction of the vehicle 2, the Y-axis direction is the front-back direction of the vehicle 2, and the Z-axis direction is the vertical direction. Further, the +X side is the right side of the vehicle 2, the +Y side is the front side of the vehicle 2, and the +Z side is the vertically upper side. Therefore, the -X side is the left side of the vehicle 2, the -Y side is the rear side of the vehicle 2, and the -Z side is the vertically lower side.

First, refer to FIG. FIG. 7 is a diagram of the vehicle 2 viewed from the vertically upward direction (+Z side), and shows five reference virtual viewpoints VPa (VPat, VPaf, VPab, VPal, VPar) that serve as references of the virtual viewpoint VP.

Each of the five reference virtual viewpoints VPa has five reference positions (VLt, VLf, VLb, VLl, VLr), five movement center points RC (RCt, RCf, RCb, RCl, RCr), and five reference directions (VDt). , VDf, VDb, VDtl, VDr).

The reference position VLt is located directly above the vehicle 2. The reference position VLf is located slightly forward of the vehicle 2 (on the +Y side). The reference position VLb is located slightly behind the vehicle 2 (on the −Y side). The reference position VLl is located diagonally left rear of the vehicle 2 (−Y side and −X side). The reference position VLr is located diagonally to the right rear of the vehicle 2 (+X side and −Y side). The height (on the +Z side) of each reference position VL is the same as the height from the ground plane GR with which the vehicle 2 is in contact. For example, it is twice as high as the vehicle height.

FIG. 8 is a diagram of the vehicle 2 viewed from the right side (+X side) in the horizontal direction, and is a diagram illustrating the movement center point RC and the reference direction VD. The movement center point RC is a reference point that serves as a center when the virtual viewpoint VP moves. Therefore, even if the viewpoint position of the virtual viewpoint VP moves, the viewpoint direction always faces the movement center point RC. A method of moving the virtual viewpoint VP viewpoint position using the moving center point RC will be described later.

The movement center point RCt is a specific point set at a specific position of the vehicle. The movement center point RCt is located at the center of the vehicle 2. The movement center point RCf is located at the left-right center of the vehicle 2 and slightly forward (on the +Y side) of the front end of the vehicle 2. The movement center point RCb is located at the left-right center of the vehicle 2 and slightly rearward (−Y side) of the rear end of the vehicle 2. The movement center point RCl is located at the front-rear center of the vehicle 2 and slightly outside of the vehicle 2 (−X side). The movement center point RCr is located at the center of the vehicle 2 in the front-rear direction and slightly outside the vehicle 2 (on the +X side). The height (on the +Z side) of each movement center point RC is the height h of the line of sight of the user sitting in the vehicle 2.

The reference direction VDt faces from the reference position VLt to the movement center point RCt (−Z side). The reference direction VDf faces the direction (−Z side and +Y side) of the movement center point RCf from the reference position VLf. The reference direction VDb faces the direction (−Z side and −Y side) of the movement center point RCb from the reference position VLb. The reference direction VDl faces the direction (−Z side and +Y side) of the movement center point RCl from the reference position VL1. The reference direction VDr faces the direction (−Z side and +Y side) of the movement center point RCr from the reference position VLr.

As described above, the virtual viewpoint VP is a viewpoint in which the movement center point RC is viewed from the viewpoint position VD. The reference virtual viewpoint VPat is a viewpoint (top view) that looks down at the vehicle 2 from directly above. The reference virtual viewpoint VPaf is a viewpoint where the vehicle 2 is viewed from the front and the front of the vehicle 2 from above (front view). The reference virtual viewpoint VPab is a viewpoint (back view) in which the vehicle 2 is viewed from behind and above the vehicle 2. The reference virtual viewpoint VPal is a viewpoint (left side view) in which the vehicle 2 is viewed from the diagonally left rear side and from above of the left side region of the vehicle 2. The reference virtual viewpoint VPar is a viewpoint (right side view) in which the vehicle 2 is viewed from the diagonally right rear side and the right side area of the vehicle 2 from above.

Such a reference virtual viewpoint VPa is selected by the user. The selection by the user is made by touching a touch panel button associated with each reference virtual viewpoint VP. That is, when the user selects one of the reference virtual viewpoints VPa while the composite image CP is displayed on the display, the image generation unit 81b generates a composite image viewed from the reference virtual viewpoint VPa. Thereby, even if the user cannot change the virtual viewpoint to the reference virtual viewpoint VP after changing the position of the virtual viewpoint variously, the user can easily set the virtual viewpoint to the reference virtual viewpoint VP by touching the touch panel button. can do.

It should be noted that, in the initial stage of image display before the user selects the viewpoint, one of the reference virtual viewpoints VPa is preset. At this time, it is preferable to generate the composite image CP at the reference virtual viewpoint VPat that looks down at the vehicle 2 from directly above. This is because a wide area around the vehicle 2 can be presented to the user and is suitable as an initial display.

Next, a method of moving the viewpoint position VL using the movement center point RC will be described. First, a method of moving the viewpoint position VL in the left-right direction (horizontal direction) when the user views the display will be described with reference to FIGS. 9 and 10. The movement of the viewpoint position VL described below is performed in response to a flick operation on the touch panel by the user in the left-right direction (horizontal direction).

FIG. 9 is a diagram of the vehicle 2 viewed from the vertically upward direction (+Z side), and shows a trajectory along which the reference position VLl of the reference virtual viewpoint VPal moves. The reference position VLl moves along the movement locus OBlh around the movement center point RCl. The movement of the viewpoint position VL is based on an orthogonal coordinate system cc (world coordinate system) that is relatively fixed to the vehicle 2 whose origin is the position of the movement center point RCl. The coordinate axis that serves as a reference for movement of the viewpoint position VLl is the Z axis of the orthogonal coordinate system cc at the position of the movement center point RCl.

FIG. 10 is a diagram of the vehicle 2 viewed from the right side (+X side) in the horizontal direction, and shows a trajectory along which the reference position VLl of the reference virtual viewpoint VPal moves, as in FIG. 9. The reference position VLl moves along the movement locus OBlh around the movement center point RCl. The coordinate axis that serves as a reference for movement of the viewpoint position VLl is the Z axis (AXl) of the orthogonal coordinate system cc at the position of the movement center point RC.

9 and 10, the viewpoint direction VDl faces the direction of the movement center point RCl from the viewpoint position VLl, wherever the viewpoint position VLl is located on the movement trajectory OBlh. Therefore, when the reference position VLl moves on the movement locus OBlh, the user looking at the display 83 looks as if the image is moving in the horizontal direction.

When the user performs a flick operation on the touch panel 83a from left to right (horizontal direction), the VLl moves counterclockwise on the movement locus OBlh. On the other hand, when the user performs a flick operation from the right direction to the left direction (horizontal direction) on the touch panel 83a, VLl moves clockwise on the movement track OBlh. As a result, the direction of the user's flick operation matches the direction of movement of the image, and the user can intuitively perform the touch panel operation.

Next, a method of moving the viewpoint position VL in the up-down direction (vertical direction) when the user views the display will be described with reference to FIGS. 11 and 12. The movement of the viewpoint position VL described below is performed in response to a flick operation in the vertical direction (vertical direction) on the touch panel by the user.

11 and 12, a three-dimensional XYZ Cartesian coordinate system VPcc (viewpoint coordinate system) is used in addition to the three-dimensional XYZ Cartesian coordinate system cc, and directions and directions are shown as appropriate. Each coordinate axis of the orthogonal coordinate system VPcc is fixed relative to the virtual viewpoint VP. That is, the X-axis (VPx) direction is the left-right direction of the virtual viewpoint VP, the Y-axis (VPy) direction is the front-back direction of the virtual viewpoint VP, and the Z-axis (VPz) direction is the vertical direction of the virtual viewpoint VP. The +X side is the right side of the virtual viewpoint VP, the +Y side is the front side of the virtual viewpoint VP, and the +Z side is the vertical upper side of the virtual viewpoint VP. Therefore, the −X side is the left side of the virtual viewpoint VP, the −Y side is the rear side of the virtual viewpoint VP, and the −Z side is the vertically lower side.

FIG. 11 is a diagram of the vehicle 2 viewed from the vertically upward direction (+Z side), and shows a trajectory along which the reference position VLl of the reference virtual viewpoint VPal moves. The reference position VLl moves along the movement locus OBlv around the movement center point RCl. The movement of the viewpoint position VL is based on the orthogonal coordinate system VPcc relatively fixed to the virtual viewpoint VP having the origin of the position of the movement center point RCl. The coordinate axis on which the viewpoint position VLl moves is the X axis (AXlv) of the orthogonal coordinate system VPcc at the position of the movement center point RCl.

FIG. 12 is a diagram of the vehicle 2 viewed from the right side (−X side) in the horizontal direction, and shows the trajectory along which the reference position VLl of the reference virtual viewpoint VPal moves, as in FIG. 11. The reference position VLl moves along the movement locus OBlv around the movement center point RCl. The coordinate axis that serves as a reference for moving the viewpoint position VLl is the X axis of the orthogonal coordinate system cc at the position of the moving center point RC.

11 and 12, the viewpoint direction VDl faces the direction of the movement center point RCl from the viewpoint position VLl, wherever the viewpoint position VLl is located on the movement trajectory OBlv. Therefore, when the reference position VLl moves on the movement locus OBlv, the user looking at the display 83 looks as if the image is moving vertically.

The movement locus OBlv is set above the ground contact surface GR of the vehicle 2. Therefore, the reference position VLl moves along the movement locus OBlv indicated by the broken line, and the movement locus OBlx indicated by the solid line does not move. As a result, it is possible to prevent the image display from a viewpoint position where the vehicle 2 is viewed from below the ground contact surface, which would not normally occur. The user can move the viewpoint position without feeling discomfort.

When the user performs a flick operation on the touch panel 83a from the upper direction to the lower direction (vertical direction), the VLl moves clockwise on the movement locus OBlv. On the other hand, when the user performs a flick operation from the lower direction to the upper direction (vertical direction) on the touch panel 83a, the VLl moves on the movement locus OBlv in the counterclockwise direction. As a result, the direction of the user's flick operation matches the direction of movement of the image, and the user can intuitively perform the touch panel operation.

<1-5. Example of displaying composite image>
Next, an example of the composite image CP displayed on the display 83 will be described with reference to FIGS. 13 to 24.

FIG. 13 is an example in which the reference virtual viewpoint VPat is set to the virtual viewpoint VP, and the composite image CPt1 in which the vehicle 2 is viewed from the reference virtual viewpoint VPat is displayed on the display 83. The composite image CPt1 displays the vehicle body image 10 in addition to the peripheral image showing the periphery of the vehicle 2. The composite image CPt1 is displayed when the user performs a touch operation on the touch panel button TB13 indicating the periphery of the vehicle 2 and when the composite image CP is initially displayed. By referring to the composite image CPt1, the user can check the situation around the vehicle 2 in a list.

In FIG. 14, after the composite image CPt1 (FIG. 13) is displayed, the reference virtual viewpoint VPat is set by moving clockwise on OBlh in FIG. 9, and the vehicle 2 is viewed from the set reference virtual viewpoint VPat. CPt2 is an example displayed on the display 83. The composite image CPt2 displays the vehicle body image 10 in addition to the peripheral image showing the periphery of the vehicle 2 which is rotated leftward when the display 83 is viewed from the user. The composite image CPt2 is displayed when the flick operation FH is performed from the left direction to the right direction (horizontal direction) on the touch panel 83a while the composite image CPt1 (FIG. 13) is displayed. By referring to the composite image CPt2, the user can confirm the situation around the vehicle 2 from an angle different from that of the composite image CPt1.

In FIG. 15, after the composite image CPt1 (FIG. 13) is displayed, the reference virtual viewpoint VPat is set by moving counterclockwise on OBlh in FIG. 12, and the vehicle 2 is viewed from the set reference virtual viewpoint VPat. The image CPt3 is an example displayed on the display 83. The composite image CPt3 is displayed including immediately below the front end of the vehicle 2. The combined image CPt2 is displayed when the combined image CPt1 (FIG. 13) is displayed and the flick operation FV is performed on the touch panel 83a from the upper direction to the lower direction (vertical direction). By referring to the composite image CPt2, the user can confirm the situation around the vehicle 2 including the area immediately below the front end.

FIG. 16 is an example in which the reference virtual viewpoint VPaf is set as the virtual viewpoint VP, and the composite image CPf1 of the vehicle 2 viewed from the reference virtual viewpoint VPaf is displayed on the display 83. The composite image CPf1 displays the vehicle body image 10 in addition to the peripheral image showing the front of the vehicle 2. The composite image CPf1 is displayed when the user touch-operates the touch panel button TB16 indicating the front of the vehicle 2. By referring to the composite image CPf1, the user can confirm the situation in front of the vehicle 2.

17 is a composite image in which the vehicle 2 is viewed from the set reference virtual viewpoint VPaf by moving the OBlh in FIG. 9 in the clockwise direction after the display of the composite image CPf1 (FIG. 16) and setting the reference virtual viewpoint VPaf. CPf2 is an example displayed on the display 83. The composite image CPf2 displays the vehicle body image 10 in addition to the peripheral image showing the periphery of the vehicle 2 which is rotated leftward when the display 83 is viewed from the user. The composite image CPf2 is displayed when the flick operation FH is performed from the left direction to the right direction (horizontal direction) on the touch panel 83a while the composite image CPf1 (FIG. 16) is displayed. By referring to the composite image CPf2, the user can confirm the situation in front of the vehicle 2 from an angle different from that of the composite image CPf1.

In FIG. 18, after the combined image CPf1 (FIG. 16) is displayed, the reference virtual viewpoint VPaf is set by moving the object OBlh in the counterclockwise direction in FIG. The image CPf3 is an example displayed on the display 83. The composite image CPf3 displays the front area of the vehicle 2 in more detail than the composite image CPf1. The composite image CPt3 is displayed when the flick operation FV is performed from the upper direction to the lower direction (vertical direction) on the touch panel 83a while the composite image CPf1 (FIG. 16) is displayed. The user can check the situation in front of the vehicle 2 in detail by referring to the composite image CPt3.

FIG. 19 is an example in which the reference virtual viewpoint VPab is set as the virtual viewpoint VP, and the combined image CPb1 of the vehicle 2 viewed from the reference virtual viewpoint VPab is displayed on the display 83. The composite image CPb1 displays the vehicle body image 10 in addition to the peripheral image showing the rear of the vehicle 2. The composite image CPb1 is displayed when the user touch-operates the touch panel button TB19 indicating the rear of the vehicle 2. By referring to the composite image CPb1, the user can confirm the situation behind the vehicle 2.

20 is a composite image in which the vehicle 2 is viewed from the set reference virtual viewpoint VPab by moving the OBlh in FIG. 9 in the clockwise direction after setting the composite image CPb1 (FIG. 19) and setting the reference virtual viewpoint VPab. CPb2 is an example displayed on the display 83. The composite image CPb2 displays the vehicle body image 10 in addition to the peripheral image showing the rear of the vehicle 2 displayed by moving to the left when the user views the display 83. The composite image CPb2 is displayed when the flick operation FH is performed from the left direction to the right direction (horizontal direction) on the touch panel 83a while the composite image CPb1 (FIG. 19) is displayed. By referring to the composite image CPb2, the user can confirm the situation behind the vehicle 2 from an angle different from that of the composite image CPb1.

In FIG. 21, after the combined image CPb1 (FIG. 19) is displayed, the reference virtual viewpoint VPab is set by moving counterclockwise on OBlh in FIG. 12, and the vehicle 2 is viewed from the set reference virtual viewpoint VPab. The image CPb3 is an example displayed on the display 83. The composite image CPb3 displays the rear area of the vehicle 2 in more detail than the composite image CPb1. The composite image CPb3 is displayed when the flick operation FV is performed from the upper direction to the lower direction (vertical direction) on the touch panel 83a while the composite image CPb1 (FIG. 16) is displayed. The user can check the situation behind the vehicle 2 in detail by referring to the composite image CPb3.

FIG. 22 is an example in which the reference virtual viewpoint VPal is set to the virtual viewpoint VP, and the composite image CPl1 in which the vehicle 2 is viewed from the reference virtual viewpoint VPal is displayed on the display 83. The composite image CPl1 displays the vehicle body image 10 in addition to the peripheral image showing the left side of the vehicle 2. The composite image CPl1 is displayed when the user touch-operates the touch panel button TB22 indicating the left side of the vehicle 2. The user can check the situation on the left side of the vehicle 2 by referring to the composite image CPl1.

23 is a composite image in which the vehicle 2 is viewed from the set reference virtual viewpoint VPal by moving the OBlh in FIG. 9 in the clockwise direction after displaying the composite image CPl1 (FIG. 22). CPb2 is an example displayed on the display 83. The composite image CP12 displays the vehicle body image 10 in addition to the peripheral image showing the left side of the vehicle 2 displayed by moving to the left when the user views the display 83. The composite image CP12 is displayed when the flick operation FH is performed from the left direction to the right direction (horizontal direction) on the touch panel 83a in a state where the composite image CP12 (FIG. 22) is displayed. By referring to the composite image CPl2, the user can confirm the situation on the left side of the vehicle 2 from an angle different from that of the composite image CPl1.

In FIG. 24, after the composite image CPl1 (FIG. 22) is displayed, the reference virtual viewpoint VPal is set by moving counterclockwise on OBlh in FIG. 12, and the vehicle 2 is viewed from the set reference virtual viewpoint VPal. The image CPl3 is an example displayed on the display 83. The composite image CPl3 displays the left area of the vehicle 2 in more detail than the composite image CPl1. The combined image CPl3 is displayed when the combined image CPl1 (FIG. 22) is displayed and a flick operation FV is performed on the touch panel 83a from the upper direction to the lower direction (vertical direction). The user can check the situation on the left side of the vehicle 2 in detail by referring to the composite image CP13.

The synthetic image CP showing the right side of the vehicle 2 may be operated and controlled symmetrically with respect to the case of the left side of the vehicle 2 described with reference to FIGS. 22 to 24. By referring to the composite image CP showing the right side of the vehicle 2, the user can confirm the situation on the right side of the vehicle 2.

<1-6. Processing>
Next, a processing procedure of the image display system 1 will be described with reference to FIGS. 25 and 26. FIG. 25 shows a processing procedure of the image acquisition device 3, the security device 5, and the mobile terminal 8. FIG. 26 shows a processing procedure of the alarm device 6 and the mobile terminal 8. This process is repeatedly executed at a predetermined cycle.

First, the security device 5 determines whether or not the monitoring sensor 5a has operated (step S101 in FIG. 25).

When the security device 5 determines that the monitoring sensor 5a has operated (Yes in step S101), the security device 5 controls the mail notification unit 5b to indicate to the mobile terminal 8 of the user that the monitoring sensor 5a has operated, that is, the vehicle 2 has stolen it. An e-mail having the content of the occurrence of a prior phenomenon leading to is transmitted (step S102). In addition, when the monitoring sensor 5a operates, when the monitoring sensor 5a detects the occurrence of vibration or inclination in the vehicle 2, when the glass is broken, or when an intruder into the vehicle 2 is detected, etc. Is.

If the security device 5 determines that the monitoring sensor 5a is not operating (No in step S101), and if the mail notification unit 5b sends an e-mail, the processing of the security device 5 ends and the predetermined time elapses. After the lapse of time, the process is started again.

When the communication unit 82 of the mobile terminal 8 receives the email sent from the email notification unit 5b, the display control unit 81c displays the content of the email on the display 83 (step S201). The user confirms the content of the email displayed on the display and determines whether or not to display the peripheral image of the vehicle 2. When the user desires to display a peripheral image of the vehicle 2, the user may perform a touch operation on a predetermined position on the touch panel 83a.

The operation detection unit 81d determines whether or not the user has performed a touch operation on a predetermined position on the touch panel 83a where the peripheral image should be displayed (step S202).

When the operation detection unit 81d performs a touch operation on a predetermined position on the touch panel 83a where the user should display the peripheral image and determines that the user desires to display the image (Yes in step S202), the notification unit 81e causes the image acquisition device 3 to operate. A signal requesting transmission of an image is transmitted (step S203).

On the other hand, when the operation detection unit 81d performs a touch operation on the touch panel 83a by the user at a predetermined position indicating that the peripheral image is not displayed, that is, when it is determined that the image display is not desired (No in step S202), the process ends. To do. This is because it is not necessary to continue the process unless the user wants to display an image.

When the request receiving unit 31a of the image acquisition device 3 receives a signal requesting the transmission of an image from the mobile terminal 8, the camera control unit 31b controls the cameras 4 (4F, 4B, 4L, 4R) to start shooting. (Step S301).

When the image acquisition unit 31c receives the image data captured by the camera 4 (4F, 4B, 4L, 4R), the image transmission unit 31d transmits the image data to the mobile terminal 8 via the communication unit 32 (step S302). ..

When the image acquisition unit 81a of the mobile terminal 8 receives the image data transmitted from the image acquisition device 3, the image generation unit 81b sets the position and direction of the virtual viewpoint for generating the peripheral image. (Step S204). When generating the peripheral image for the first time, the image generation unit 81b sets the position of the virtual viewpoint to be directly above the vehicle 2 and sets the direction of the virtual viewpoint to be downward (top view). The viewpoint is such that the user looks downward from the position directly above the vehicle 2. Such a viewpoint is preferable as the position and direction of the virtual viewpoint that should be first displayed to the user. Since the entire circumference of the vehicle is displayed, it is possible to inform the user of a wide range of situations.

When the virtual viewpoint is set, the image generation unit 81b generates the peripheral image by the method described above. Then, the vehicle body image data 84a is read from the storage unit 84, and a synthesized image in which the vehicle image is synthesized with the generated peripheral image is generated (step S205).

When the image generation unit 81b generates the composite image, the display control unit 81c displays the composite image on the display 83 (step S206). This allows the user to check the state around the vehicle 2.

When the display control unit 81c displays the composite image on the display 83, the operation detection unit 81d determines whether the user has performed a flick operation on the display 83 (step S207).

When the operation detection unit 81d determines that the user has performed the flick operation (Yes in step S207), the image generation unit 81b sets the virtual viewpoint again (step S204).

At this time, when the user performs a flick operation in the left-right direction (horizontal direction) on the display 83, as described above, the virtual viewpoint is rotated based on the vertical axis of the world coordinate system to set the position of the virtual viewpoint. To do.

Furthermore, when the flick operation is an operation from the right side to the left side with respect to the display 83, the position of the virtual viewpoint becomes so-called counterclockwise when the vertical axis of the world coordinate system is viewed from above. To rotate. When the flick operation is an operation from the left side to the right side of the display 83, the position of the virtual viewpoint is rotated so as to be so-called clockwise when the vertical axis of the world coordinate system is viewed from above.

On the other hand, when the user performs a flick operation in the up-down direction (vertical direction) on the display 83, as described above, the virtual viewpoint is rotated around the horizontal axis of the viewpoint coordinate system to set the position of the virtual viewpoint. ..

Furthermore, when the flick operation is an operation from the upper side to the lower side with respect to the display 83, the virtual viewpoint is so-called counterclockwise when viewed from the right side of the horizontal axis of the viewpoint coordinate system. Rotate the viewpoint position. Further, when the flick operation is an operation from the lower side to the upper side with respect to the display 83, the position of the virtual viewpoint is set so as to be so-called clockwise when the horizontal axis of the viewpoint coordinate system is viewed from the right of the viewpoint. Rotate.

With such a method of rotating the position of the virtual viewpoint, the direction in which the user wants to move the viewpoint by flicking matches the direction in which the displayed image moves, and the user intuitively operates the touch panel 83a. The image can be moved by.

On the other hand, when the operation detection unit 81d determines that the flick operation is not performed by the user (No in step S207), the operation detection unit 81d determines whether or not there is an operation for changing the virtual viewpoint to the reference viewpoint (step S208). The operation detection unit 81d determines whether or not the user has performed a touch operation on any touch panel button indicating the reference viewpoint.

When the operation detection unit 81d determines that there is an operation for changing the virtual viewpoint to the reference viewpoint (Yes in step S208), the image generation unit 81b sets the position and direction of the virtual viewpoint to the reference viewpoint desired by the user by touch operation. (Step S204). For example, when the user desires the reference viewpoint for displaying the left area of the vehicle 2, the image generation unit 81b sets the position and direction of the virtual viewpoint as shown in FIGS. 9 and 10. The reference viewpoints are five viewpoints immediately above the vehicle, the left side of the vehicle 2, the right side of the vehicle 2, the front side of the vehicle 2, and the rear side of the vehicle 2. The position of the movement center point RC, which is the specific point set at the specific position of the vehicle, is set at a position according to the reference viewpoint desired by the user.

On the other hand, if the operation detecting unit 81d determines that there is no operation for changing the virtual viewpoint to the reference viewpoint (No in step S208), it determines whether the user is trying to activate the alarm device 6 provided in the vehicle 2 ( Step S209). The operation detection unit 81d determines whether or not the user has performed a touch operation on a predetermined position that indicates to activate the alarm device on the touch panel 83a.

When the operation detection unit 81d determines that the user intends to activate the alarm device 6 (Yes in step S209), the notification unit 81e transmits a signal requesting the image acquisition device 3 to activate the alarm device 6 ( Step S210).

When the notification unit 81e transmits a signal requesting to activate the alarm device 6, the anti-theft unit 31e of the image acquisition device 3 causes the alarm device 6 to issue an alarm (step S401). The alarm by the alarm device 6 ends after a predetermined time has elapsed. The predetermined time may be a time sufficient to give a warning to a suspicious person. For example, 5 seconds. In the first place, the user may operate the end time of the alarm.

When the operation detection unit 81d determines that the user is not trying to activate the alarm device 6 (No in step S209) and when the anti-theft unit 31e causes the alarm device 6 to issue an alarm (step S210), the operation is performed. The detection unit 81d determines whether or not the user is about to end the image display (step S211).

When the operation detection unit 81d determines that the user is about to end the image display (Yes in step S211), the notification unit 81e transmits a signal requesting the image acquisition device 3 to end the image capturing (step S212). ). When the notification unit 81e transmits a signal requesting the image acquisition device 3 to end shooting, the process of the mobile terminal 8 ends.

On the other hand, when the operation detection unit 81d determines that the user is not trying to end the image display (No in step S211), the display control unit 81c continues to generate and display the image, and the process is performed again from step S205. To do.

When the image transmitting unit 31d transmits the image data in step S302, the image acquisition device 3 determines whether the request receiving unit 31a has transmitted from the mobile terminal 8 an end request signal requesting to end the image capturing (step S303). ). The image acquisition device 3 repeatedly transmits the image data most recently obtained by the camera 4 (4F, 4B, 4L, 4R) to the mobile terminal 8 until receiving the end request signal. As a result, the image generation unit 81b of the mobile terminal 8 can generate a peripheral image that shows a state around the vehicle 2 in substantially real time based on the image data obtained most recently.

When the request receiving unit 31a determines that the end request signal requesting the end of shooting is transmitted from the mobile terminal 8 (Yes in step S303), the camera control unit 31b controls the camera 4 (4F, 4B, 4L, 4R). It controls and stops photographing (step S304). When the camera control unit 31b controls the cameras 4 (4F, 4B, 4L, 4R) and stops the shooting, the process of the image acquisition device 3 ends.

As described above, in the first embodiment, the position of the virtual viewpoint is changed based on the user's operation to display the composite image. Accordingly, the subject can be easily displayed from the viewpoint desired by the user.

<2. Second Embodiment>
<2-1. Overview>
Next, a second embodiment will be described. In the above-described first embodiment, when the viewpoint position VL is directly above the vehicle 2 and the viewpoint direction VD is downward (−Z side) (that is, the top view), the user's left-right direction (horizontal direction). ) The movement of the viewpoint position VL with respect to the flick operation is based on the vertical axis (Z axis) of the orthogonal coordinate system cc. As a result, the composite image CP is displayed rotated to the left or right.

However, rotating and displaying the composite image CP does not present a new peripheral region to the user.

Therefore, in the second embodiment, the movement of the viewpoint position VL in the top view is based on the axis (Y axis) which is the front and rear of the vehicle 2 in the orthogonal coordinate system cc. As a result, when the user performs a flick operation in the left-right direction (horizontal direction) on the touch panel 83a in the top view, the left and right regions of the vehicle 2 including the side surface of the vehicle 2 are displayed. You can check the left and right areas in detail.

Since the second embodiment includes the same configuration and processing as those of the first embodiment, the differences from the first embodiment will be mainly described below.

<2-2. Composition>
First, the configuration of the mobile terminal 8 according to the second embodiment will be described. FIG. 27 shows the configuration of the mobile terminal 8 according to the second embodiment. The main difference from the first embodiment is that the control unit 81 of the mobile terminal 8 includes a coordinate axis conversion unit 81f. Other configurations are the same as those of the first embodiment and function similarly.

When the virtual viewpoint is the top view, the coordinate axis conversion unit 81f converts the coordinate axis serving as a reference for moving the viewpoint position VL from the vertical axis (Z axis) of the orthogonal coordinate system cc to the vehicle front-rear axis (Y axis).

<2-3. Setting virtual viewpoint>
Next, a method of moving the viewpoint position VL in the left-right direction (horizontal direction) when the user views the display in the top view display will be described with reference to FIGS. 28 and 29. The movement of the viewpoint position VL described below is performed in response to a flick operation in the left-right direction (horizontal direction) on the touch panel by the user in the top view display.

FIG. 28 is a diagram of the vehicle 2 viewed from the vertically upward direction (+Z side), and shows a trajectory along which the reference position VLt of the reference virtual viewpoint VPat moves. The reference position VLt moves along the movement locus OBth around the movement center point RCt. The movement of the viewpoint position VLt is based on the orthogonal coordinate system cc relatively fixed to the virtual viewpoint VP with the position of the movement center point RCt as the origin. The coordinate axis that serves as a reference for movement of the viewpoint position VLt is the Y axis (AXth) of the orthogonal coordinate system cc at the position of the movement center point RCt. That is, it is an axis along the front-rear direction of the vehicle 2.

FIG. 29 is a diagram of the vehicle 2 viewed from the rear side (−Y side) in the horizontal direction, and shows a trajectory along which the reference position VLt of the reference virtual viewpoint VPat moves, similarly to FIG. 28. The reference position VLt moves along the movement locus OBth around the movement center point RCt. The movement trajectory OBth has a circular shape when viewed from the rear side (−Y side) in the horizontal direction. The coordinate axis on which the viewpoint position VLl moves is the Y axis of the orthogonal coordinate system cc at the position of the movement center point RCt.

28 and 29, the viewpoint direction VDt is directed from the viewpoint position VLt to the movement center point RCt wherever the viewpoint position VLt is located on the movement trajectory OBth. Therefore, when the reference position VLt moves on the movement locus OBth, the user looking at the display 83 appears to move the image in the left-right direction (horizontal direction). In particular, since the coordinate axis that serves as the reference for moving the viewpoint position VLl is the Y axis of the orthogonal coordinate system cc at the position of the movement center point RCt, the user refers to the image around the vehicle 2 by turning around the side surface of the vehicle 2. can do.

The movement locus OBth is not set below the ground contact surface GR of the vehicle 2. Therefore, the reference position VLt moves along the movement locus OBth indicated by the broken line, and the movement locus OBtx indicated by the solid line does not move.

When the user performs a flick operation on the touch panel 83a from the left direction to the right direction (horizontal direction), the VLt moves counterclockwise on the movement track OBth. On the other hand, when the user performs a flick operation on the touch panel 83a from the left direction to the right direction (horizontal direction), the VLt moves clockwise on the movement track OBth. As a result, the direction of the user's flick operation matches the direction of movement of the image, and the user can intuitively perform the touch panel operation.

<2-4. Example of displaying composite image>
The upper diagram of FIG. 30 is an example in which the reference virtual viewpoint VPat is set to the virtual viewpoint VP, and the composite image CPt1 of the vehicle 2 viewed from the reference virtual viewpoint VPat is displayed on the display 83. That is, the composite image CPt1 is a composite image when the virtual viewpoint is the top view.

The lower diagram of FIG. 30 shows a composite image CPt4 when the coordinate axis for moving the viewpoint position VL is converted from the vertical axis (Z axis) of the orthogonal coordinate system cc to the vehicle longitudinal axis (Y axis) after the display of the composite image CPt1. is there. That is, this is an example in which the reference virtual viewpoint VPat is set by moving it counterclockwise on OBth in FIG. 29, and the composite image CPt4 in which the vehicle 2 is viewed from the set reference virtual viewpoint VPat is displayed on the display 83. .. The composite image CPt4 displays the vehicle body image 10 in addition to the peripheral image of the vehicle 2 including the side surface of the vehicle 2. Therefore, the user can check the left and right regions of the vehicle 2 in detail from the top view. The composite image CPt4 is displayed when the flick operation FH is performed from the left direction to the right direction (horizontal direction) on the touch panel 83a while the composite image CPt1 is displayed. By referring to the composite image CPt2, the user can check the left and right regions of the vehicle 2 including the side surface of the vehicle 2 in detail.

<2-5. Processing>
Next, a processing procedure in the second embodiment will be described with reference to FIGS. 31 and 32. FIG. 31 shows the processing procedure of the image acquisition device 3, the security device 5, and the mobile terminal 8 in the second embodiment. The difference from the processing procedure in the first embodiment shown in FIG. 25 is that the processing of step S209 is provided.

First, the image generation unit 81b sets the position and direction of the virtual viewpoint for generating the peripheral image. (Step S204). When generating the peripheral image for the first time, the image generation unit 81b sets the position of the virtual viewpoint to be directly above the vehicle 2 and sets the direction of the virtual viewpoint to be downward (top view). Further, the coordinate axis conversion unit 81f sets the axis serving as a reference for moving the viewpoint position VL to the vehicle front-rear axis (Y axis).

When step S205, step S206, and step S207 are executed, the operation detection unit 81d determines whether or not there is an operation for changing the virtual viewpoint to the reference viewpoint (step S208). The operation detection unit 81d determines whether or not the user has performed a touch operation on any touch panel button indicating the reference viewpoint.

When the operation detection unit 81d determines that there is an operation for changing the virtual viewpoint to the reference viewpoint (Yes in step S208), the image generation unit 81b executes the coordinate axis setting process (step S209).

FIG. 32 shows details of the coordinate axis setting process (step S209). When the process proceeds to the coordinate axis setting process (step S209), the operation detection unit 81d determines whether the virtual viewpoint has been changed to the top view viewpoint (step S401).

When it is determined that the virtual viewpoint has been changed to the top-view viewpoint (Yes in step S401), the coordinate axis conversion unit 81f sets the coordinate axis that is the reference for moving the viewpoint position VL to the vehicle front-rear axis (Y axis) (step S402). ).

On the other hand, when determining that the virtual viewpoint has been changed to a viewpoint other than the top view (No in step S401), the coordinate axis conversion unit 81f sets the coordinate axis serving as the reference for moving the viewpoint position VL to the vertical axis (Z axis) ( Step S403).

When step S402 or step S403 is executed, the process returns to FIG. 31, and the processes after step S204 are executed again.

As described above, in the second embodiment, the movement of the viewpoint position VL in the top view is based on the axis (Y axis) that is the front and rear of the vehicle 2 in the orthogonal coordinate system cc. As a result, when the user performs a flick operation in the left-right direction (horizontal direction) on the touch panel 83a in the top view, the left and right regions of the vehicle 2 including the side surface of the vehicle 2 are displayed. You can check the left and right areas in detail.

<3. Modification>
The embodiments of the present invention have been described above. However, the present invention is not limited to the above embodiment. Various modifications are possible. Hereinafter, modified examples will be described. The above and the following embodiments can be combined as appropriate.

In the above-described embodiment, the image acquisition device 3 starts shooting when receiving the image transmission request from the user. However, shooting may be started before the user receives a request for image transmission. That is, the photographing may be started when the monitoring sensor 5a is activated. In this case, the user can refer to the situation around the vehicle 2 from the time when the abnormality of the vehicle 2 occurs such that the monitoring sensor 5a operates.

FIG. 33 shows an outline of the image display system 1a in which the image acquisition device 3 starts photographing when the monitoring sensor 5a operates. When the monitoring sensor 5a detects the abnormality of the vehicle 2, the security device 5 transmits a signal requesting the start of shooting to the image acquisition device 3. Upon receiving such a signal, the image acquisition device 3 activates the camera 4 to start photographing. That is, the image acquisition device 3 does not wait for a request for image transmission from the user, and starts capturing an image when an abnormality occurs in the vehicle 2. When the image acquisition device 3 starts photographing, the image acquisition device 3 transmits the image data to the server SV installed outside. As a result, the image data is stored in the server SV from the time when the abnormality of the vehicle 2 occurs. Therefore, when the user receives the email notifying of the abnormality occurrence of the vehicle 2 from the security device 5, the user requests the server SV to transmit the image. Upon receiving the image transmission request, the server SV transmits the image data from the time when the abnormality of the vehicle 2 occurs to the mobile terminal 8 owned by the user. Accordingly, the user can refer to the state around the vehicle 2 when the monitoring sensor 5a is activated, that is, when the abnormality of the vehicle 2 occurs. Further, if the server SV is a dedicated server for transmitting and receiving image data, there is no fear that the image data will be intercepted by others, and the confidentiality of the image data can be enhanced. When the user wants to confirm the current surroundings of the vehicle 2, the user may request the server SV to transmit the current image data. In this case, the server SV omits the transmission of the image data from the time when the abnormality occurs until the present, and transmits the current image data.

Another modification will be described. In the above-mentioned embodiment, it was explained that the image acquisition device 3, the camera 4, the security device 5, and the alarm device 6 are mounted on the vehicle 2. However, it may not be the vehicle 2. It may be a house or building. In short, it can be any land, building or article to be monitored.

Further, in the above-described embodiment, the touch panel has been described as a user performing an input operation. However, it does not have to be a touch panel. A push button type switch may be used. For example, a so-called cross key. In short, it suffices if the user's input operation can be determined.

In the above embodiment, the image acquisition device 3 and the mobile terminal 8 are described as separate devices. However, the image acquisition device 3 and the mobile terminal 8 may be configured as an integrated device.

Further, in the above embodiment, it has been described that each function is realized as software according to a program. However, it need not be software. It may be realized as an electrical hardware circuit.

1 image display system 2 vehicle 3 image acquisition device 4 camera 5 security device 6 alarm device 7 network 8 mobile terminal

Claims (3)

An image display device for generating a composite image including a vehicle body image using a captured image captured by a camera, and displaying the generated composite image,
Display control for displaying, on a display device, a composite image viewed from the virtual viewpoint associated with the one button when the selection of one of the plurality of buttons associated with each of the plurality of virtual viewpoints is accepted. Means and
An operation detection unit that detects a slide operation or a pinch operation, which is a touch operation on the composite image viewed from the virtual viewpoint,
When the touch operation is detected by the operation detection unit after the selection of the one button, the display control unit changes the composite image viewed from the virtual viewpoint based on the selection of the one button according to the touch operation. An image display device for displaying. The image display device according to claim 1, wherein the plurality of buttons include a button for selecting a virtual viewpoint for initial display and buttons corresponding to front, rear, left, and right of a vehicle equipped with the camera. A terminal device that displays a composite image including a vehicle body image, which is generated using a captured image captured by a camera,
Display control for displaying, on a display device, a composite image viewed from the virtual viewpoint associated with the one button when the selection of one of the plurality of buttons associated with each of the plurality of virtual viewpoints is accepted. Means and
An operation detection unit that detects a slide operation or a pinch operation, which is a touch operation on the composite image viewed from the virtual viewpoint,
When the touch operation is detected by the operation detection unit after the selection of the one button, the display control unit changes the composite image viewed from the virtual viewpoint based on the selection of the one button according to the touch operation. And display the terminal device.