JP6831504B2 - Terminals and terminal control methods - Google Patents

Description

The present invention relates to a terminal and a method for controlling the terminal.

Conventionally, a terminal (portable terminal 4) that communicates with an information processing device (vehicle side device 3) provided in a vehicle and displays an image based on a shooting result of a camera (imaging unit 5) provided in the vehicle has been known. (See, for example, Patent Document 1).

JP-A-2015-54530

In the case of a terminal capable of displaying an image based on a shooting result of a camera provided in a vehicle such as the terminal described above, there is a need to improve user convenience by performing appropriate processing when displaying the image.
The present invention has been made in view of the above circumstances, and an object of the present invention is to improve user convenience when displaying an image based on a shooting result of a camera provided in a vehicle on a terminal.

In order to achieve the above object, the terminal of the present invention includes a housing, a display panel provided on the housing, a housing state detection unit for detecting the state of the housing, and a camera provided on the vehicle. The shooting data based on the shooting result of the above is acquired, and the image based on the acquired shooting data and the touch-operable operation button are displayed on the display panel, and the housing detected by the housing state detection unit is displayed. The terminal control unit includes a terminal control unit that switches an image to be displayed on the display panel according to the state of the display panel, and a touch sensor that detects an operation position with respect to the display panel. The terminal control unit detects the image by the housing state detection unit. When the image to be displayed on the display panel is switched according to the state of the housing, the display position of the operation button is detected by the touch sensor when the image to be displayed on the display panel is switched. It is characterized in that it is set to the touch position .

Further, in the present invention, the terminal control unit detects that the orientation of the housing is the first orientation by the housing state detection unit, and the orientation of the housing is the first. It is characterized in that the image to be displayed on the display panel is switched between the case where it is detected that the second orientation is different from the orientation.

Further, in the present invention, the housing is a plate-shaped member having a rectangular front surface, the display panel is provided on the front surface of the housing, and the first orientation is the front surface of the housing. The direction of the long side of the housing is the direction along the vertical direction, and the second direction is the direction in which the direction of the long side of the front surface of the housing is along the direction orthogonal to the vertical direction. It is characterized by being oriented.

Further, in the present invention, when the terminal control unit detects that the orientation of the housing is the first orientation by the housing state detection unit, the terminal control unit uses the vehicle as an image based on the shooting data. When a bird's-eye view image, which is a bird's-eye view image, is displayed on the display panel and the housing state detection unit detects that the housing is oriented in the second orientation, the image is based on the shooting data. The display panel is characterized in that an image of the outside of the vehicle, which is an image of the outside of the vehicle taken by the camera, is displayed on the display panel.

Further, in the present invention, the vehicle is provided with a plurality of the cameras for photographing the outside of the vehicle, and the terminal control unit has the housing oriented to the second orientation by the housing state detecting unit. When it is detected, one of the cameras is selected from the plurality of cameras according to a predetermined priority corresponding to the state of the vehicle, and the outside image based on the shooting result of the selected camera is displayed. It is characterized by being displayed on a panel.

Further, the present invention is characterized in that the predetermined priority is a higher priority as the camera capable of photographing the traveling direction of the vehicle.

Further, the present invention is characterized in that the predetermined priority is a higher priority as the camera capable of photographing an object close to the vehicle.

Further, in the present invention, after the terminal control unit displays the outside image of 1 based on the shooting result of the camera on the display panel, the housing state detection unit moves the housing to a predetermined position. When it is detected that the above-mentioned image has been detected, the display panel is displayed with another image of the outside of the vehicle based on the shooting result of the other camera different from the camera of the first one, instead of the image of the outside of the vehicle of the one. It is a feature.

Further, according to the present invention, the plurality of the cameras take pictures of the outside of the vehicle in different directions, and the predetermined movement is a first movement in which the state of the housing changes in the first aspect, or The state of the housing is one of the second movements in which the state of the housing changes in a second aspect different from the first aspect, and the terminal control unit uses the housing state detection unit to cause the housing to change. When it is detected that the movement of 1 is performed, the images outside the vehicle are switched in the first order according to the direction in which the camera takes a picture, and the housing makes the second movement by the housing state detection unit. When this is detected, the outside image is switched in a second order different from the first order, which is an order according to the shooting direction of the camera.

The front SL terminal control unit of the present invention, together to display an image based on the imaging data to the display panel, so as to overlap with the image based on the captured data, to display the touch operation can be the operation buttons, the Even when the image to be displayed on the display panel is switched, the display of the operation button is maintained, and while the operation button is touch-operated, the information processing device provided in the vehicle is instructed to propel the vehicle. It is characterized by transmitting a propulsion instruction signal.
Further, the terminal control unit of the present invention causes the display panel to display a slide button that is moved by a touch operation as the operation button, and also displays a slide bar that clearly indicates the range in which the slide button can be moved. An image based on the shooting data according to the state of the housing detected by the housing state detection unit during the period in which the slide button is maintained in a state of being moved from one end to the other end of the slide bar. When the image is switched and displayed on the display panel, another operation button that causes the touch position of the slide button detected by the touch position detection unit to function as an operation button different from the slide button when the image is switched. Is displayed, and the position of the other operation button is moved following the movement of the touch position detected by the touch position detecting unit.

Further, the present invention is a method for controlling a terminal having a housing and a display panel provided in the housing, and acquires and acquires shooting data based on the shooting results of a camera provided in the vehicle. An image based on the shooting data and a touch-operable operation button are displayed on the display panel, the state of the housing is detected, and an image displayed on the display panel according to the detected state of the housing. When switching the image to be displayed on the display panel according to the detected state of the housing by performing the process of switching and the process of detecting the operation position with respect to the display panel, the display position of the operation button is changed to the above. It is characterized in that it is set to the touch position detected when switching the image to be displayed on the display panel .

According to the present invention, it is possible to improve the convenience of the user when displaying an image based on the shooting result of a camera provided in the vehicle on the terminal.

The figure which shows the outline of the information processing system 1 which concerns on this embodiment. A block diagram showing a functional configuration of a mobile terminal. The figure used for the vertical explanation. The figure used for the sideways explanation. The figure used to explain the right movement. The figure used to explain the left movement. The block diagram which shows the functional structure of an information processing apparatus. The figure used to explain the shooting direction of the camera for shooting outside the vehicle. The block diagram which shows the functional structure of the vehicle drive control device. The figure used for explaining the automatic driving route information. A flowchart showing the operation of each device of the information processing system. The figure which shows an example of a parking lot. The figure which shows the lock screen. The figure used for explaining the screen displayed on a touch panel. A flowchart showing the operation of a mobile terminal. A flowchart showing the operation of the information processing device. A flowchart showing the operation of the information processing device. The figure which shows the composite image. A flowchart showing the operation of a mobile terminal. A flowchart showing the operation of the information processing device. A flowchart showing the operation of the information processing device. A flowchart showing the operation of the information processing device. A flowchart showing the operation of each device of the information processing system. The figure which shows the selection screen. The figure used for explaining the screen displayed on a touch panel. The figure used for explaining the screen displayed on a touch panel. The figure used for explaining the screen displayed on a touch panel.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing an outline of the information processing system 1 according to the present embodiment.
As shown in FIG. 1, the information processing system 1 includes a vehicle system 2 provided in the vehicle S and a mobile terminal 3 owned by the user.
The vehicle system 2 includes an information processing device 4 and a vehicle drive control device 5. The information processing device 4 and the vehicle drive control device 5 can communicate with each other via the bus 6.
The mobile terminal 3 is a tablet-type mobile phone (so-called smartphone) carried by the user. The information processing device 4 of the vehicle system 2 and the mobile terminal 3 can communicate with each other by wireless communication. The mobile terminal 3 includes a housing 8. The housing 8 is a plate-shaped member having a rectangular front surface 8a, and a touch panel 9 (display panel) is provided in a wide area of the front surface 8a. The rectangular shape does not mean a perfect rectangular shape, and includes, for example, a substantially rectangular shape in which four vertices are rounded. Hereinafter, the embodiment will be described assuming that the mobile terminal 3 is a tablet-type mobile phone, but the mobile terminal 3 is not limited to a smartphone and may be a tablet-type computer or the like.
The user is a person who has the authority to control the vehicle S, such as the owner of the vehicle S.
The vehicle S is a four-wheeled vehicle. In particular, the vehicle S according to the present embodiment has the following functions regarding on / off of the engine switch and locking / unlocking of the door. That is, the vehicle S has a function of turning on / off the engine switch and locking / unlocking the door by operating the dedicated remote controller. Since the existing technology is used for the function, a detailed description of the function will be omitted. After getting off the vehicle S, the user can operate a dedicated remote controller outside the vehicle to turn on / off the engine switch, and can lock / unlock the door of the vehicle S.

The information processing system 1 is a system that enables the following. That is, the information processing system 1 enables the user to operate the mobile terminal 3 to move the vehicle S stopped at one position to another position and stop the vehicle S without driving the vehicle S. .. For example, when the information processing system 1 parks the vehicle S at a parking position in a parking lot, a garage, or the like, the user operates the mobile terminal 3 without driving the vehicle S to park in the vicinity of the parking position. It is possible to move the vehicle S stopped at the front position to the parking position and stop the vehicle S. Further, for example, in the information processing system 1, when the vehicle S parked at the parking position is discharged, the user operates the mobile terminal 3 without driving the vehicle S to park the vehicle at the parking position. It is possible to move S to a stop position after leaving the garage near the parking position and stop the vehicle. Therefore, by using the information processing system 1, the user can park and leave the vehicle S without being on the vehicle S. Therefore, regarding parking, if another vehicle is already parked at a parking position adjacent to the parking position desired to be parked and the vehicle S is parked at the parking position desired to be parked, the vehicle S and another adjacent vehicle S are parked. When the vehicle is in close proximity and it is difficult to open and close the door of the vehicle S, the user can park the vehicle S in the parking position in the following manner. That is, the user can disembark the vehicle S before stopping the vehicle S at the parking position and use the information processing system 1 to stop the vehicle S at the parking position. As a result, it is not necessary to open and close the door of the vehicle S after the vehicle S has stopped at the parking position, and the user can smoothly park the vehicle S. Similarly, regarding leaving the garage, another vehicle is already parked at a parking position adjacent to the parking position where the vehicle S is parked, and it is difficult to open and close the door of the vehicle S to board the vehicle S. In this case, by using the information processing system 1, the user can smoothly leave the vehicle S without boarding the vehicle S and then board the vehicle S.
In the following description, the information processing system 1 indicates that the vehicle S stopped at one position moves to another position and stops at another position without the user driving the vehicle S. Expressed as "move".

FIG. 2 is a block diagram showing a functional configuration of the mobile terminal 3.
As shown in FIG. 2, the mobile terminal 3 includes a terminal control unit 20, a terminal communication unit 21, a touch panel 9, a terminal storage unit 22, and a housing state detection unit 23.

The terminal control unit 20 includes a CPU, ROM, RAM, ASIC, signal processing circuit, and the like, and controls each unit of the mobile terminal 3. In the terminal control unit 20, for example, the CPU reads the program stored in the ROM into the RAM and executes the processing, executes the processing by the function implemented in the ASIC, for example, and performs the signal processing in the signal processing circuit, for example. Processes are executed in collaboration with hardware and software.

The terminal communication unit 21 wirelessly communicates with an external device (including the information processing device 4) according to the Wi-Fi (registered trademark) standard under the control of the terminal control unit 20.
In the present embodiment, the communication protocol used for wireless communication between the mobile terminal 3 and the information processing device 4 is Wi-Fi, but the communication used for wireless communication between these devices is used. The protocol is not limited to Wi-Fi and may be anything. For example, the communication protocol may be Bluetooth®.

The touch panel 9 includes a display panel such as a liquid crystal panel or an organic EL panel, and displays an image on the display panel under the control of the terminal control unit 20. Further, the touch panel 9 includes a touch sensor arranged so as to be superimposed on the display panel, detects an operation on the touch panel 9 by the user, and outputs a signal corresponding to the detected operation to the terminal control unit 20. The terminal control unit 20 executes a process corresponding to an operation on the touch panel 9 by the user based on the input from the touch panel 9.

The terminal storage unit 22 includes a non-volatile memory such as EEPROM, and stores various data in a rewritable non-volatile manner.
The terminal storage unit 22 stores the dedicated application AP. As will be described later, the dedicated application AP is an application used when a user uses the information processing system 1 to automatically move a vehicle. The dedicated application AP is, for example, an application provided by a company that manufactures and sells the information processing device 4. The user downloads the dedicated application AP to the mobile terminal 3 in advance using a predetermined application download system.

The housing state detection unit 23 includes an acceleration sensor, a gyro sensor, and an inclination sensor provided in the housing 8, and detects the state of the housing 8 based on the detection values of these sensors. The housing state detection unit 23 outputs information indicating the detected state of the housing 8 to the terminal control unit 20. The sensor used by the housing state detection unit 23 to detect the state of the housing 8 is not limited to the acceleration sensor, the gyro sensor, and the tilt sensor. Further, any method may be used when the housing state detection unit 23 detects the state of the housing 8.
The housing state detection unit 23 has at least (1) the orientation of the housing 8 being "vertical" and (2) the orientation of the housing 8 being "horizontal" as the state of the housing 8. That, (3) when the orientation of the housing 8 is "sideways", the housing 8 "moves to the right", and (4) when the orientation of the housing is "sideways", the housing 8 is "sideways". Detects "left movement".

FIG. 3A is a diagram used for explaining "portrait".
Here, as described above, the housing 8 is a plate-shaped member having a rectangular front surface 8a. Further, as shown in FIG. 3A, the shape of the touch panel 9 is a rectangular shape having a long side along the long side T1 of the housing 8 and a short side along the short side T2 of the housing 8.
Since the front surface 8a is rectangular, the outer edge of the front surface 8a is composed of a long side T1 and a short side T2. Based on this, hereinafter, a virtual line extending along the long side T1 of the housing 8 is expressed as a "virtual long side straight line", and "KA" is added as a code. Further, a virtual line extending in the vertical direction is expressed as a "virtual vertical straight line", and "EN" is added as a code. Further, a virtual line orthogonal to the virtual vertical straight line EN is expressed as a "virtual orthogonal straight line", and "TN" is added as a code.
As shown in FIG. 3A, "vertical orientation" is a state in which the angle of the virtual long side straight line KA with respect to the virtual vertical straight line EN in the three-dimensional space is within the range of the angle θ1. The value of the angle θ1 is appropriately determined from the viewpoint of the margin when the user intentionally sets the orientation of the mobile terminal 3 to “vertical orientation”. When the orientation of the mobile terminal 3 is "vertical", the direction of the long side T1 of the housing 8 is along the vertical direction, and the direction of the long side of the touch panel 9 is along the vertical direction. Become.
"Vertical orientation" corresponds to "first orientation".

FIG. 3B is a diagram used for the explanation of “sideways”.
As shown in FIG. 3B, "horizontal orientation" is a state in which the angle of the virtual long side straight line KA with respect to the virtual orthogonal straight line TN in the three-dimensional space is within the range of the angle θ2. The angle θ2 is appropriately determined from the viewpoint of a margin when the user intentionally sets the orientation of the mobile terminal 3 to “sideways”. When the orientation of the mobile terminal 3 is "horizontal", the direction of the long side T1 of the housing 8 is along the direction orthogonal to the vertical direction, and the direction of the long side of the touch panel 9 is orthogonal to the vertical direction. It will be in a state along the direction of
"Horizontal" corresponds to "second orientation".

FIG. 4A is a diagram used for explaining “right movement”.
As shown in FIG. 4A, “right movement” means that the housing 8 is oriented “sideways” and the distance K1 (for example, the housing 8 is directed to the right when the housing 8 is viewed from the front). It is a movement to move within a period of GG1 (for example, 0.5 seconds) over a distance of more than 3,3 cm. The right direction does not have to be completely along the direction orthogonal to the vertical direction, and a certain margin is considered. When the user who grips the housing 8 and visually recognizes the touch panel 9 lightly moves the housing 8 toward the right when viewed from the user, the housing 8 "moves to the right". The distance K1 and the period GG1 are appropriately determined from the viewpoint of being used as a threshold value for determining that the user intentionally causes the housing 8 to "move to the right".
The "right movement" corresponds to the "first movement in which the state of the housing 8 changes in the first aspect".

FIG. 4B is a diagram used for explaining “left movement”.
As shown in FIG. 4B, “left movement” means that the housing 8 is oriented “sideways” and the distance K2 (for example, the housing 8 is directed to the left when the housing 8 is viewed from the front). It is a movement to move within a period of GG2 (for example, 0.5 seconds) over a distance of more than 3,3 cm. The left direction does not have to be completely along the direction orthogonal to the vertical direction, and a certain margin is considered. When the user who grips the housing 8 and visually recognizes the touch panel 9 lightly moves the housing 8 toward the left when viewed from the user, the housing 8 "moves to the left". The distance K2 and the period GG2 are appropriately determined from the viewpoint of being used as a threshold value for determining that the user intentionally causes the housing 8 to "move left".
The "left movement" corresponds to the "second movement in which the state of the housing 8 changes in the second mode".

FIG. 5 is a block diagram showing a functional configuration of the information processing device 4.
As shown in FIG. 5, the information processing device 4 includes an information processing device control unit 30, an information processing device communication unit 31, a display device 32, an information processing device storage unit 33, an operation input unit 34, and a GPS unit. 35, a relative orientation detection unit 36, a bus interface 37, and a camera interface 38 are provided.

The information processing device control unit 30 includes a CPU, ROM, RAM, ASIC, signal processing circuit, and the like, and controls each unit of the information processing device 4. In the information processing device control unit 30, for example, the CPU reads the program stored in the ROM into the RAM and executes the process, executes the process by the function implemented in the ASIC, and processes the signal by the signal processing circuit, for example. The process is executed in cooperation with the hardware and software, such as executing the process.

The information processing device communication unit 31 wirelessly communicates with an external device (including the mobile terminal 3) according to the Wi-Fi standard under the control of the information processing device control unit 30.
The display device 32 includes a display panel such as a liquid crystal panel or an organic EL panel, and displays an image on the display panel under the control of the information processing device control unit 30.
The information processing device storage unit 33 includes a non-volatile memory such as an EEPROM and a hard disk, and stores various data in a rewritable non-volatile manner. The information processing device storage unit 33 stores the map data 33a. The map data 33a stores information used when displaying a map on the display device 32 and information used for route search (information on so-called links, information on so-called nodes, and the like). Further, the map data 33a stores a detailed map of the facility such as a parking lot where the vehicle S can park, and information indicating the structure of the facility (hereinafter, referred to as "parking map information").

The operation input unit 34 includes an operator such as a switch provided in the information processing device 4, detects an operation on the operator by the user, and outputs a signal corresponding to the detected operation to the information processing device control unit 30. The information processing device control unit 30 executes a process corresponding to an operation on the operator by the user based on the input from the operation input unit 34.

The GPS unit 35 receives GPS radio waves from GPS satellites via GPS antennas (not shown), and calculates the current position of the vehicle S and the traveling direction of the vehicle S from the GPS signals superimposed on the GPS radio waves. The GPS unit 35 outputs the calculated information indicating the current position of the vehicle S and the calculated information indicating the traveling direction of the vehicle S to the information processing device control unit 30.
The relative bearing detection unit 36 includes a gyro sensor and an acceleration sensor. The gyro sensor is composed of, for example, a vibrating gyro, and detects a relative orientation of the vehicle S (for example, a turning amount in the yaw axis direction). The acceleration sensor detects the acceleration acting on the vehicle S (for example, the inclination of the vehicle with respect to the traveling direction). The relative orientation detection unit 36 outputs information indicating the relative orientation of the detected vehicle S and information indicating the acceleration acting on the detected vehicle S to the information processing device control unit 30.

The bus interface 37 includes a communication interface corresponding to the bus 6, and communicates with an external device (including a vehicle drive control device 5) connected to the bus 6 under the control of the information processing device control unit 30.
The camera interface 38 communicates with a camera provided in the vehicle S under the control of the information processing device control unit 30.
The vehicle S includes a front shooting camera CA1, a front right shooting camera CA2, a side right shooting camera CA3, a rear right shooting camera CA4, and a rear shooting camera CA5 as cameras for shooting the outside of the vehicle. A rear left side shooting camera CA6, a side left side shooting camera CA7, and a front left side shooting camera CA8 are provided. Hereinafter, when the cameras that shoot outside the vehicle are not distinguished, they are referred to as "cameras for shooting outside the vehicle".

FIG. 6 is a diagram used to explain the direction in which the camera for shooting outside the vehicle shoots. Note that FIG. 6 is also used for a description of switching the outside image G2, which will be described later.
The front camera CA1 is a camera that shoots the front of the vehicle S (see FIG. 6).
The front right side photographing camera CA2 is a camera that photographs the front right side (see FIG. 6) of the vehicle S.
The side right side photographing camera CA3 is a camera that photographs the side right side (see FIG. 6) of the vehicle S.
The rear right side photographing camera CA4 is a camera that photographs the rear right side (see FIG. 6) of the vehicle S.
The rear-view camera CA5 is a camera that shoots the rear of the vehicle S (see FIG. 6).
The rear left side photographing camera CA6 is a camera that photographs the rear left side (see FIG. 6) of the vehicle S.
The side left side photographing camera CA7 is a camera that photographs the side left side (see FIG. 6) of the vehicle S.
The front left side photographing camera CA8 is a camera that photographs the front left side (see FIG. 6) of the vehicle S.
Each of the cameras for outside the vehicle executes shooting at a predetermined cycle, generates captured image data based on the captured result, and transmits the generated captured image data to the information processing device control unit 30 via the camera interface 38. Output.
The mode of the camera for outside-vehicle photography provided in the vehicle S is not limited to the mode according to the present embodiment. As an example, the vehicle S may be provided with a camera for shooting outside the vehicle in the following manner. That is, the vehicle S is provided with four cameras, a front camera CA1, a side right camera CA3, a rear camera CA5, and a side left camera CA7, as cameras for outside the vehicle, and the side right side. The side right side and front right side of the vehicle S are photographed by the photographing camera CA3, the rear, rear right side, and rear left side of the vehicle S are photographed by the rear photographing camera CA5, and the side left side photographing camera vehicle S side. The left side of the camera and the left side of the front may be photographed.

Further, among the above-mentioned cameras for external photography, the front camera CA1, the side right camera CA3, the rear camera CA5, and the side left camera CA7 are used to generate bird's-eye view image data (described later). It is used as a camera to be used. Hereinafter, when the cameras used for generating the bird's-eye view image data are not distinguished, they are referred to as "bird's-eye view cameras". Each of the bird's-eye view cameras is a wide-angle camera, and is provided at an appropriate position of the vehicle S from the viewpoint of being used for generating bird's-eye view image data.

FIG. 7 is a block diagram showing a functional configuration of the vehicle drive control device 5.
As shown in FIG. 7, the vehicle drive control device 5 includes a vehicle drive control device control unit 50, a road-to-vehicle communication unit 51, a vehicle-to-vehicle communication unit 52, a radar device 53, a vehicle speed sensor 54, and a yaw rate sensor 55. And a bus interface 56.
The vehicle drive control device control unit 50 includes a CPU, ROM, RAM, ASIC, signal processing circuit, and the like, and controls each unit of the vehicle drive control device 5. In the vehicle drive control device control unit 50, for example, the CPU reads the program stored in the ROM into the RAM and executes the process, executes the process by the function implemented in the ASIC, for example, and signals in the signal processing circuit, for example. The process is executed in cooperation with hardware and software, such as performing the process and executing the process.

The road-to-vehicle communication unit 51 receives information transmitted by narrow-band wireless communication such as an optical beacon, a radio wave beacon, and DSRC (Dedicated Short Range Communications) from a roadside unit installed on a road or the like. The information transmitted from the roadside unit to the road-to-vehicle communication unit 51 includes, for example, information about other vehicles, information about pedestrians, and the like. The road-to-vehicle communication unit 51 outputs the information received from the roadside unit to the vehicle drive control device control unit 50.

The vehicle-to-vehicle communication unit 52 transmits and receives information to and from other vehicles located in the vicinity of the vehicle S by wireless communication. The information transmitted and received by the vehicle-to-vehicle communication unit 52 includes, for example, identification information for identifying the vehicle S and other vehicles, information indicating the position of the vehicle S and other vehicles, and information indicating the speed of the vehicle S and other vehicles. , Information indicating the traveling direction of the vehicle S and other vehicles, information indicating the timing at which the vehicle S and other vehicles stop, and the like are included. The vehicle-to-vehicle communication unit 52 outputs the received information to the vehicle drive control device control unit 50.

The radar device 53 irradiates, for example, a radio wave such as a millimeter wave radar or a laser radar, a sound wave such as an ultrasonic radar, or the like toward the outside of the vehicle. The radar device 53 detects an object existing in the vicinity of the vehicle S by receiving a reflected wave reflected by an object (for example, another vehicle or a person) existing in the vicinity of the vehicle S. The radar device 114 outputs information about the detected object to the vehicle drive control device control unit 50.

The vehicle speed sensor 54 detects the speed of the vehicle S (hereinafter, referred to as “vehicle speed”) and outputs information indicating the detected vehicle speed to the vehicle drive control device control unit 50.
The yaw rate sensor 55 detects the yaw rate applied to the vehicle S, and outputs information indicating the detected yaw rate to the vehicle drive control device control unit 50.

The bus interface 56 includes a communication interface corresponding to the bus 6, and communicates with an external device (including an information processing device 4) connected to the bus 6 under the control of the vehicle drive control device control unit 50.

As shown in FIG. 7, the engine ECU 60, the transmission ECU 61, the brake ECU 62, and the steering ECU 63 are connected to the bus 6 as ECUs. The vehicle drive control device control unit 50 of the vehicle drive control device 5 outputs a control signal to each ECU via the bus 6 to control each ECU.

The engine ECU 60 controls a throttle actuator that opens and closes an electronic throttle valve provided in the intake pipe of the engine based on a control signal input from the vehicle drive control device control unit 50 to adjust the engine rotation speed.
The transmission ECU 61 controls the hydraulic control device that adjusts the oil pressure of the hydraulic oil supplied to the transmission based on the control signal input from the vehicle drive control device control unit 50, and the hydraulic oil supplied to the transmission. It adjusts the flood control, switches the gear ratio of the transmission, and changes the number of revolutions and torque transmitted from the engine. In particular, the transmission ECU 61 changes the gear state of the vehicle S to parking (P), reverse (R), drive (D), and neutral (N) based on the control signal input from the vehicle drive control device control unit 50. , And switch between low (L).
The brake ECU 62 controls the brake device provided on the wheel of the vehicle S based on the control signal input from the vehicle drive control device control unit 50 to brake the vehicle S.
The steering ECU 63 controls the steering device provided in the vehicle S based on the control signal input from the vehicle drive control device control unit 50 to steer the vehicle S.

Based on the automatic driving route information, the vehicle drive control device 5 moves the vehicle S from the stop position where the vehicle S stops and stops the vehicle S from the stop position without the user driving the vehicle S. It can be automatically moved to the target position, which is the target position, and stopped. As will be described later, the information processing device 4 outputs the automatic driving route information to the vehicle drive control device 5 when the vehicle automatically moves.

FIG. 8 is a diagram used for explaining the automatic driving route information.
The automatic driving route information is from the stop position to the target position in a virtual coordinate system in which the vehicle S stopped at the stop position is the origin, the front-rear direction of the vehicle S is the y-axis, and the left-right direction of the vehicle S is the x-axis. Includes trajectory information indicating the trajectory of the vehicle S of the vehicle S. The locus information includes information indicating the coordinates of the points representing the locus of the vehicle S in the virtual coordinate system and information indicating the direction of the vehicle S in the virtual coordinate system at each point.
FIG. 8 shows an example of the virtual coordinate system KZ1 which is an example of the virtual coordinate system and an example of the trajectory of the vehicle S in the virtual coordinate system KZ1 in a simplified manner suitable for explanation. In the case of the locus illustrated in FIG. 8, the locus information includes point P0 (the origin and the point corresponding to the stop position), point P1, point P2, point P3, point P4, and point P5 (corresponding to the target position). Includes information indicating the coordinates of each point) and information indicating the direction of the vehicle S in the virtual coordinate system KZ1 at each point.

The vehicle drive control device control unit 50 of the vehicle drive control device 5 is based on the automatic driving route information and the inputs from the vehicle-to-vehicle communication unit 52, the radar device 53, the vehicle speed sensor 54, and the yaw rate sensor 55. A control signal is output to the transmission ECU 61, the brake ECU 62, and the steering ECU 63, and the vehicle S stopped at the stop position is moved to the target position on the track corresponding to the track information included in the automatic driving route information, and stopped at the target position.

The method in which the vehicle drive control device 5 automatically moves the vehicle S from the stopped position to the target position based on the automatic driving route information has been described above. However, the method in which the vehicle drive control device 5 moves the vehicle S from the stopped position to the target position is not limited to the above-mentioned method, and may be any method. Further, the sensors, ECUs, devices, etc. used by the vehicle drive control device 5 when the vehicle S is automatically moved from the stopped position to the target position are limited to the sensors, ECUs, devices, etc. according to the present embodiment. I can't. Further, the content of the automatic driving route information may be any content as long as it is information used when the vehicle drive control device 5 automatically moves the vehicle S from the stopped position to the target position.

Next, the operation of each device of the information processing system 1 when the vehicle is automatically moved will be described by exemplifying a case where the vehicle S is parked and a case where the vehicle S is discharged. As described above, parking means moving the vehicle S parked at the position before parking near the parking position to the parking position and stopping the vehicle S. Further, the warehousing means that the vehicle S parked at the parking position is moved to the warehousing stop position near the parking position and then stopped.

<When parking vehicle S>
First, a case where the vehicle S is parked will be described.
The flowchart FA of FIG. 9 is a flowchart showing the operation of the information processing device 4, and the flowchart FB is a flowchart showing the operation of the mobile terminal 3.
Hereinafter, the operations of the information processing device 4 and the mobile terminal 3 will be described by taking as an example a case where the vehicle S is parked at a parking position corresponding to one parking area of a parking lot provided with a plurality of parking areas.

When parking by automatic vehicle movement, the user stops the vehicle S at a position before parking near the parking position.
FIG. 10 is a diagram showing a parking lot ST1 which is an example of a parking lot, a parking lot position PS1 which is an example of a parking lot pre-parking position in the parking lot ST1, and a parking lot position PS2 which is an example of a parking position in the parking lot ST1. is there.
The parking lot ST1 has three parking areas AR1, a parking area AR2, and a parking area AR3 as parking areas where vehicles can park. Another vehicle different from the vehicle S is already parked in the parking area AR1 and the parking area AR3. In such a case, for example, the user determines the parking position PS2 in the parking area AR2 as a desired position for parking the vehicle S, and stops the vehicle S at the pre-parking position PS1 near the parking position PS2.

As shown in the flowchart FA of FIG. 9, after the vehicle S is stopped at the position before parking, the user operates the operation input unit 34 of the information processing device 4 to change the operation mode of the information processing device 4 to the parking mode. (Step S1).
When the operation mode is changed to the parking mode by the user, the information processing device control unit 30 of the information processing device 4 executes the parking position determination process (step SA1).
The parking position determination process is a process of identifying a parking area in which the vehicle S can be parked and determining the parking position.
In the parking position determination process of step SA1, the information processing device control unit 30 bases the vehicle based on the information input from the GPS unit 35, the information input from the relative orientation detection unit 36, the information contained in the map data 33a, and the like. Identify the current position of S (position before parking). The method for specifying the current position of the vehicle S may be any method, and information other than the above-mentioned information may be used. For example, when specifying the current position of the vehicle S, information based on signals of a positioning satellite system such as GLONASS, Galileo, Beidou, or QZSS (MICHIBIKI) may be used.

Next, the information processing device control unit 30 parks in the vicinity of the current position of the vehicle S based on the captured image data input from each of the cameras provided in the vehicle S, the information of the map data 33a, and the like. A parking area in which the vehicle S can be parked is specified. When the map data 33a has parking map information related to the parking lot to which the current position of the vehicle S belongs, the information processing device control unit 30 specifies the parking area by using the parking map information. Any method may be used to identify the parking area existing in the vicinity of the current position of the vehicle S.
Next, the information processing device control unit 30 determines the position corresponding to the specified parking area as the parking position.

After determining the parking position in step SA1, the information processing apparatus control unit 30 executes the automatic driving route information generation process (step SA2).
The automatic driving route information generation process is a process of generating automatic driving route information including track information indicating a track from the current position where the vehicle S stops (position before parking) to the parking position determined in step SA1. is there.
In the automatic driving route information generation process in step SA2, the information processing device control unit 30 of the vehicle S is based on the captured image data input from each of the cameras provided in the vehicle S, the information of the map data 33a, and the like. From the current position, the track of the vehicle S reaching the parking position is calculated.
As shown in FIG. 10, when the vehicle S stopped at the pre-parking position PS1 is moved to the parking position PS2, the information processing device control unit 30 uses, for example, the track of the vehicle S from the pre-parking position to the parking position. Calculate the orbit KD. The track KD illustrated in FIG. 10 has a track KD1 from the position PS1 before parking to the position MD on the track KD, and a track KD2 from the position MD to the parking position PS2. The track KD1 is a track in which the vehicle S "forwards" while turning left in the direction of travel from the position PS1 before parking, and temporarily stops at the position MD. The track KD2 is a track in which the vehicle S "reverses" while turning left in the direction of travel from the position MD and stops at the parking position PS2.
Next, the information processing device control unit 30 defines a virtual coordinate system with the current position of the vehicle S as the origin, and develops and develops a line segment corresponding to the trajectory of the vehicle S in the defined virtual coordinate system. Track information is generated based on the line segment, and automatic operation route information including track information is generated.

On the other hand, after changing the operation mode of the information processing device 4 to the parking mode in step S1, the user gets off the vehicle S with the mobile terminal 3 carried and goes out of the vehicle. The user does not necessarily have to disembark the vehicle S, but in the present embodiment, for convenience of explanation, the user will disembark the vehicle S.
After getting out of the vehicle, the user operates the touch panel 9 of the mobile terminal 3 to activate the dedicated application AP (step S2). When the user activates the dedicated application AP, the orientation of the housing 8 of the mobile terminal 3 is set to "vertical orientation".
The terminal control unit 20 of the mobile terminal 3 executes the process of the flowchart FB by the function of the dedicated application AP activated by the user in step S2.

In response to the activation of the dedicated application AP, the terminal control unit 20 causes the touch panel 9 to display the lock screen GM1 (see FIG. 11) (step SB1).
In the following description, even if no particular explanation is given, the process of displaying the screen on the touch panel 9, the process of changing the contents of the screen according to the operation by the user, and other events, and the process of displaying on the touch panel 9. The main body that executes the process of switching the screen from one screen to another is the terminal control unit 20.
After displaying the lock screen GM1, the terminal control unit 20 monitors whether or not there is an input instructing the lock screen GM1 to shift to the vehicle propulsion control mode (step SB2).
Hereinafter, the contents of the lock screen GM1 and the processing of step SB2 will be described in detail.

FIG. 11 is a diagram showing the lock screen GM1.
The lock screen GM1 is a screen that accepts input of an instruction to shift the operation mode of the mobile terminal 3 to the vehicle propulsion control mode (described later). When an instruction to shift the operation mode is input to the lock screen GM1 in the manner described later, the terminal control unit 20 shifts the operation mode to the vehicle propulsion control mode and locks the screen displayed on the touch panel 9. Switch from the screen GM1 to the vehicle control screen GM2 (described later). As will be described later, while the operation mode is the vehicle propulsion control mode, the user can operate the touch panel 9 to control the propulsion of the vehicle S.

As shown in FIG. 11, the slide unit SU1 is displayed on the lock screen GM1. The slide unit SU1 has a slide bar BR1 and a slide button SB1.
The slide bar BR1 is a strip-shaped object extending in the left-right direction, clearly indicates the range in which the slide button SB1 can move, and restricts the movement of the slide button SB1.
The slide button SB1 is a button that can be touch-operated by the user (hereinafter, the button that can be touch-operated is referred to as a "touch operation button"), and moves along the slide bar BR1 in response to a swipe operation of the user. .. As shown in FIG. 11, the slide button SB1 is maintained in a state of being positioned at the left end BR1a (one end) of the slide bar BR1 when no touch operation is performed by the user. The slide button SB1 can be moved from the left end BR1a of the slide bar BR1 to the right end BR1b (the other end).
On the lock screen GM1, above the slide unit SU1, a wording that simply expresses the operation to be performed by the user when inputting the instruction for shifting to the vehicle propulsion control mode is displayed.

FIG. 12 is a diagram used for explaining the screen displayed on the touch panel 9. FIG. 12 shows a part of a user's finger that operates the screen together with the screen displayed on the touch panel 9.
The state J1 in FIG. 12 shows the lock screen GM1 in a state in which the user has not performed a touch operation. When inputting an instruction to shift to the vehicle propulsion control mode, the user touches the slide button SB1 located at the left end BR1a of the slide bar BR1 on the lock screen GM1 in the state J1 with a finger. The user then swipes to move the slide button SB1 to the right along the slide bar BR1. The swipe operation means an operation of moving the position where the finger touches while maintaining the state where the finger touches the touch panel 9. In response to the swipe operation, the terminal control unit 20 follows the position where the finger touches and moves the slide button SB1 along the slide bar BR1. The state J2 in FIG. 12 shows the lock screen GM1 in a state where the slide button SB1 is located at the center of the slide bar BR1 by a swipe operation.

Here, on the lock screen GM1, the vehicle control screen GM2 (specifically, the bird's-eye view image display screen GM2a described later) is displayed as a background. When the lock screen GM1 is in the state J1, the terminal control unit 20 masks the bird's-eye view image display screen GM2a displayed as the background with a dark black mask image that covers the entire background. Then, the terminal control unit 20 makes the color of the mask image lighter as the distance between the left end BR1a of the slide bar BR1 and the slide button SB1 increases. As a result, the more the slide button SB1 moves to the right in response to the swipe operation by the user, the clearer the bird's-eye view image display screen GM2a displayed as the background becomes. By referring to how the bird's-eye view image display screen GM2a displayed as the background becomes clearer in response to the swipe operation on the slide button SB1, the user can intuitively input to the lock screen GM1 appropriately. You can recognize that.

The user who performs the swipe operation moves the slide button SB1 to the right end BR1b of the slide bar BR1 and maintains the state in which the slide button SB1 is located at the right end BR1b for a predetermined period (for example, 3 seconds). As a result, the input of the instruction for shifting to the vehicle propulsion control mode is completed, and the operation mode shifts to the vehicle propulsion control mode. The state J3 in FIG. 12 shows the lock screen GM1 in which the slide button SB1 is located at the right end BR1b of the slide bar BR1. When the state J3 continues for a predetermined period, the input of the instruction for shifting to the vehicle propulsion control mode is completed. As is clear from the comparison of the states J1, the state J2, and the state J3 in FIG. 12, the more the slide button SB1 is located to the right, the clearer the background bird's-eye view image display screen GM2a is.
If the swipe operation is released before the input of the instruction for shifting to the vehicle propulsion control mode is completed, the terminal control unit 20 presses the slide button SB1 on the slide bar BR1 in response to the release of the swipe operation. Move to the leftmost BR1a.

In step SB2, the terminal control unit 20 monitors whether or not the slide button SB1 is positioned on the right end BR1b of the slide bar BR1 for a predetermined period based on the input from the touch panel 9. Then, the terminal control unit 20 determines that when the slide button SB1 is located at the right end BR1b of the slide bar BR1 for a predetermined period of time, there is an input instructing the shift to the vehicle propulsion control mode.

As described above, in the present embodiment, the input of the instruction to shift to the vehicle propulsion control mode is completed only when the slide button SB1 is positioned at the right end BR1b of the slide bar BR1 for a predetermined period of time. This configuration prevents the user's finger from accidentally touching the touch panel 9 and completing the input of the instruction to shift to the vehicle propulsion control mode against the user's intention.

When it is determined in step SB2 that there is an input instructing the transition to the vehicle propulsion control mode (step SB2: YES), the terminal control unit 20 shifts the operation mode to the vehicle propulsion control mode (step SB3).
The terminal control unit 20 switches the screen displayed on the touch panel 9 from the lock screen GM1 to the vehicle control screen GM2 according to the transition of the operation mode (step SB4).

The state J4 of FIG. 12 shows the vehicle control screen GM2 immediately after the screen displayed on the touch panel 9 is switched from the lock screen GM1 to the vehicle control screen GM2. Immediately after the screen is switched to the vehicle control screen GM2, the orientation of the housing 8 of the mobile terminal 3 is "vertical orientation".
As will be described in detail later, as the vehicle control screen GM2, either the bird's-eye view image display screen GM2a or the vehicle exterior image display screen GM2b is displayed on the touch panel 9 according to the orientation of the housing 8. Immediately after the screen is switched to the vehicle control screen GM2, the orientation of the housing 8 is "vertical", and the touch panel 9 displays the bird's-eye view image display screen GM2a as the vehicle control screen GM2.
As shown in the state J4 of FIG. 12, the terminal control unit 20 changes the slide button SB1 on the lock screen GM1 to the propulsion control button QB (operation button) which is a touch operation button in response to the screen switching accompanying the transition of the operation mode. ), And the propulsion control button QB is displayed on the vehicle control screen GM2. That is, the terminal control unit 20 does not display the propulsion control button QB again as a button different from the slide button SB1 according to the screen switching, but instead displays the slide button SB1 located at the right end BR1b of the slide bar BR1 on the screen. It functions as a propulsion control button QB without changing the position according to the switching. Therefore, immediately after the screen is switched, the propulsion control button QB is located at the right end BR1b of the slide bar BR1 on the lock screen GM1, and the propulsion control button QB is in a state of being touch-operated by the user.

As shown in the flowchart FA and the flowchart FB of FIG. 9, after the operation mode of the mobile terminal 3 shifts to the vehicle propulsion control mode, the information processing device control unit 30 of the information processing device 4 and the terminal control of the mobile terminal 3 The unit 20 cooperates to execute the vehicle propulsion control process (step SA3, step SB5). Hereinafter, the vehicle propulsion control process will be described separately for the process related to the control of the propulsion of the vehicle S and the process related to the display on the touch panel 9.
When the dedicated application AP is activated, the terminal control unit 20 establishes a communication path between the mobile terminal 3 and the information processing device 4 in accordance with the Wi-Fi standard, and the mobile terminal 3 and the information processing device 4 Establish a state where communication is possible with. Information necessary for establishing a communication path (for example, a password required for the information processing device 4 to authenticate the mobile terminal 3) is registered in advance.

<Processing related to control of propulsion of vehicle S>
First, the process related to the control of the propulsion of the vehicle S in the vehicle propulsion control process will be described.
In the vehicle propulsion control process, the terminal control unit 20 of the mobile terminal 3 controls the terminal communication unit 21 while the propulsion control button QB displayed on the vehicle control screen GM2 is touch-operated to provide information on the propulsion instruction signal. It is transmitted to the processing device 4. On the other hand, the terminal control unit 20 stops the transmission of the propulsion instruction signal in response to the release of the touch operation of the propulsion control button QB. That is, the terminal control unit 20 does not transmit the propulsion instruction signal to the information processing device 4 while the propulsion control button QB is not touch-operated. As will be described later, the screen displayed as the vehicle control screen GM2 is switched according to the orientation of the housing 8 of the mobile terminal 3, and the position of the propulsion control button QB on the vehicle control screen GM2 is changed according to the user's operation. Moves. However, the terminal control unit 20 gives a propulsion instruction while the propulsion control button QB is touch-operated regardless of the screen switching displayed on the vehicle control screen GM2 and regardless of the position of the propulsion control button QB. The signal is transmitted to the information processing device 4, and the propulsion instruction signal is not transmitted to the information processing device 4 while the touch operation is not performed.

In the vehicle propulsion control process, the information processing device control unit 30 of the information processing device 4 outputs the automatic driving route information generated in the automatic driving route information generation process of step SA2 to the vehicle drive control device 5. Further, the information processing device control unit 30 outputs the drive instruction signal to the vehicle drive control device 5 while receiving the propulsion instruction signal from the mobile terminal 3. On the other hand, the information processing device control unit 30 does not output the drive instruction signal to the vehicle drive control device 5 while the propulsion instruction signal is not received from the mobile terminal 3.

The vehicle drive control device 5 executes the following processing based on the drive instruction signal input from the information processing device 4. That is, the vehicle drive control device control unit 50 of the vehicle drive control device 5 is based on the track information included in the automatic driving route information, and the track indicated by the track information while the drive instruction signal is input from the information processing device 4. The vehicle S is moved along. On the other hand, when the state in which the drive instruction signal is input changes to the state in which the drive instruction signal is not input, the vehicle drive control device control unit 50 brakes the vehicle S and stops the movement of the vehicle S. Then, when the drive instruction signal is not input to the input state, the vehicle drive control device control unit 50 starts the vehicle S and moves the vehicle S along the track indicated by the track information. Move. Explaining with reference to FIG. 10, the vehicle drive control device control unit 50 orbits the vehicle S while the drive instruction signal is input after starting the vehicle S located at the position PS1 before parking. Move along the KD. Then, when the vehicle S is located at the position PS3 and the drive instruction signal is not input, the vehicle drive control device control unit 50 brakes the vehicle S and stops the movement of the vehicle S. In this state, when the drive instruction signal is input again, the vehicle drive control device control unit 50 starts the vehicle S located at the position PS3 and moves the vehicle S along the track KD.

As a result of the above processing, while the user touches the propulsion control button QB on the vehicle control screen GM2, the vehicle S moves along the track, while the user does not touch the propulsion control button QB. During that time, the movement of the vehicle S stops. Therefore, when the user wishes to stop the movement of the vehicle S for some reason, the movement of the vehicle S can be stopped quickly by performing a simple operation of releasing the touch operation of the propulsion control button QB. Can be done.

<Processing related to display on touch panel 9>
Next, the process related to the display on the touch panel 9 in the vehicle propulsion control process will be described.
In the present embodiment, regarding the processing related to the display on the touch panel 9, there are two operation modes, the first mode and the second mode, as the operation modes. The user can switch the operation mode between the first mode and the second mode by a predetermined method.
Hereinafter, the operations of the information processing device 4 and the mobile terminal 3 when the operation mode is the first mode will be described, and then the operations of the information processing device 4 and the mobile terminal 3 when the operation mode is the second mode will be described. To do.

<1st mode>
The flowchart FC of FIG. 13 is a flowchart showing the operation of the mobile terminal 3 when the operation mode is the first mode. The flowchart FD of FIG. 14A and the flowchart FE of FIG. 14B are flowcharts showing the operation of the information processing apparatus 4 when the operation mode is the first mode, respectively.

As shown in the flowchart FC of FIG. 13, the terminal control unit 20 of the mobile terminal 3 controls the terminal communication unit 21 in accordance with the transition to the vehicle propulsion control mode, indicating that the terminal communication unit 21 has shifted to the vehicle propulsion control mode. The vehicle propulsion control mode information is transmitted to the information processing device 4 (step SC1).

As shown in the flowchart FD of FIG. 14A, the information processing device control unit 30 of the information processing device 4 receives the vehicle propulsion control mode information (step SD1).
The information processing device control unit 30 displays the composite image G on the display device 32 based on the captured image data input from each of the cameras provided in the vehicle S in response to the reception of the vehicle propulsion control mode information. Start (step SD2).

FIG. 15 is a front view of the display device 32 on which the composite image G is displayed.
As shown in FIG. 15, the shape of the display area AR capable of displaying an image on the display device 32 is a rectangle having a short side in the vertical direction and a long side in the horizontal direction. The vertical direction is a direction corresponding to the vertical direction, and the horizontal direction is a direction corresponding to a direction orthogonal to the vertical direction.
As shown in FIG. 15, the composite image G has a bird's-eye view image G1 and a vehicle exterior image G2.

The bird's-eye view image G1 is an image of a bird's-eye view of the vehicle S. As shown in FIG. 15, the bird's-eye view image G1 is a rectangular image having a long side in the vertical direction and a short side in the horizontal direction, and is displayed on the left side of the display area AR. At the lower right of the bird's-eye view image G1, a mark M1 indicating that the displayed image is the bird's-eye view image G1 is displayed.
The outside-vehicle image G2 is an image based on the result of shooting outside the vehicle by one of the outside-vehicle shooting cameras. As shown in FIG. 15, the vehicle exterior image G2 is a rectangular image having a short side in the vertical direction and a long side in the horizontal direction. The vehicle exterior image G2 is displayed on the right side of the display area AR in a state where the right side of the bird's-eye view image G1 and the left side of the vehicle exterior image G2 overlap. The mark M2 is displayed at the lower right of the vehicle exterior image G2. The mark M2 is a mark that clearly indicates the out-of-vehicle photography camera that is the output source of the photographed image data corresponding to the out-of-vehicle image G2 displayed on the display device 32 among the out-of-vehicle photography cameras. For example, when the camera for outside the vehicle that outputs the photographed image data corresponding to the outside image G2 displayed on the display device 32 is the front camera CA1, the mark M2 is a symbol representing the vehicle S. It is a decorated mark that clearly indicates that the front of the vehicle S was photographed. By referring to the mark M2, the user can accurately recognize which of the outside-vehicle shooting cameras the outside-vehicle image G2 displayed on the display device 32 is based on the shooting result of the outside-vehicle shooting camera.

When displaying the composite image, the information processing device control unit 30 acquires captured image data input from each of the bird's-eye view cameras. Next, based on the captured image data input from each of the bird's-eye view cameras, image data of an image of a bird's-eye view of the vehicle S (hereinafter, referred to as "overhead image data") is generated. The bird's-eye view of the vehicle S is an image showing the entire vehicle S and the periphery of the vehicle S as viewed from above. In the present embodiment, the image data is a dot matrix in which dots having information on colors (for example, information representing color components of each color of RGB by gradation values of predetermined gradations) have a predetermined resolution. It is the arranged bitmap data. The bird's-eye view image data is image data having a predetermined resolution having a size corresponding to the size of the bird's-eye view image G1 (length in the vertical direction and length in the horizontal direction). The process of generating the bird's-eye view image data based on the photographed image data input from the bird's-eye view camera is appropriately performed by the existing technology.
Each of the bird's-eye view cameras synchronously executes shooting, and synchronously outputs captured image data based on the shooting result to the information processing device control unit 30. Each of the bird's-eye view cameras executes shooting and outputs captured image data at a predetermined cycle (for example, 10 milliseconds). The information processing device control unit 30 acquires photographed image data input from each of the bird's-eye view cameras at a cycle corresponding to the predetermined cycle, and generates bird's-eye view image data based on the acquired photographed image data. When generating the bird's-eye view image data, the information processing apparatus control unit 30 adds the image data corresponding to the mark M1 to an appropriate position to generate the bird's-eye view image data.

In addition, each of the cameras for shooting outside the vehicle executes shooting at a predetermined cycle, and outputs captured image data based on the shooting result to the information processing device control unit 30. The information processing device control unit 30 selects one of the cameras for outside photography, acquires the photographed image data input from the selected camera for outside photography, and based on the photographed image data, outside the vehicle. Image data of image G2 (hereinafter referred to as "outside vehicle image data") is generated. The information processing device control unit 30 generates vehicle exterior image data at a cycle corresponding to the predetermined cycle. When generating the vehicle exterior image data, the information processing device control unit 30 adds the image data corresponding to the mark M2 to an appropriate position to generate the vehicle exterior image data. The vehicle exterior image data is image data having a predetermined resolution having a size corresponding to the size of the vehicle exterior image G2 (length in the vertical direction and length in the horizontal direction).

The information processing device control unit 30 synchronously executes the generation of the bird's-eye view image data based on the input from the bird's-eye view camera and the generation of the vehicle exterior image data based on the input from the vehicle exterior photography camera at a predetermined cycle. The information processing device control unit 30 synthesizes the generated bird's-eye view image data and the generated vehicle exterior image data in a predetermined cycle according to the positional relationship between the bird's-eye view image G1 and the vehicle exterior image G2, and combines the composite image G Generate composite image data. The information processing device control unit 30 controls the display device 32 based on the generated composite image data, and causes the display device 32 to display the composite image G based on the composite image data.
As a result, the display device 32 displays the composite image G in which the bird's-eye view image G1 and the vehicle exterior image G2 are combined. The content of the bird's-eye view image G1 changes at any time based on the shooting result of the bird's-eye view camera, and the content of the outside vehicle image G2 changes at any time based on the shooting result of the selected outside-vehicle shooting camera.

As described above, the information processing device control unit 30 generates composite image data at a predetermined cycle, and displays the composite image G on the display device 32.
In response to the display of the composite image G on the display device 32, the information processing device control unit 30 transmits the generated composite image data to the mobile terminal 3 at the predetermined cycle (step SD3).
The first mode is different from the second mode described later in that the information processing device control unit 30 transmits the composite image data to the mobile terminal 3.

As shown in the flowchart FC, the terminal control unit 20 of the mobile terminal 3 starts receiving the composite image data transmitted by the information processing device 4 (step SC2), and based on the received composite image data, a bird's-eye view on the touch panel 9. The image display screen GM2a is displayed (step SC3). At the time when the process of step SC3 is executed, the orientation of the housing of the mobile terminal 3 is "vertical orientation".
The bird's-eye view image display screen GM2a is a screen in which the bird's-eye view image G1 is displayed and the propulsion control button QB is displayed superimposed on the bird's-eye view image G1 as shown in the state J3 of FIG.
In step SC3, the terminal control unit 20 performs the following processing to display the bird's-eye view image display screen GM2a on the touch panel 9. That is, the terminal control unit 20 extracts the bird's-eye view image data from the composite image data for each of the composite image data received at a predetermined cycle. The area of the bird's-eye view image data in the composite image data is predetermined, and the terminal control unit 20 extracts the bird's-eye view image data by cutting out the data belonging to the area of the bird's-eye view image data in the composite image data from the composite image data. To do. Next, the terminal control unit 20 performs necessary image processing (processing for adjusting the resolution, processing for adjusting the size, etc.) on the extracted bird's-eye view image data, and the bird's-eye view image data after performing the image processing. The bird's-eye view image display screen GM2a is displayed on the touch panel 9 based on the above.
The content of the bird's-eye view image G1 displayed on the bird's-eye view image display screen GM2a changes at any time in synchronization with the bird's-eye view image G1 displayed on the display device 32 of the information processing device 4 based on the shooting result of the bird's-eye view camera. ..

Here, as described above, the bird's-eye view image G1 has a long side in the vertical direction (direction corresponding to the vertical direction) and a short side in the left-right direction (direction corresponding to the direction orthogonal to the vertical direction). It is a rectangular image having. When the orientation of the housing 8 of the mobile terminal 3 is "vertical", the touch panel 9 is in a state where the long side direction is along the vertical direction and the short side direction is along the direction orthogonal to the vertical direction. .. Therefore, when the orientation of the housing 8 is "vertical", by displaying the bird's-eye view image G1 on the touch panel 9, the bird's-eye view image G1 has a long side in the vertical direction and a short side in the horizontal direction. The bird's-eye view image G1 can be displayed by using the entire display area of the touch panel 9 without unnecessarily reducing the bird's-eye view image G1. Therefore, the bird's-eye view image G1 is easy for the user to see, and the convenience of the user is improved.

After displaying the bird's-eye view image display screen GM2a, the terminal control unit 20 changes the orientation of the housing 8 of the mobile terminal 3 from "vertical" to "horizontal" based on the input from the housing state detection unit 23. It is monitored whether or not it has changed (step SC4).
When it is detected that the orientation of the housing 8 of the mobile terminal 3 has changed from "vertical orientation" to "horizontal orientation" (step SC4: YES), the terminal control unit 20 determines the predetermined orientation corresponding to the state of the vehicle S. Among the cameras for outside-vehicle photography, one camera for outside-vehicle photography is selected according to the priority of (step SC5).

For example, the priority is a higher priority as the camera for outside the vehicle capable of photographing the traveling direction of the vehicle S. In this case, if the vehicle S is "advancing" toward the front, the front camera CA1 has the highest priority, and if the vehicle S is "advancing" toward the front right side, the vehicle S has the highest priority. The front right camera CA2 has the highest priority, and when the vehicle S is "backward" toward the rear, the rear camera CA5 has the highest priority. Here, when the vehicle S is automatically moving, it is assumed that the user often wants to recognize the state of the vehicle S in the traveling direction.
Further, for example, the priority is a higher priority as the camera capable of photographing an object closer to the vehicle S. The object close to the vehicle S is, for example, a vehicle that has already been in the parking area adjacent to the parking area to be parked by automatic vehicle movement, and is, for example, a person other than the user. When an object exists at a position close to the vehicle S, it is assumed that the user often wants to recognize the object and recognize the positional relationship between the object and the vehicle S.

In step SC5, the terminal control unit 20 appropriately communicates with the information processing device 4 to acquire information necessary for selecting the camera for outside-vehicle photography in a predetermined priority order.

After selecting one camera for out-of-vehicle photography in step SC5, the terminal control unit 20 processes selected camera information indicating the selected camera for out-of-vehicle photography (for example, identification information for identifying the selected camera for out-of-vehicle photography). It is transmitted to the device 4 (step SC6).

As shown in the flowchart FE of FIG. 14B, the information processing device control unit 30 of the information processing device 4 receives the selected camera information (step SE1).
Next, the information processing device control unit 30 changes the vehicle exterior image G2 displayed on the display device 32 to an image based on the shooting result of the vehicle exterior shooting camera indicated by the selected camera information received in step SE1 (step SE2).
In step SE2, the information processing device control unit 30 synthesizes the bird's-eye view image data and the vehicle exterior image data based on the shooting result of the vehicle exterior camera indicated by the selected camera information to generate composite image data, and the generated composite image. The composite image G is displayed based on the data.
Before and after the process of step SE2, the information processing device control unit 30 continuously transmits the composite image data corresponding to the composite image G to be displayed on the display device 32. As a result, the composite image data transmitted by the information processing apparatus 4 to the mobile terminal 3 after the processing of step SE2 includes the vehicle exterior image data based on the imaging result of the vehicle exterior camera selected in step SC5. Become.

As shown in the flowchart FC of FIG. 13, the terminal control unit 20 of the mobile terminal 3 transmits the selected camera information in step SC6, and then, based on the composite image data received from the information processing device 4, the vehicle exterior image is displayed on the touch panel 9. The display screen GM2b is displayed (step SC7).
The composite image data used for displaying the vehicle exterior image display screen GM2b in step SC7 is composite image data including the vehicle exterior image data based on the shooting result of the vehicle exterior shooting camera selected in step SC5.
The state J6 in FIG. 12 shows the vehicle exterior image display screen GM2b.
The vehicle exterior image display screen GM2b is a screen on which the vehicle exterior image G2 is displayed and the propulsion control button QB is displayed superimposed on the vehicle exterior image G2, as shown by the state J6 in FIG.

In step SC7, the terminal control unit 20 extracts the vehicle exterior image data from the composite image data for each of the composite image data received at a predetermined cycle. The area of the vehicle exterior image data in the composite image data is predetermined, and the terminal control unit 20 extracts the vehicle exterior image data by cutting out the data belonging to the vehicle exterior image data area in the composite image data from the composite image data. To do. Next, the terminal control unit 20 performs necessary image processing (processing for adjusting the resolution, processing for adjusting the size, etc.) on the extracted vehicle exterior image data, and the vehicle exterior image data after performing the image processing. The vehicle exterior image display screen GM2b is displayed on the touch panel 9 based on the above.
The content of the vehicle exterior image G2 displayed on the vehicle exterior image display screen GM2b is synchronized with the vehicle exterior image G2 displayed on the display device 32 of the information processing device 4 based on the shooting result of the vehicle exterior shooting camera selected in step SC5. And it changes from time to time.

As described above, in the present embodiment, when the orientation of the housing 8 of the mobile terminal 3 changes from "portrait" to "horizontal", the touch panel 9 is replaced with the bird's-eye view image display screen GM2a, and the vehicle exterior image display screen is used. GM2b is displayed. Therefore, when the user wishes to check the vehicle exterior image G2 instead of the bird's-eye view image G1, the user performs a simple operation of changing the orientation of the housing 8 of the mobile terminal 3 to display the vehicle exterior image display screen GM2b. Can be displayed to confirm the outside image G2. In particular, the operation of changing the orientation of the housing 8 of the mobile terminal 3 is an operation that can be easily and accurately executed even when the user holds the mobile terminal 3 with one hand, which is convenient for the user. Highly sexual.
In particular, in the present embodiment, the vehicle exterior image display screen GM2b displayed in response to the orientation of the housing 8 of the mobile terminal 3 changing from "vertical" to "horizontal" is predetermined according to the state of the vehicle S. It is a screen based on the shooting result of the camera for shooting outside the vehicle selected according to the priority of. Therefore, the vehicle exterior image G2 displayed on the vehicle exterior image display screen GM2b is likely to be an image based on the shooting result of the vehicle exterior shooting camera desired by the user, which is highly convenient for the user.
Further, as described above, the vehicle exterior image G2 has a long side in the left-right direction (direction corresponding to the direction orthogonal to the vertical direction) and a short side in the vertical direction (direction corresponding to the vertical direction). It is a rectangular image. When the orientation of the housing 8 of the mobile terminal 3 is "horizontal", the touch panel 9 is in a state where the long side direction is orthogonal to the vertical direction and the short side direction is along the vertical direction. Therefore, when the orientation of the housing 8 is "horizontal", by displaying the vehicle exterior image G2 on the touch panel 9, the vehicle exterior image G2 has a long side in the left-right direction and a short side in the vertical direction. By preferably utilizing it, the vehicle exterior image G2 can be displayed by using the entire display area of the touch panel 9 without unnecessarily reducing the vehicle exterior image G2. Therefore, the user can easily see the image G2 outside the vehicle, and the convenience of the user is improved.

After displaying the vehicle exterior image display screen GM2b in step SC7, the terminal control unit 20 monitors whether or not the housing 8 has moved “rightward” or “leftward” (step SC8). , It is monitored whether or not the orientation of the housing 8 is "vertical" (step SC9).

When the orientation of the housing 8 is "vertical" (step SC9: YES) without the housing 8 moving "right" or "left" (step SC8: NO), the terminal The control unit 20 shifts the processing procedure to step SC3. As described above, in step SC3, the terminal control unit 20 causes the touch panel 9 to display the bird's-eye view image display screen GM2a based on the composite image data received from the information processing device 4.
As described above, in the present embodiment, when the orientation of the housing 8 of the mobile terminal 3 changes from "horizontal" to "vertical", the touch panel 9 is replaced with the external image display screen GM2b and the bird's-eye view image display screen GM2a. Is displayed. Therefore, when the user wishes to confirm the bird's-eye view image G1 instead of the vehicle exterior image G2, the user performs a simple operation of changing the orientation of the housing 8 of the mobile terminal 3 to display the bird's-eye view image display screen GM2a. Can be displayed to confirm the bird's-eye view image G1. In particular, the operation of changing the orientation of the housing 8 of the mobile terminal 3 is an operation that can be easily and accurately executed even when the user holds the mobile terminal 3 with one hand, which is convenient for the user. Highly sexual.
In particular, the bird's-eye view image G1 is assumed to be the image most frequently confirmed by the user during the automatic movement of the vehicle. Then, according to the present embodiment, the user can confirm the bird's-eye view image G1 at an arbitrary timing by setting the orientation of the housing 8 of the mobile terminal 3 to "vertical orientation", which is convenient for the user. high.

On the other hand, when the housing 8 makes either "right movement" or "left movement" without the orientation of the housing 8 being "vertical" (step SC9: NO) (step SC8: YES), the terminal The control unit 20 shifts the processing procedure to step SC10. That is, the terminal control unit 20 shifts the processing procedure to step SC10 when the housing 8 moves “rightward” or “leftward” when the housing 8 is oriented “sideways”.

In step SC10, the terminal control unit 20 determines whether the movement of the housing 8 is "right movement" or "left movement".
When the movement of the housing 8 is "right movement" (step SC10: "right movement"), the terminal control unit 20 is used for outside-vehicle shooting according to the first order according to the direction in which the outside-vehicle shooting camera shoots. Among the cameras, one camera for out-of-vehicle photography is selected (step SC11).
As shown in FIG. 6, the first order is "front"->"frontright"->"sideright"->"rearright"->"rear"->"rearleft"->"sideleft"->"front""Leftside" → "Front". That is, the first order is the order in which the direction in which the camera for shooting outside the vehicle shoots changes clockwise.
In step SC11, the terminal control unit 20 takes a picture of the direction next to the direction taken by the car-out camera selected before executing the process of step SC11 according to the first order. Select. For example, if the front shooting camera CA1 is selected before the process of step SC11 is executed, in step SC11, the terminal control unit 20 selects the front right shooting camera CA2 according to the first order. Further, for example, if the rear shooting camera CA5 is selected before executing the process of step SC11, in step SC11, the terminal control unit 20 selects the rear left side shooting camera CA6 according to the first order. ..

After selecting the camera for outside-vehicle photography in step SC11, the terminal control unit 20 transmits the selected camera information indicating the selected camera for outside-vehicle photography to the information processing device 4 (step SC12). After the processing of step SC12, the terminal control unit 20 returns the processing procedure to step SC8.
In response to the transmission of the selected camera information in step SC12, the information processing device 4 executes the process of the flowchart FE of FIG. 14B, and displays the vehicle exterior image G2 displayed on the display device 32 of the vehicle exterior shooting camera indicated by the selected camera information. Change to an image based on the shooting result. Along with this, the synthetic image data transmitted from the information processing apparatus 4 to the mobile terminal 3 includes the vehicle exterior image data based on the shooting result of the vehicle exterior shooting camera indicated by the selected camera information. As a result, the vehicle exterior image G2 displayed on the vehicle exterior image display screen GM2b of the touch panel 9 of the mobile terminal 3 becomes an image based on the shooting result of the vehicle exterior shooting camera selected in step SC11.

As described above, in the present embodiment, the user can switch the vehicle exterior image G2 displayed on the vehicle exterior image display screen GM2b by performing a simple operation of causing the housing 8 to "move to the right". Is highly convenient. In particular, when the housing 8 “moves to the right”, the shooting direction corresponding to the vehicle exterior image G2 displayed on the vehicle exterior image display screen GM2b changes clockwise according to the first order. Then, the "right direction" in which the housing 8 moves in the "right movement" and the "clockwise" are linked, and the user takes a picture in which direction after the housing 8 is made to "right move". It is possible to intuitively grasp whether the vehicle exterior image G2 based on the shooting result of the vehicle exterior photographing camera is displayed on the touch panel 9, which is highly convenient for the user.

On the other hand, when it is determined in step SC10 that the movement of the housing 8 is "left movement" (step SC10: "left movement"), the terminal control unit 20 executes the following processing. In the case of "right movement" (step SC10: "right movement"), the terminal control unit 20 is one of the outside shooting cameras according to the second order according to the direction in which the outside shooting camera shoots. Select a camera for shooting outside the vehicle (step SC13).
As shown in FIG. 6, the second order is "front"->"frontleft"->"sideleft"->"rearleft"->"rear"->"rearright"->"sideright"->"front""Rightside" → "Front". That is, the second order is the order in which the direction taken by the camera for outside the vehicle changes counterclockwise.
In step SC13, the terminal control unit 20 takes a picture of the direction next to the direction taken by the car-out camera selected before executing the process of step SC13 according to the second order. Select. For example, if the front camera CA1 is selected before the process of step SC13 is executed, in step SC13, the terminal control unit 20 selects the front left camera CA8 according to the second order. Further, for example, if the rear shooting camera CA5 is selected before executing the process of step SC13, in step SC13, the terminal control unit 20 selects the rear right shooting camera CA4 according to the second order. ..

After selecting the camera for outside-vehicle photography in step SC13, the terminal control unit 20 transmits the selected camera information indicating the selected camera for outside-vehicle photography to the information processing device 4 (step SC14). After the processing of step SC14, the terminal control unit 20 returns the processing procedure to step SC8.
In response to the transmission of the selected camera information in step SC14, the information processing device 4 executes the process of the flowchart FE of FIG. 14B, and displays the vehicle exterior image G2 displayed on the display device 32 of the vehicle exterior shooting camera indicated by the selected camera information. Change to an image based on the shooting result. Along with this, the synthetic image data transmitted from the information processing apparatus 4 to the mobile terminal 3 includes the vehicle exterior image data based on the shooting result of the vehicle exterior shooting camera indicated by the selected camera information. As a result, the vehicle exterior image G2 displayed on the vehicle exterior image display screen GM2b of the touch panel 9 of the mobile terminal 3 becomes an image based on the shooting result of the vehicle exterior shooting camera selected in step SC11.

As described above, in the present embodiment, the user can switch the outside image G2 displayed on the outside image display screen GM2b by performing a simple operation of causing the housing 8 to "move left", and the user can switch the outside image G2. Is highly convenient. In particular, when the housing 8 “moves to the left”, the shooting direction corresponding to the vehicle exterior image G2 displayed on the vehicle exterior image display screen GM2b changes counterclockwise according to the second order. Then, the "left direction" in which the housing 8 moves in the "left movement" and the "counterclockwise" are linked, and the user makes the housing 8 "left movement" and then in which direction. It is possible to intuitively grasp whether the vehicle exterior image G2 based on the shooting result of the vehicle exterior photographing camera to be photographed is displayed on the touch panel 9, which is highly convenient for the user.

The first mode has been described above. In the first mode, as compared with the second mode described later, it is not necessary for the information processing device 4 to execute a process of extracting the bird's-eye view image data and the vehicle exterior image data from the composite image data, so that the load on the information processing device 4 is increased. Is small.

<Second mode>
The flowchart FF of FIG. 16 is a flowchart showing the operation of the mobile terminal 3 when the operation mode is the second mode. The flowchart FG of FIG. 17A, the flowchart FH of FIG. 17B, and the flowchart FI of FIG. 17C are flowcharts showing the operation of the information processing apparatus 4 when the operation mode is the second mode, respectively.

As shown in the flowchart FF of FIG. 16, the terminal control unit 20 of the mobile terminal 3 controls the terminal communication unit 21 in accordance with the transition to the vehicle propulsion control mode, indicating that the terminal communication unit 21 has shifted to the vehicle propulsion control mode. The vehicle propulsion control mode information is transmitted to the information processing device 4 (step SF1).

As shown in the flowchart FG of FIG. 17A, the information processing device control unit 30 of the information processing device 4 receives the vehicle propulsion control mode information (step SG1).
The information processing device control unit 30 receives the vehicle propulsion control mode information, and based on the captured image data input from each of the cameras for outside the vehicle and the camera for bird's-eye view, the composite image on the display device 32. The display of G is started (step SG2).

As shown in the flowchart FF of FIG. 16, the terminal control unit 20 of the mobile terminal 3 has the orientation of the housing 8 of the mobile terminal 3 "vertical" in succession to the transmission of the vehicle propulsion control mode information in step SF1. The portrait information indicating the existence is transmitted (step SF2). At the time when the process of step SC3 is executed, the orientation of the housing of the mobile terminal 3 is "vertical orientation".

As shown in the flowchart FH of FIG. 17B, the information processing device control unit 30 of the information processing device 4 receives the vertical information (step SH1).
The information processing device control unit 30 starts transmitting the bird's-eye view image data to the mobile terminal 3 in response to the reception of the portrait information (step SH2).
In step SH2, the information processing apparatus control unit 30 extracts the bird's-eye view image data from the composite image data generated at a predetermined cycle. Next, the information processing device control unit 30 performs image processing on the extracted bird's-eye view image data according to the resolution and size of the touch panel 9 of the mobile terminal 3, and carries the bird's-eye view image data after the image processing. Send to terminal 3. The information processing apparatus control unit 30 executes the transmission of the bird's-eye view image data at a cycle corresponding to the cycle of generating the composite image data.
If the terminal control unit 20 has transmitted the vehicle exterior image data before the process of step SH2, the terminal control unit 20 stops the transmission of the vehicle exterior image data and then starts the transmission of the bird's-eye view image data.

As shown in the flowchart FF of FIG. 16, the terminal control unit 20 of the mobile terminal 3 starts receiving the bird's-eye view image data (step SF3).
Next, the terminal control unit 20 causes the touch panel 9 to display the bird's-eye view image display screen GM2a on which the bird's-eye view image G1 is displayed based on the bird's-eye view image data that started reception in step SF3 (step SF4).
The content of the bird's-eye view image G1 displayed on the bird's-eye view image display screen GM2a changes at any time in synchronization with the bird's-eye view image G1 displayed on the display device 32 of the information processing device 4 based on the shooting result of the bird's-eye view camera. ..

After displaying the bird's-eye view image display screen GM2a, the terminal control unit 20 changes the orientation of the housing 8 of the mobile terminal 3 from "vertical" to "horizontal" based on the input from the housing state detection unit 23. It is monitored whether or not it has changed (step SF5).
When it is detected that the orientation of the housing 8 of the mobile terminal 3 has changed from "vertical orientation" to "horizontal orientation" (step SF5: YES), the terminal control unit 20 determines the predetermined orientation corresponding to the state of the vehicle S. Among the cameras for outside-vehicle photography, one camera for outside-vehicle photography is selected according to the priority of (step SF6). The predetermined priority is, for example, the priority exemplified in the first mode.

Next, the terminal control unit 20 transmits the sideways information including the information indicating that the orientation of the mobile terminal 3 is “landscape” and the information indicating the camera for outdoor photography selected in step SF6 to the information processing device 4 ( Step SF7).

As shown in the flowchart FI of FIG. 17C, the information processing device control unit 30 of the information processing device 4 receives the sideways information (step SI1).
In response to the reception of the landscape information, the information processing device control unit 30 sets the vehicle exterior image G2 to be displayed on the display device 32 based on the shooting result of the vehicle exterior shooting camera indicated by the selected camera information included in the landscape information received in step SI1. Change to an image (step SI2).
Next, the information processing device control unit 30 stops the transmission of the bird's-eye view image data to the mobile terminal 3 and starts the transmission of the vehicle exterior image data (step SI3).
In step SI3, the information processing apparatus control unit 30 extracts the vehicle exterior image data from the composite image data generated at a predetermined cycle. Next, the information processing device control unit 30 performs image processing on the extracted vehicle exterior image data according to the resolution and size of the touch panel 9 of the mobile terminal 3, and carries the vehicle exterior image data after the image processing. Send to terminal 3. The information processing device control unit 30 executes the transmission of the vehicle exterior image data at a cycle corresponding to the cycle of generating the composite image data.

As shown in the flowchart FF of FIG. 16, the terminal control unit 20 of the mobile terminal 3 starts receiving the image data outside the vehicle (step SF8).
Next, the terminal control unit 20 causes the touch panel 9 to display the vehicle exterior image display screen GM2b on which the vehicle exterior image G2 is displayed, based on the vehicle exterior image data that started reception in step SF8 (step SF9).
The content of the vehicle exterior image G2 displayed on the vehicle exterior image display screen GM2b changes at any time in synchronization with the vehicle exterior image G2 displayed on the display device 32 of the information processing device 4 based on the shooting result of the bird's-eye view camera. ..
As a result of the process of step SF9, the vehicle exterior image display screen GM2b is displayed instead of the bird's-eye view image display screen GM2a according to the orientation of the housing 8 of the mobile terminal 3 changing from "vertical" to "horizontal". .. The effect of this is as described in the description of the first mode.

After displaying the vehicle exterior image display screen GM2b in step SF9, the terminal control unit 20 monitors whether or not the housing 8 has moved either “rightward” or “leftward” (step SF10). , It is monitored whether or not the orientation of the housing 8 is "vertical" (step SF11).

When the orientation of the housing 8 is "vertical" (step SF11: YES) without the housing 8 moving "rightward" or "leftward" (step SF10: NO), the terminal The control unit 20 shifts the processing procedure to step SF2. As a result, when the orientation of the housing 8 of the mobile terminal 3 changes from "horizontal" to "vertical", the bird's-eye view image display screen GM2a is displayed on the touch panel 9 instead of the vehicle exterior image display screen GM2b. The effect of this is as described in the description of the first mode.

On the other hand, when the housing 8 does either "right movement" or "left movement" without being "vertical" (step SF11: NO) (step SF10: YES), the terminal The control unit 20 shifts the processing procedure to step SC10. That is, the terminal control unit 20 shifts the processing procedure to step SF12 when the housing 8 moves “rightward” or “leftward” when the housing 8 is oriented “sideways”.

In step SF12, the terminal control unit 20 determines whether the movement of the housing 8 is “right movement” or “left movement”.
When the movement of the housing 8 is "right movement" (step SF12: "right movement"), the terminal control unit 20 is used for outside-vehicle shooting according to the first order according to the direction in which the outside-vehicle shooting camera shoots. Among the cameras, one camera for out-of-vehicle photography is selected (step SF13).
Next, the terminal control unit 20 transmits sideways information including information indicating that the orientation of the mobile terminal 3 is “sideways” and information indicating the camera for outside-vehicle photography selected in step SF13 to the information processing device 4 ( Step SF14). After the processing of step SF14, the terminal control unit 20 shifts the processing procedure to step SF10.
In response to the transmission of the landscape information in step SF14, the information processing device 4 executes the process of the flowchart FI of FIG. 17C, and the vehicle exterior image G2 to be displayed on the display device 32 is captured outside the vehicle indicated by the selected camera information included in the landscape information. Change to an image based on the shooting result of the camera. Further, the information processing device 4 starts transmitting the vehicle outside image data to the mobile terminal 3 based on the shooting result of the vehicle outside shooting camera indicated by the selected camera information. As a result, the vehicle exterior image G2 displayed on the vehicle exterior image display screen GM2b of the touch panel 9 of the mobile terminal 3 becomes an image based on the shooting result of the vehicle exterior shooting camera selected in step SF13.

On the other hand, in step SF12, when the movement of the housing 8 is “left movement” (step SF12: “left movement”), the terminal control unit 20 has a second second according to the direction in which the camera for shooting outside the vehicle shoots. According to the order, one camera for outside the vehicle is selected from the cameras for outside the vehicle (step SF15).
Next, the terminal control unit 20 transmits the sideways information including the information indicating that the orientation of the mobile terminal 3 is “landscape” and the information indicating the out-of-vehicle camera selected in step SF15 to the information processing device 4 ( Step SF16). After the processing of step SF16, the terminal control unit 20 shifts the processing procedure to step SF10.
In response to the transmission of the landscape information in step SF16, the information processing device 4 executes the process of the flowchart FI of FIG. 17C, and the vehicle exterior image G2 to be displayed on the display device 32 is captured outside the vehicle indicated by the selected camera information included in the landscape information. Change to an image based on the shooting result of the camera. Further, the information processing device 4 starts transmitting the vehicle outside image data to the mobile terminal 3 based on the shooting result of the vehicle outside shooting camera indicated by the selected camera information. As a result, the vehicle exterior image G2 displayed on the vehicle exterior image display screen GM2b of the touch panel 9 of the mobile terminal 3 becomes an image based on the shooting result of the vehicle exterior shooting camera selected in step SF16.

The second mode has been described above. In the second mode, the amount of data transmitted by the information processing device 4 to the mobile terminal 3 is smaller and the communication efficiency is higher than in the first mode described above.

Next, regarding the process related to the display on the touch panel 9 in the vehicle propulsion control process, another process executed by the mobile terminal 3 will be described.
As described above, after shifting to the vehicle propulsion control mode, the vehicle control screen GM2 (the bird's-eye view image display screen GM2a or the vehicle exterior image display screen GM2b) is displayed on the touch panel 9 of the mobile terminal 3. Then, the propulsion control button QB is displayed on the vehicle control screen GM2. As described above, the propulsion control button QB is a button that controls the propulsion of the vehicle S, and while the button is touch-operated, the vehicle S is propelled along the track and the button is not touch-operated. Meanwhile, the propulsion of the vehicle S is stopped.

Then, the terminal control unit 20 moves the touch operation position when the touch operation position moves while the touch panel 9 is in contact after the propulsion control button QB displayed on the vehicle control screen GM2 is touch-operated. The position of the propulsion control button QB is moved according to. As a result, even if the position of the touch operation is moved while the touch panel 9 is in contact after the propulsion control button QB is touch-operated, the state in which the propulsion control button QB is touch-operated is maintained.
As described above, the state J4 of FIG. 12 shows the vehicle control screen GM2 immediately after the screen displayed on the touch panel 9 is switched from the lock screen GM1 to the vehicle control screen GM2. The state J5 in FIG. 12 shows the vehicle control screen GM2 after the position of the touch operation is moved from the position of the touch operation in the state J4. As shown in the state J5 of FIG. 12, the position of the propulsion control button QB moves following the touch position in response to the movement of the touch position, and the state in which the propulsion control button QB is touch-operated is maintained.
In the state J6 of FIG. 12, the direction of the mobile terminal 3 is set to be horizontal from the state J5, and the vehicle exterior image display screen GM2b is displayed on the touch panel 9 instead of the bird's-eye view image display screen GM2a displayed in the state J5. It shows how it was done. At the time of transition from the state J5 to the state J6, it is assumed that the user maintains the state in which the finger touching the propulsion control button QB in the state J5 is in contact with the touch panel 9. As shown in the states J5 and J6, the terminal control unit 20 displays the state in which the propulsion control button QB is displayed even when the screen to be displayed on the touch panel 9 is switched from the bird's-eye view image display screen GM2a to the vehicle exterior image display screen GM2b. At the same time, the position of the propulsion control button QB is moved according to the movement to the position of the touch operation, and the state in which the propulsion control button QB is touch-operated is maintained.

When the vehicle control screen GM2 is displayed, the terminal control unit 20 performs the above processing, so that the following effects are obtained.
That is, even if the position of the finger to be touch-operated shifts after the user touch-operates the propulsion control button QB, the state in which the propulsion control button QB is touch-operated is maintained. Therefore, when operating the propulsion control button QB, the user does not have to keep touching a specific position with a pinpoint, which is highly convenient for the user. In particular, as described above, in the present embodiment, the user may change the orientation of the housing 8 of the mobile terminal 3 while the vehicle is automatically moved. Then, even if the touch-operated position shifts due to the change in the orientation of the housing 8, the propulsion control button QB is maintained in the touch-operated state, which is highly convenient for the user. When the housing 8 is gripped with one hand and the touch panel 9 is operated with one hand, it is assumed that the touch operation position is likely to shift. Therefore, the user grips the housing 8 with one hand and the touch panel 9 with one hand. In the case of operation, the above-mentioned processing is more effective.

Further, when the touch operation of the propulsion control button QB is released, the terminal control unit 20 continuously displays the propulsion control button QB at the position of the propulsion control button QB when the touch operation is released. According to this configuration, when the propulsion control button QB is touch-operated again, the user only needs to touch the position when the touch operation is released, which is highly convenient for the user.
When the touch operation of the propulsion control button QB is released, the terminal control unit 20 may execute the following processing instead of the above processing. That is, when the touch operation of the propulsion control button QB is released, the propulsion control button QB may be displayed at a predetermined specific position. The predetermined specific position is, for example, a position corresponding to the right end BR1b of the slide bar BR1 on the lock screen GM1. According to this configuration, when the touch operation on the propulsion control button QB is released, the propulsion control button QB is displayed at the same position, so that the user can easily find the position of the propulsion control button QB and the user. Convenience is improved.

Further, the terminal control unit 20 changes the image showing the propulsion control button QB depending on whether the propulsion control button QB is touch-operated or not.
The propulsion control button QB in the states J4 to J6 in FIG. 12 is in a touch-operated state, and the propulsion control button QB in the state J7 in FIG. 12 is in a non-touch-operated state.
As shown in the states J4 to J6, when the propulsion control button QB is touch-operated, the image showing the propulsion control button QB is composed of a circular image and a circular image surrounding the circular image. It is composed. The ring-shaped image may be an image whose mode changes with the passage of time due to animation. On the other hand, as shown in the state J7, when the propulsion control button QB is not touch-operated, the image showing the propulsion control button QB is a circular image and does not include an annular image.
By visually recognizing the image showing the propulsion control button QB, the user can easily and accurately recognize that the propulsion control button QB is touch-operated and that the propulsion control button QB is not touch-operated.

Further, as shown in the states J4 and J5 of FIG. 12, the terminal control unit 20 superimposes the state of the gear of the vehicle S on the image showing the vehicle S when the bird's-eye view image display screen GM2a is displayed. Display the indicated mark. In the present embodiment, when the gear state of the vehicle S is parking (P), the terminal control unit 20 superimposes and displays a mark representing the character "P" on the image indicating the vehicle S, and reverses (R). In the case of, the mark representing the character "R" is superimposed and displayed on the image indicating the vehicle S, and in the case of the drive (D), the mark representing the character "D" is superimposed and displayed on the image indicating the vehicle S. However, in the case of neutral (N), the mark representing the character "N" is superimposed and displayed on the image indicating the vehicle S, and in the case of low (L), the mark representing the character "L" indicates the vehicle S. Display by superimposing on the image. The terminal control unit 20 communicates with the information processing device 4 to recognize the state of the gear of the vehicle S.
The user can accurately recognize the state of the gear of the vehicle S by referring to the mark displayed superimposed on the image showing the vehicle S.

As shown in FIG. 9, the vehicle S moves to the parking position and stops at the parking position by the automatic vehicle driving, so that the vehicle propulsion control process of the flowchart FA and the vehicle propulsion control process of the flowchart FB are completed. When the vehicle S stops at the vehicle position, the user is notified by a predetermined method. A user outside the vehicle can leave the vicinity of the vehicle S by turning off the engine switch and locking the door by using the dedicated remote controller described above.

In the first mode and the second mode, the composite image data, the bird's-eye view image data, and the vehicle exterior image data transmitted by the information processing device 4 to the mobile terminal 3 are "in the shooting result of the camera provided in the vehicle S, respectively. Corresponds to "based shooting data".

<When leaving vehicle S>
Next, a case where the vehicle S is delivered will be described.
The flowchart FJ of FIG. 18 is a flowchart showing the operation of the information processing device 4, and the flowchart FK is a flowchart showing the operation of the mobile terminal 3.
Hereinafter, the operations of the information processing device 4 and the mobile terminal 3 will be described by taking as an example the case where the vehicle S is discharged from the parking position corresponding to one parking area of the parking lot.

The user goes to the vicinity of the parking position when leaving the garage by automatic vehicle movement. The user may board the vehicle S, but for convenience of explanation, the user will be described below assuming that the user gets off the vehicle S.
Next, the user operates the dedicated remote controller described above to turn on the engine switch of the vehicle S. When the engine switch of the vehicle S is turned on, electric power is supplied to the information processing device 4 from the accessory power supply, and the power of the information processing device 4 is turned on.
Next, the user operates the touch panel 9 of the mobile terminal 3 to activate the dedicated application AP (step S3). When the user activates the dedicated application AP, the orientation of the housing 8 of the mobile terminal 3 is set to "vertical orientation".
The terminal control unit 20 of the mobile terminal 3 executes the processing of the flowchart FK by the function of the dedicated application AP started by the user in step S3.
Further, the user performs a predetermined operation on the touch panel 9 to give an instruction to display the delivery start instruction screen on the touch panel 9 (step S4).
As shown in the flowchart FK of FIG. 18, in response to the instruction in step S4, the terminal control unit 20 causes the touch panel 9 to display the delivery start instruction screen (step SK1).
The warehousing start instruction screen is a screen on which an instruction to perform warehousing by automatic vehicle movement can be input, and a touch operation button that is touch-operated when the instruction is given is displayed. The user operates the touch operation button displayed on the warehousing start instruction screen when warehousing is performed by the vehicle automatic Ito (step S5).

When it is detected that the touch operation button on the delivery start instruction screen has been operated by the user, the terminal control unit 20 transmits the delivery start instruction information to the information processing device 4 (step SK2).

As shown in the flowchart FJ of FIG. 18, the information processing device control unit 30 of the information processing device 4 receives the delivery start instruction information (step SJ1).
Next, the information processing device control unit 30 changes the operation mode of the information processing device 4 to the delivery mode in response to the reception of the delivery start instruction information (step SJ2).

The information processing apparatus control unit 30 executes the candidate position determination process according to the shift of the operation mode to the delivery mode (step SJ3).
The candidate position determination process is a process of determining one or a plurality of positions where the vehicle S can move the parking position and stop. One or a plurality of positions determined by the candidate position determination process are candidates for positions at which the vehicle S is stopped by leaving the vehicle. Hereinafter, each of the positions determined by the candidate position determination process is expressed as "candidate position".
In the candidate position determination process of step SJ3, the information processing apparatus control unit 30 performs the same process as the parking position determination process described above to determine one or a plurality of candidate positions. In the candidate position determination process, any method may be used for the process of determining the candidate position.

After determining one or more candidate positions in step SJ3, the information processing apparatus control unit 30 executes the candidate position route information generation process (step SJ4).
The candidate position route information generation process is a process of generating automatic driving route information including track information indicating a track from the parking position to the candidate position determined in step SJ3. Hereinafter, the automatic driving route information including the track information indicating the track from the parking position to the candidate position is referred to as "candidate position route information". In step SJ4, when the information processing apparatus control unit 30 determines a plurality of candidate positions in the candidate position determination process of step SJ3, the information processing apparatus control unit 30 generates a plurality of candidate position route information corresponding to the plurality of candidate positions.

Next, the information processing apparatus control unit 30 transmits each of the candidate position route information generated in step SJ4 to the mobile terminal 3 (step SJ5).
Further, the information processing apparatus control unit 30 starts generating the bird's-eye view image data based on the input from the bird's-eye view camera and transmitting the bird's-eye view image data to the mobile terminal 3 (step SJ6). The information processing device control unit 30 generates and transmits the bird's-eye view image data at a cycle corresponding to the predetermined cycle based on the captured image data input from the bird's-eye view camera at a predetermined cycle.

As shown in the flowchart FK of FIG. 18, the terminal control unit 20 of the mobile terminal 3 receives one or more candidate position route information transmitted by the information processing apparatus 4 in step SJ5 (step SK3). Further, the terminal control unit 20 starts receiving the bird's-eye view image data (step SK4).

Next, the terminal control unit 20 causes the touch panel 9 to display the selection screen GM3 based on each of the candidate position route information received in step SK3 and the bird's-eye view image data that started reception in step SK4 (step SK5).
After displaying the selection screen GM3, the terminal control unit 20 monitors whether or not there is an input for selecting a candidate position on the selection screen GM3 (step SK6).
Hereinafter, the content of the selection screen GM3 and the processing of step SK6 will be described in detail.

FIG. 19 is a diagram showing an example of the selection screen GM3.
The selection screen GM3 is a screen in which the user inputs to select one candidate position among the candidate positions.
As shown in FIG. 19, the bird's-eye view image G1 is displayed on the selection screen GM3. The terminal control unit 20 displays the bird's-eye view image G1 based on the bird's-eye view image data received from the information processing device 4. At the center of the bird's-eye view image G1 displayed on the selection screen GM3, the current position image X1 showing the vehicle S is displayed. The current position image X1 is an image showing the current position of the vehicle S.

As shown in FIG. 19, the candidate position image X2 is displayed on the selection screen GM3. The candidate position image X2 is an image showing a candidate position (a candidate for a position where the vehicle S stops due to leaving the vehicle). When there are a plurality of candidate positions (when a plurality of candidate position route information is received), a plurality of candidate position images X2 are displayed.
In the selection screen GM3 illustrated in FIG. 19, the candidate position image X2 is an image of a rectangular frame. The shape of the candidate position image X2 is not limited to the shape shown in FIG.
The candidate position image X2 is displayed at a position corresponding to the actual candidate position in the real space on the selection screen GM3. Therefore, the positional relationship between the current position image X1 and the candidate position image X2 on the selection screen GM3 is in a state corresponding to the positional relationship between the current position of the vehicle S in the real space and the candidate position.
The terminal control unit 20 superimposes the bird's-eye view image G1 on the candidate position route information and displays one or a plurality of candidate position images X2.
The user can accurately recognize the positional relationship between the vehicle S and the candidate position based on the candidate position image X2 displayed on the selection screen GM3.

As shown in FIG. 19, on the selection screen GM3, the slide unit SU2 is displayed below the current position image X1. The slide unit SU2 has one or more slide bars BR2 and one slide button SB2.
The number of slide bars BR2 displayed on the selection screen GM3 is the same as the number of candidate position images X2. As shown in FIG. 19, each of the slide bar BR2 is a strip-shaped object extending from the common base end PP1 located below the current position image X1 toward each of the candidate position images X2. For example, as illustrated in FIG. 19, it is assumed that three candidate position images X2, a first image, a second image, and a third image, are displayed as candidate position images X2 on the selection screen GM3. In this case, "3" slide bars BR2 are displayed on the selection screen GM3. When the three slide bars BR2 are expressed as the first bar, the second bar, and the third bar, respectively, the first bar is a strip-shaped object extending from the base end PP1 toward the first image. The second bar is a band-shaped object extending from the base end PP1 toward the second image, and the third bar is a band-shaped object extending from the base end PP1 toward the third image.

The slide button SB2 is a touch operation button, and can be moved along the slide bar BR2 (when a plurality of slide bars BR2 are displayed, all the slide bars BR2) according to the swipe operation of the user. .. As shown in FIG. 19, the slide button SB2 is maintained in a state of being positioned at the base end PP1 when no touch operation is performed by the user. The slide button SB2 can be moved from the base end PP1 of the slide bar BR2 within the range of the tip end PP2.

FIG. 20 is a diagram for explaining the transition of the screen displayed on the touch panel 9. FIG. 20 shows a part of a user's finger that operates the screen together with the screen displayed on the touch panel 9 as appropriate.
The state J10 in FIG. 20 shows the selection screen GM3 in a state in which the user is not performing a touch operation. As described above, the selection screen GM3 is a screen in which the user inputs to select one candidate position among the candidate positions.
When inputting to select a candidate position, the user touches the slide button SB2 located at the base end PP1 with a finger on the selection screen GM3 of the state J10.
Next, the user swipes to move the slide button SB2 along the slide bar BR2 extending toward the candidate position image X2 of the candidate position where the vehicle S is desired to be stopped by leaving the slide bar BR2. .. In response to the swipe operation, the terminal control unit 20 follows the position where the finger touches and moves the slide button SB2 along the slide bar BR2. The state J11 in FIG. 12 shows the selection screen GM3 in which the slide button SB2 is moved along the slide bar BR2 by the swipe operation.

As shown in the state J11 of FIG. 12, in response to the swipe operation, the track image X3 showing the track through which the vehicle S passes is displayed on the selection screen GM3 when the vehicle S moves from the parking position to the candidate position. .. The track image X3 is a band-shaped image connecting the current position image X1 and the candidate position image X2 along the track when the vehicle S moves to the candidate position, and has a width corresponding to the width of the vehicle S.
The terminal control unit 20 superimposes the bird's-eye view image G1 on the bird's-eye view image G1 based on the candidate position route information, and displays one or a plurality of trajectory images X3.
Based on the track image X3 displayed on the selection screen GM3, the user can accurately recognize the track (route) when the vehicle S moves to the candidate position by the vehicle automatic movement.

The user who performs the swipe operation moves the slide button SB2 to the tip PP2 of the slide bar BR2, and keeps the slide button SB2 positioned at the tip PP2 for a predetermined period (for example, 3 seconds). This completes the input for selecting the candidate position.
If the swipe operation is released before the input of the instruction to shift the candidate position to the vehicle propulsion control mode is completed, the terminal control unit 20 bases the slide button SB2 in response to the release of the swipe operation. Move to end PP1.

In step SK6, the terminal control unit 20 monitors whether or not the slide button SB2 is positioned on the tip PP2 of the slide bar BR2 for a predetermined period based on the input from the touch panel 9. Then, the terminal control unit 20 determines that when the slide button SB2 is positioned at the tip PP2 of the slide bar BR2 for a predetermined period of time, there is an input for selecting a candidate position.

When it is determined in step SK6 that there is an input for selecting a candidate position (step SK6: YES), the terminal control unit 20 determines that the slide button SB2 is a candidate corresponding to the slide bar BR2 located at the tip PP2 for a predetermined period. The candidate position of the position image X2 is determined as a position to stop the vehicle S after leaving the warehouse (step SK7).

As described above, in the present embodiment, the user can stop the vehicle S from the candidate positions after the vehicle S has been delivered by a simple operation of moving the slide button SB2 along the desired slide bar BR2. You can determine the desired candidate position. Therefore, the convenience of the user is high. In particular, since each of the slide bar BR2 extends toward each of the candidate position images X2, the user can intuitively know which slide bar BR2 corresponds to which candidate position image X2. I can grasp it.
Further, in the present embodiment, the input is completed only when the slide button SB2 is positioned at the tip PP2 of the slide bar BR2 for a predetermined period of time. This configuration prevents the user's finger from accidentally touching the touch panel 9 and selecting a candidate position against the user's intention.

After determining the position to stop the vehicle S in step SK7, the terminal control unit 20 shifts the operation mode to the vehicle propulsion control mode (step SK8).
The terminal control unit 20 switches the screen to be displayed on the touch panel 9 from the selection screen GM3 to the vehicle control screen GM2 according to the transition of the operation mode (step SK9).

The state J12 of FIG. 20 shows the vehicle control screen GM2 immediately after the screen displayed on the touch panel 9 is switched from the selection screen GM3 to the vehicle control screen GM2. Immediately after the screen is switched to the vehicle control screen GM2, the orientation of the housing 8 of the mobile terminal 3 is "vertical orientation".
The vehicle control screen GM2 displayed at the time of leaving the garage is the same as the vehicle control screen GM2 displayed at the time of parking. When switching the screen from the selection screen GM3 to the vehicle control screen GM2, the terminal control unit 20 switches the slide button SB2 on the selection screen GM3 to the propulsion control button QB (operation button). That is, the terminal control unit 20 does not display the propulsion control button QB again as a button different from the slide button SB2 in response to the screen switching, but instead displays the slide button SB2 located at the tip PP2 of the slide bar BR2 on the screen. It functions as a propulsion control button QB without changing the position according to the switching. Therefore, immediately after the screen is switched, the propulsion control button QB is located at the tip PP2 of the slide bar BR2 on the selection screen GM3, and the propulsion control button QB is in a state of being touch-operated by the user.
Instead of the state J12 of FIG. 20, the vehicle control screen GM2 shown in the state J13 of FIG. 20 may be displayed. In the vehicle control screen GM2 shown in the state J13, the track image X3 from the parking position of the vehicle S toward the selected candidate position has the shape of an arrow clearly indicating the traveling direction. By displaying the vehicle control screen GM2 in the mode shown in the state J13, the user can accurately recognize the traveling direction of the vehicle S after leaving the parking position.

In response to the shift to the vehicle propulsion control mode, the terminal control unit 20 transmits information indicating the candidate position determined in step SK7 to the information processing apparatus 4 (step SK10).
As shown in the flowchart FJ of FIG. 18, the information processing device control unit 30 of the information processing device 4 receives information indicating a candidate position (step SJ7).

As shown in the flowchart FJ and the flowchart FK of FIG. 18, after the operation mode of the mobile terminal 3 shifts to the vehicle propulsion control mode, the information processing device control unit 30 of the information processing device 4 and the terminal control of the mobile terminal 3 The unit 20 cooperates to execute the vehicle propulsion control process (step SJ8, step SK11).
In the vehicle propulsion control process at the time of leaving the garage, the same process as the vehicle propulsion control process at the time of parking is performed. The information processing device control unit 30 outputs the automatic driving route information corresponding to the candidate position indicated by the information received in step SJ7 to the vehicle drive control device 5.

By automatic vehicle driving, the vehicle S moves from the parking position to the selected candidate position (stop position after leaving the vehicle) and stops at the selected candidate position, whereby the vehicle propulsion control process of the flowchart FJ and the flowchart FK Vehicle propulsion control process is completed. When the vehicle S stops at the selected candidate position, the user is notified by a predetermined method. A user outside the vehicle can board the vehicle S by unlocking the door using the dedicated remote controller described above.

By the way, there are a plurality of patterns of warehousing depending on the state of the vehicle S parked at the parking position.
The terminal control unit 20 changes the display mode of the candidate position image X2 and the slide unit SU2 on the selection screen GM3 according to the delivery pattern. For example, the terminal control unit 20 changes the display mode of the candidate position image X2 and the slide unit SU2 on the selection screen GM3 according to the delivery pattern by the following method. That is, in advance, the terminal storage unit 22 stores image data for displaying the candidate position image X2 and the slide unit SU2 in a mode corresponding to the delivery pattern for each delivery pattern. Then, when the selection screen GM3 is displayed on the touch panel 9, the terminal control unit 20 specifies the delivery pattern, reads out the image data corresponding to the specified delivery pattern from the terminal storage unit 22, and displays the image as necessary. The data is processed, and the selection screen GM3 is displayed using the processed image data. According to this configuration, the selection screen GM3 can be displayed using the image data prepared in advance, so that the processing efficiency can be improved.
As the delivery pattern, for example, there are the following first to sixth patterns.

In the first pattern, the vehicle S leaves the garage when it "forwards", and the candidate positions where the vehicle S can stop after the warehousing are the position in front of the vehicle S, the position on the front right side of the vehicle S, and the front of the vehicle S. There are three patterns with the position on the left side. For example, when the vehicle S is parked in a parking position having a large space in front of it, the delivery pattern is the first pattern.
The selection screen GM3 illustrated in FIGS. 19 and 20 is a selection screen GM3 corresponding to the first pattern.

In the second pattern, the vehicle S leaves the garage when it "reverses", and the candidate positions where the vehicle S can stop after the warehousing are the position behind the vehicle S, the position on the rear right side of the vehicle S, and the rear of the vehicle S. There are three patterns with the position on the left side. For example, when the vehicle S is parked in a parking position having a large space behind it, the delivery pattern is the second pattern.
The state J20 in FIG. 21 shows the selection screen GM3 corresponding to the second pattern.

In the third pattern, the vehicle S leaves the garage when it "forwards", and there are two candidate positions where the vehicle S can stop after the warehousing: the position on the front right side of the vehicle S and the position on the front left side of the vehicle S. It is a pattern to do. For example, when the vehicle S is parallel parked and there are spaces on the front right side and the front left side of the vehicle S where the vehicle S can stop, the warehousing pattern is the third pattern.
The state J21 in FIG. 21 shows the selection screen GM3 corresponding to the third pattern.

In the fourth pattern, the vehicle S leaves the garage when it "reverses", and there are two candidate positions where the vehicle S can stop after the warehousing: the position on the rear right side of the vehicle S and the position on the rear left side of the vehicle S. It is a pattern to do. For example, when the vehicle S is parallel parked and there are spaces on the rear right side and the rear left side of the vehicle S where the vehicle S can stop, the warehousing pattern is the fourth pattern.
The state J22 in FIG. 21 shows the selection screen GM3 corresponding to the fourth pattern.

In the fifth pattern, it is possible for the vehicle S to leave the vehicle regardless of whether it is "forward" or "backward", and the candidate positions where the vehicle S can stop after leaving the vehicle are the position on the front right side of the vehicle S and the front left side of the vehicle S. There are four patterns: the position of, the position on the rear right side of the vehicle S, and the position on the rear left side of the vehicle S. For example, when the vehicle S is parallel parked and there are spaces in the front right side, the front left side, the rear right side, and the rear left side of the vehicle S where the vehicle S can stop, the warehousing pattern is the fifth pattern.
The state J23 in FIG. 21 shows the selection screen GM3 corresponding to the fifth pattern.

In the sixth pattern, it is possible for the vehicle S to leave the vehicle regardless of whether it is "forward" or "backward", and the candidate positions where the vehicle S can stop after leaving the vehicle are the position in front of the vehicle S and the right side in front of the vehicle S. There are six patterns: a position, a position on the front left side of the vehicle S, a position behind the vehicle S, a position on the rear right side of the vehicle S, and a position on the rear left side of the vehicle S. For example, when the vehicle S is parked in a parking position where there is a large space in front and behind, the delivery pattern is the sixth pattern.
The state J24 in FIG. 21 shows the selection screen GM3 corresponding to the sixth pattern.

By the way, when there are a plurality of candidate positions of the vehicle S, there may be a candidate position in the candidate positions where the movement of the vehicle S is actually prohibited due to the existence of obstacles or regulations.
In such a case, the terminal control unit 20 executes the following processing when displaying the selection screen GM3.
That is, the terminal control unit 20 displays an image indicating that movement to the candidate position is prohibited on the slide bar BR2 corresponding to the candidate position where movement is prohibited, and corresponds to the candidate position where movement is prohibited. The movement of the slide button SB2 along the slide bar BR2 is prohibited.

The state J30 of FIG. 22 is displayed by the terminal control unit 20 on the selection screen GM3 of FIG. 19 when the movement to the candidate position corresponding to the slide bar BR2 extending from the base end PP1 toward the upper left side is prohibited. An example of the selection screen GM3 to be selected is shown.
As shown in the state J30 of FIG. 22, on the selection screen GM3, an image indicating that the movement to the candidate position is prohibited is displayed on the slide bar BR2 corresponding to the candidate position where the movement is prohibited. Further, the candidate position image X2 corresponding to the candidate position where the movement is prohibited also displays an image indicating that the movement is prohibited.
When there is a candidate position where movement is prohibited, the user can accurately recognize the candidate position where movement is prohibited by performing the above processing. In addition, it is possible to prevent the user from selecting a candidate position where movement is prohibited.

In addition, when there is a candidate position where movement is prohibited, the terminal control unit 20 may execute the following processing instead of the above processing.
That is, the terminal control unit 20 prohibits the display of the slide bar BR2 corresponding to the candidate position where the movement is prohibited.
The state J31 of FIG. 22 is displayed by the terminal control unit 20 on the selection screen GM3 of FIG. 19 when the movement to the candidate position corresponding to the slide bar BR2 extending from the base end PP1 toward the upper left side is prohibited. An example of the selection screen GM3 to be selected is shown.
As shown in the state J31 of FIG. 22, the slide bar BR2 corresponding to the candidate position where the movement is prohibited is not displayed on the selection screen GM3. This can prevent the user from selecting a candidate position where movement is prohibited.

<First modification>
Next, the first modification will be described.
In the above-described embodiment, the slide unit SU2 selects one candidate position from the candidate positions at the time of delivery.
Regarding this configuration, the candidate position image X2 may be made to function as a touch operation button, and the candidate position image X2 may be touch-operated so that the candidate position can be selected.
In this case, if the candidate position image X2 is touch-operated a plurality of times (for example, twice) in succession, the selection of the candidate position is confirmed, the user mistakenly makes the candidate position image X2. It is possible to prevent the candidate position corresponding to the erroneously operated candidate position image X2 from being selected when the touch operation is performed.
When the candidate position image X2 is touch-operated to select the candidate position, the terminal control unit 20 shifts the operation mode to the vehicle control mode and displays the vehicle control screen GM2 instead of the selection screen GM3. You may. Further, when the candidate position image X2 is touch-operated to select the candidate position, the terminal control unit 20 does not immediately shift the operation mode to the vehicle control mode, and instead of the selection screen GM3, the lock screen GM1 is used. When the display is performed and a predetermined input is made to the lock screen GM1, the operation mode may be changed to the vehicle control mode and the vehicle control screen GM2 may be displayed instead of the lock screen GM1.

<Second modification>
Next, a second modification will be described.
In the above-described embodiment, the processing at the time of parking has been described on the premise that there is only one parking position. However, there may be a configuration in which one parking position is selected from a plurality of parking position candidates, and the selected parking position candidate is determined as a position for parking the vehicle S.
In this case, the terminal control unit 20 may display a screen similar to the selection screen GM3 on the touch panel 9 before displaying the lock screen GM1 on the touch panel 9, and determine the parking position based on the operation on the touch panel 9. ..

As described above, the mobile terminal 3 (terminal) according to the present embodiment has a housing 8, a touch panel 9 (display panel) provided on the housing 8, and a housing state for detecting the state of the housing 8. It includes a detection unit 23 and a terminal control unit 20. The terminal control unit 20 acquires shooting data based on the shooting results of the camera provided in the vehicle S, displays an image based on the acquired shooting data on the touch panel 9, and has a housing detected by the housing state detection unit 23. The image to be displayed on the touch panel 9 is switched according to the state of the body 8.
According to this configuration, the user can switch the image based on the captured image data displayed on the touch panel 9 by performing a simple operation of changing the state of the housing 8 of the mobile terminal 3, and the vehicle. When displaying an image based on the shooting result of the camera provided in S on the mobile terminal 3, the convenience of the user can be improved.

Further, in the present embodiment, the terminal control unit 20 detects that the orientation of the housing 8 is the first orientation by the housing state detection unit 23, and the orientation of the housing 8 is the first. The image to be displayed on the touch panel 9 is switched between the case where it is detected that the second orientation is different from the orientation.
According to this configuration, the user performs a simple operation of switching the orientation of the housing of the mobile terminal 3 to the first orientation or the second orientation to obtain the captured image data displayed on the touch panel 9. The image based on the image can be switched, and the convenience of the user can be improved when the image based on the shooting result of the camera provided in the vehicle S is displayed by the mobile terminal 3.

Further, in the present embodiment, the housing 8 is a plate-shaped member having a rectangular front surface 8a. The touch panel 9 is provided on the front surface 8a of the housing 8. The first orientation is such that the direction of the long side of the front surface 8a of the housing 8 is along the vertical direction, and the second orientation is the direction of the long side of the front surface 8a of the housing 8. This is the direction in which the state is along the direction orthogonal to the vertical direction.
According to this configuration, the user can set the orientation of the housing 8 of the mobile terminal 3 to "the direction in which the long side of the front surface 8a of the housing 8 is in the vertical direction" or "the orientation of the housing 8". Switching the image based on the captured image data displayed on the touch panel 9 by performing a simple operation of switching the direction of the long side of the front surface 8a to the direction in which the direction is perpendicular to the vertical direction. This makes it possible to improve the convenience of the user when displaying an image based on the shooting result of the camera provided in the vehicle S on the mobile terminal 3.

Further, in the present embodiment, when the housing state detection unit 23 detects that the housing 8 is in the first orientation, the terminal control unit 20 views the vehicle S as an image based on the shooting data. When the bird's-eye view image G1 which is the image obtained is displayed on the touch panel 9 and the housing state detection unit 23 detects that the housing 8 is in the second direction, the camera captures the image as an image based on the shooting data. The vehicle exterior image G2, which is a photographed vehicle exterior image, is displayed on the touch panel 9.
According to this configuration, the user performs a simple operation of changing the orientation of the housing 8 of the mobile terminal 3 to move the image displayed on the touch panel 9 between the bird's-eye view image G1 and the vehicle exterior image G2. You can switch with.

Further, in the present embodiment, the vehicle S is provided with a plurality of cameras for taking a picture of the outside of the vehicle. When the housing state detection unit 23 detects that the housing 8 is in the second direction, the terminal control unit 20 takes a plurality of out-of-vehicle shots according to a predetermined priority corresponding to the state of the vehicle S. One of the cameras for outside the vehicle is selected, and the outside image G2 based on the shooting result of the selected camera for outside the vehicle is displayed on the touch panel 9.
According to this configuration, the vehicle exterior image G2, which is likely to be needed by the user, can be displayed on the touch panel 9, which is highly convenient for the user.

Here, the predetermined priority is, for example, a higher priority as the camera for outside-vehicle photography capable of photographing the traveling direction of the vehicle S.
According to this configuration, the vehicle exterior image G2, which is likely to be needed by the user, can be displayed on the touch panel 9, which is highly convenient for the user.

Further, the predetermined priority is, for example, a higher priority as the camera for outside-vehicle photography capable of photographing an object close to the vehicle S.
According to this configuration, the vehicle exterior image G2, which is likely to be needed by the user, can be displayed on the touch panel 9, which is highly convenient for the user.

Further, in the present embodiment, the terminal control unit 20 displays the outside image G2 of 1 based on the shooting result of the camera for outside vehicle 1 on the touch panel 9, and then the housing 8 is determined by the housing state detection unit 23. When it is detected that the movement of the vehicle is detected, the touch panel 9 replaces the vehicle exterior image G2 of 1 with another vehicle exterior image G2 based on the shooting result of another vehicle exterior camera different from the vehicle exterior camera of 1. To display.
According to this configuration, the user can switch the outside image G2 displayed on the touch panel 9 by performing a simple operation of causing the housing 8 to perform a predetermined movement, which is highly convenient for the user.

Further, in the present embodiment, the plurality of cameras for external photography photograph the outside of the vehicle in different directions. The predetermined movement is a first movement in which the state of the housing 8 changes in the first aspect, or a second movement in which the state of the housing 8 changes in a second mode different from the first aspect. It is one of the movements. When the housing state detection unit 23 detects that the housing 8 has made the first movement, the terminal control unit 20 displays the vehicle exterior image G2 in the first order according to the shooting direction of the vehicle exterior shooting camera. Switch. When the housing state detection unit 23 detects that the housing 8 has made a second movement, the terminal control unit 20 is in the order according to the direction in which the camera for shooting outside the vehicle shoots, and is in the first order. The outside image G2 is switched in the second order different from the above.
According to this configuration, the user can switch the images in the order desired by the user, which is highly convenient for the user.

Further, the terminal control unit 20 displays an image based on the shooting data on the touch panel 9 and displays a touch-operable propulsion control button QB (operation button) on the touch panel 9 by superimposing the image based on the shooting data on the touch panel 9. Even when the image to be displayed is switched, the display of the propulsion control button QB is maintained. While the propulsion control button QB is touch-operated, the terminal control unit 20 transmits a propulsion instruction signal instructing the propulsion of the vehicle S to the information processing device 4 provided in the vehicle S.
According to this configuration, the user can switch the image to be displayed on the touch panel 9 while controlling the propulsion of the vehicle S by operating the propulsion control button QB.

Further, as described above, the mobile terminal 3 according to the present embodiment has a touch panel 9 for displaying an image and receiving a touch operation, and a terminal control unit 20. The terminal control unit 20 displays the vehicle control screen GM2 on which the touch-operable propulsion control button QB is displayed on the touch panel 9, and touches the propulsion control button QB displayed on the vehicle control screen GM2 after being touch-operated. When the position of the operation is moved, the position of the propulsion control button QB is moved following the movement of the position of the touch operation to maintain the state in which the propulsion control button QB is touch-operated. The terminal control unit 20 transmits a propulsion instruction signal instructing the propulsion of the vehicle S to the information processing device 4 provided in the vehicle S while the propulsion control button QB displayed on the vehicle control screen GM2 is touch-operated. Then, the transmission of the propulsion instruction signal is stopped in response to the release of the touch operation on the propulsion control button QB.
According to this configuration, even if the position of the finger to be touch-operated shifts after the user touch-operates the propulsion control button QB, the state in which the propulsion control button QB is touch-operated is maintained. Therefore, when operating the propulsion control button QB, the user does not have to keep touching a specific position with a pinpoint. Therefore, when the user operates the touch panel 9 of the mobile terminal 3 to control the propulsion of the vehicle S, the operation can be accurately performed.

Further, in the present embodiment, when the touch operation for the propulsion control button QB displayed on the vehicle control screen GM2 is released, the terminal control unit 20 sets the position of the propulsion control button QB when the touch operation is released. The propulsion control button QB is continuously displayed.
According to this configuration, when the user touch-operates the propulsion control button QB again, the propulsion control button QB is in a position where the user can easily touch-operate, so that the convenience of the user can be improved.

Regarding the above configuration, the terminal control unit 20 displays the propulsion control button QB at a predetermined position on the vehicle control screen GM2 when the touch operation on the propulsion control button QB displayed on the vehicle control screen GM2 is released. You may.
According to this configuration, when the user touch-operates the propulsion control button QB again, the propulsion control button QB is in a position where the user can easily touch-operate, so that the convenience of the user can be improved.

Further, in the present embodiment, the terminal control unit 20 is located on the touch panel 9 at one end of the slide bar BR1 and the slide bar BR1 and moves along the slide bar BR1 in response to the swipe operation, while releasing the swipe operation. The lock screen GM1 on which the slide button SB1 that moves to one end of the slide bar BR1 is displayed is displayed accordingly. The terminal control unit 20 switches the screen to be displayed on the touch panel 9 from the lock screen GM1 to the vehicle control screen GM2 when the slide button SB1 is located on the other end of the slide bar BR1 for a predetermined period by swiping. The slide button SB1 located at the other end of the slide bar BR1 is switched to the propulsion control button QB to function as the propulsion control button QB.
According to this configuration, the user can perform a simple operation and perform a simple operation of swiping the slide button SB1 to switch the screen from the lock screen GM1 to the vehicle control screen GM2 and the vehicle. After switching to the control screen GM2, the propulsion control button QB can be easily touch-operated.

Further, in the present embodiment, the terminal control unit 20 changes the image showing the propulsion control button QB depending on whether the propulsion control button QB is operated or not.
According to this configuration, the user can easily and accurately recognize that the propulsion control button QB is touch-operated and that the propulsion control button QB is not touch-operated.

Further, in the present embodiment, the terminal control unit 20 acquires shooting data based on the shooting results of the camera provided in the vehicle S while displaying the vehicle control screen GM2, and obtains an image based on the acquired shooting data. The touch panel 9 is displayed, and the propulsion control button QB is displayed by superimposing the image based on the shooting data. The terminal control unit 20 switches the image to be displayed on the touch panel 9 according to the state of the housing 8 detected by the housing state detection unit 23, while the propulsion control button QB also switches the image to be displayed on the touch panel 9. Maintain the display of.
According to this configuration, the user can switch the image to be displayed on the touch panel 9 while controlling the propulsion of the vehicle S by operating the propulsion control button QB.

Further, in the present embodiment, when the housing state detection unit 23 detects that the housing 8 is in the first orientation, the terminal control unit 20 views the vehicle S as an image based on the shooting data. When the bird's-eye view image G1 which is the image obtained is displayed on the touch panel 9 and the housing state detection unit 23 detects that the orientation of the housing 8 is a second orientation different from the first orientation, the shooting data As an image based on the above, the touch panel 9 displays the vehicle exterior image G2, which is an image of the vehicle exterior taken by the vehicle exterior camera.
According to this configuration, the user performs a simple operation of changing the orientation of the housing 8 of the mobile terminal 3 to move the image displayed on the touch panel 9 between the bird's-eye view image G1 and the vehicle exterior image G2. You can switch with.

Further, as described above, the terminal control unit 20 of the mobile terminal 3 according to the present embodiment has a current position image X1 showing the current position of the vehicle S, and a position where the vehicle S moves from the current position and stops. Located on one or more candidate position images X2 indicating candidate positions, a slide bar BR2 extending from the base end PP1 toward each of the candidate position images X2, and a base end PP1 and sliding according to a swipe operation. The selection screen GM3 on which the slide button SB2 that moves along the bar BR2 is displayed is displayed, and the position where the vehicle S stops is determined from the candidate positions based on the movement of the slide button SB.
According to this configuration, the user can accurately determine the position where the vehicle S is stopped.

Further, in the present embodiment, when the slide button SB2 is positioned at the tip of the slide bar BR2 of 1 for a predetermined period of time, the terminal control unit 20 sets a candidate position corresponding to the slide bar BR2 of 1 to the vehicle S. Is determined as the stop position.
According to this configuration, it is possible to prevent the position where the vehicle S stops is determined by an erroneous operation by the user.

Further, in the present embodiment, the terminal control unit 20 displays the track image X3 showing the track from the current position of the vehicle S to the candidate position on the selection screen GM3.
According to this configuration, the user can accurately recognize the track from the current position of the vehicle S to the candidate position by referring to the selection screen GM3.

Further, in the present embodiment, when the movement of the vehicle S to the candidate position is prohibited, the terminal control unit 20 prohibits the movement to the candidate position on the slide bar BR2 corresponding to the candidate position where the movement is prohibited. An image indicating that the movement is to be performed is displayed, and the movement of the slide button SB2 along the slide bar BR2 corresponding to the candidate position where the movement is prohibited is prohibited.
According to this configuration, it is possible to prevent the user from selecting a candidate position where movement is prohibited.

Regarding the above configuration, when the movement of the vehicle S to the candidate position is prohibited, the terminal control unit 20 may prohibit the display of the slide bar BR2 corresponding to the candidate position where the movement is prohibited.
According to this configuration, it is possible to prevent the user from selecting a candidate position where movement is prohibited.

Further, in the present embodiment, when the terminal control unit 20 determines the position where the vehicle S stops, the screen to be displayed on the touch panel 9 is switched from the selection screen GM3 to the vehicle control screen GM2, and the slide button SB is pressed. , Switch to touch-operable operation buttons. The terminal control unit 20 transmits a propulsion instruction signal instructing the propulsion of the vehicle S to the information processing device 4 provided in the vehicle S while the propulsion control button QB displayed on the vehicle control screen GM2 is touch-operated. Then, the transmission of the propulsion instruction signal is stopped in response to the release of the touch operation on the propulsion control button QB.
According to this configuration, the user can accurately control the propulsion of the vehicle S.

Further, in the present embodiment, the terminal control unit 20 changes the image showing the propulsion control button QB depending on whether the propulsion control button QB is operated or not.
According to this configuration, the user can easily and accurately recognize that the propulsion control button QB is touch-operated and that the propulsion control button QB is not touch-operated.

Further, in the present embodiment, the current position of the vehicle S is the parking position where the vehicle S is currently parked, and the candidate position is the stop position after the vehicle S leaves the parking position and stops. is there.
According to this configuration, the user can perform an appropriate operation at the time of warehousing and select the stop position after warehousing.

Further, as described in the modified example, the current position of the vehicle S is the pre-parking position where the vehicle S before parking at the parking position is stopped, and the candidate position is the parking position where the vehicle S is parked. There may be.
According to this configuration, the user can select a parking position in which the vehicle S is parked by performing an appropriate operation when parking.

Further, as described in the modified example, the terminal control unit 20 of the mobile terminal 3 is a candidate for the current position image X1 showing the current position of the vehicle S and the position where the vehicle S moves from the current position and stops. The selection screen GM3 on which one or more candidate position images X2 indicating a certain candidate position are displayed is displayed, and the position where the vehicle S stops is determined from the candidate positions based on the operation on the candidate position image X2. May be good.
According to this configuration, the user can accurately determine the position where the vehicle S is stopped.

Further, as described above, in the case of the first mode, the terminal control unit 20 of the mobile terminal 3 according to the present embodiment is provided on the bird's-eye view image G1 and the vehicle S from which the vehicle S is viewed from the information processing device 4. When the composite image data of the composite image including the vehicle exterior image G2 taken by the camera is received and the housing state detection unit 23 detects that the orientation of the housing 8 is the first orientation, the composite image When the bird's-eye view image G1 is displayed on the touch panel 9 based on the data and the housing state detection unit 23 detects that the orientation of the housing 8 is a second orientation different from the first orientation, the composite image The vehicle exterior image G2 is displayed on the touch panel 9 based on the data.
According to this configuration, the mobile terminal 3 can display the image in an appropriate manner when displaying the image based on the shooting result of the camera provided in the vehicle S, and the convenience of the user can be improved.

Further, in the present embodiment, in the first mode, when the housing state detection unit 23 detects that the housing 8 is in the first orientation, the terminal control unit 20 uses the composite image data. , The bird's-eye view image data corresponding to the bird's-eye view image G1 is extracted, and the bird's-eye view image G1 is displayed on the touch panel 9 based on the extracted bird's-eye view image data. When the housing state detection unit 23 detects that the housing 8 is in the second orientation, the terminal control unit 20 extracts the vehicle exterior image data corresponding to the vehicle exterior image G2 from the composite image data. , The vehicle exterior image G2 is displayed on the touch panel 9 based on the extracted vehicle exterior image data.
According to this configuration, the terminal control unit 20 can appropriately display the bird's-eye view image G1 and the vehicle exterior image G2 based on the composite image data received from the information processing device 4.

Further, in the present embodiment, the housing 8 is a plate-shaped member having a rectangular front surface 8a, the touch panel 9 is provided on the front surface 8a of the housing 8, and the bird's-eye view image G1 is viewed from the front surface 8a. In addition, the vehicle exterior image G2 is a rectangular image in which the long side extends in the vertical direction, and the vehicle exterior image G2 is a rectangular image in which the long side extends in the horizontal direction when viewed from the front 8a. The orientation is such that the direction of the long side of the front surface 8a of the housing 8 is along the vertical direction, and the second direction is the direction in which the long side of the front surface 8a of the housing 8 is the vertical direction. This is the direction in which the state is along the direction orthogonal to.
According to this configuration, the terminal control unit 20 can display an appropriate image suitable for the orientation of the housing 8 according to the orientation of the housing 8 based on the shape of the housing 8.

Further, in the present embodiment, the information processing device control unit 30 of the information processing device 4 acquires information indicating the state of the housing 8 of the mobile terminal 3 in the second mode, and the mobile terminal 3 is based on the acquired information. When it is detected that the orientation of the housing 8 is the first orientation, the bird's-eye view image data corresponding to the bird's-eye view image G1 is extracted from the composite image data, and the extracted bird's-eye view image data is transmitted to the mobile terminal 3. When it is detected that the orientation of the housing 8 of the mobile terminal 3 is a second orientation different from the first orientation based on the acquired information, the exterior image data corresponding to the vehicle exterior image G2 is obtained from the composite image data. Is extracted, and the extracted vehicle exterior image data is transmitted to the mobile terminal 3.
According to this configuration, the information processing device 4 can display an image on the mobile terminal 3 in an appropriate manner, and can improve the convenience of the user.

It should be noted that the above-described embodiment shows only one aspect of the present invention, and can be arbitrarily modified and applied within the scope of the present invention.
For example, the process described as the process of the mobile terminal 3 may be executed by the mobile terminal 3 in cooperation with an external device (including an information processing device 4) capable of communicating with the mobile terminal 3. Similarly, the processing described as the processing of the information processing device 4 may be executed by the information processing device 4 in cooperation with an external device (including the mobile terminal 3) capable of communicating with the information processing device 4. ..
For example, the functional blocks described with reference to the drawings are schematic views showing the functional configurations of the respective devices classified according to the main processing contents in order to facilitate understanding of the present invention. The configuration of each device can be further classified into more components according to the processing content. It can also be categorized so that one component performs more processing. Further, the processing of each component may be executed by one hardware or may be executed by a plurality of hardware. Further, the processing of each component may be realized by one program or may be realized by a plurality of programs.
Further, the processing unit of the flowchart shown in the figure is divided according to the main processing contents in order to make the processing of each device easy to understand. The present invention is not limited by the method of dividing the processing unit and the name. The processing of each device can be divided into more processing units according to the processing content. Further, one processing unit can be divided so as to include more processing. Further, if the same processing can be performed, the processing order of the above flowchart is not limited to the illustrated example.

1 Information information system 2 Vehicle system 3 Mobile terminal (terminal)
4 Information processing device 5 Vehicle drive control device 9 Touch panel (display panel)
23 Housing condition detection unit 30 Information processing device control unit CA1 Forward shooting camera (camera)
CA2 Front right shooting camera (camera)
CA3 side right side shooting camera (camera)
CA4 Rear right shooting camera (camera)
CA5 Rear shooting camera (camera)
CA6 Rear left side shooting camera (camera)
CA7 Side left side shooting camera (camera)
CA8 Front left side shooting camera (camera)
G1 bird's-eye view image G2 vehicle exterior image G composite image GM1 lock screen GM2 vehicle control screen GM3 selection screen

Claims (12)

With the housing
The display panel provided on the housing and
A housing state detection unit that detects the state of the housing,
Shooting data based on the shooting results of a camera provided in the vehicle is acquired, and an image based on the acquired shooting data and a touch-operable operation button are displayed on the display panel, and the housing state detection unit displays the shooting data. A terminal control unit that switches the image to be displayed on the display panel according to the detected state of the housing.
A touch sensor that detects the operation position with respect to the display panel,
Equipped with a,
The terminal control unit
When the image to be displayed on the display panel is switched according to the state of the housing detected by the housing state detection unit, and when the display position of the operation button is switched to the image to be displayed on the display panel. A terminal characterized in that it is set to the touch position detected by the touch sensor . The terminal control unit
The case where the housing state detection unit detects that the orientation of the housing is the first orientation and the orientation of the housing is a second orientation different from the first orientation. The terminal according to claim 1, wherein the image to be displayed on the display panel is switched depending on whether or not is detected. The housing is a plate-shaped member having a rectangular front surface.
The display panel is provided on the front surface of the housing.
The first direction is a direction in which the direction of the long side of the front surface of the housing is along the vertical direction.
The terminal according to claim 2, wherein the second direction is a direction in which the direction of the long side of the front surface of the housing is along a direction orthogonal to the vertical direction. The terminal control unit
When it is detected by the housing state detection unit that the orientation of the housing is the first orientation, a bird's-eye view image, which is an image of a bird's-eye view of the vehicle, is displayed on the display panel as an image based on the shooting data. Display,
When it is detected by the housing state detection unit that the orientation of the housing is the second orientation, an image outside the vehicle, which is an image of the outside of the vehicle taken by the camera, is used as an image based on the shooting data. The terminal according to claim 2 or 3, wherein the terminal is displayed on the display panel. The vehicle is provided with a plurality of the cameras for photographing the outside of the vehicle.
The terminal control unit
When the housing state detection unit detects that the housing is in the second orientation, one of the plurality of cameras is in accordance with a predetermined priority corresponding to the state of the vehicle. The terminal according to claim 4, wherein the camera is selected, and the vehicle exterior image based on the shooting result of the selected camera is displayed on the display panel. The terminal according to claim 5, wherein the predetermined priority is a higher priority as the camera capable of photographing the traveling direction of the vehicle. The terminal according to claim 5, wherein the predetermined priority is a higher priority as the camera capable of photographing an object close to the vehicle. The terminal control unit
When the housing state detection unit detects that the housing has made a predetermined movement after displaying the outside image of 1 based on the shooting result of the camera of 1 on the display panel, the case of 1 Any one of claims 5 to 7, wherein instead of the vehicle exterior image, another vehicle exterior image based on a shooting result of another camera different from the camera is displayed on the display panel. The terminal described in the section. Each of the plurality of cameras photographs the outside of the vehicle in different directions.
The predetermined movement is a first movement in which the state of the housing changes in the first aspect, or a second movement in which the state of the housing changes in a second aspect different from the first aspect. Is one of the movements
The terminal control unit
When the housing state detection unit detects that the housing has made the first movement, the outside image is switched in the first order according to the direction in which the camera shoots.
When the housing state detection unit detects that the housing has made the second movement, the second order is according to the shooting direction of the camera and is different from the first order. The terminal according to claim 8, wherein the images outside the vehicle are switched in order. Before Symbol terminal control unit,
Together to display an image based on the imaging data to the display panel, so as to overlap with the image based on the captured data, to display the touch operation possible the operation button, even when switching images to be displayed on the display panel, Maintain the display of the operation buttons
Any one of claims 1 to 9, wherein a propulsion instruction signal instructing the propulsion of the vehicle is transmitted to the information processing device provided in the vehicle while the operation button is touch-operated. The terminal described in. The terminal control unit
A slide button that is moved by a touch operation is displayed on the display panel as the operation button, and a slide bar that clearly indicates the range in which the slide button can be moved is displayed.
An image based on the shooting data according to the state of the housing detected by the housing state detection unit during the period in which the slide button is maintained in a state of being moved from one end to the other end of the slide bar. When the image is switched and displayed on the display panel, another operation button that causes the touch position of the slide button detected by the touch position detection unit when the image is switched to function as an operation button different from the slide button. Is displayed,
The position of the other operation button is moved following the movement of the touch position detected by the touch position detection unit.
The terminal according to any one of claims 1 to 10, characterized in that. A method for controlling a terminal having a housing and a display panel provided on the housing.
Acquire shooting data based on the shooting results of the camera installed in the vehicle,
The image based on the acquired shooting data and the operation buttons that can be touch-operated are displayed on the display panel, the state of the housing is detected, and the display panel is used according to the detected state of the housing. A process of switching the image to be displayed and a process of detecting the operation position with respect to the display panel are performed.
When switching the image to be displayed on the display panel according to the detected state of the housing, the display position of the operation button is set to the touch position detected when switching the image to be displayed on the display panel.
A terminal control method characterized by this.