JP2012191479A - Information processing system, server device, and in-vehicle device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a user with an image in a display form corresponding to a spot where a vehicle travels.SOLUTION: Data indicating a display form of a spot at which a user switches a display form of a photographed image is acquired from a plurality of vehicles in a server device, and the data of the display form most highly frequently used by the user at one spot is stored as a representative form corresponding to the spot. Then, when a vehicle travels through the spot for which information of the representative form is recorded, the information of the representative form is transmitted from the server device to the vehicle, and an image based on the display form is displayed on a display of the vehicle. Thus, the optimum display form of the photographed image at the spot corresponding to the position of the vehicle, where the plurality of vehicles have actually traveled, can be acquired. Also, the optimum display form of the photographed image can be provided for a user traveling through the spot for the first time.

Description

  The present invention relates to image display processing.

  Conventionally, when a captured image around a vehicle is displayed on a display of a display device, an image in a display mode selected by the user has been displayed. However, when the user operation for selecting the display mode is performed when driving the vehicle, the user's attention to driving may be distracted. On the other hand, Patent Document 1 discloses an image of a display mode having the highest use frequency in the vicinity of the point when the vehicle approaches the point due to the use history of the display mode in the vicinity of the point where the vehicle has traveled in the past. Is described on the display of a display device.

JP 2004-304416 A

  However, in the technique described in Patent Literature 1, when the display mode usage history is the history of the display mode selected by the user according to the driving environment in the vicinity of the point when the vehicle travels at one point, It was unclear whether the display mode was used before the vehicle approached the point regardless of the vehicle traveling. Therefore, when the history of the display mode is not the one selected by the user according to the travel environment of the point, even if the image corresponding to the display mode with the highest use frequency is displayed on the display, the optimum for the point where the vehicle travels The display mode may not be provided to the user.

  This invention is made | formed in view of the said subject, and it aims at providing a user with the optimal display mode corresponding to the point where a vehicle drive | works.

  In order to solve the above-mentioned problems, the invention of claim 1 is an information processing system, and is a first in-vehicle device that is used in a vehicle and can display captured images around the vehicle in different display modes, and a vehicle. A second in-vehicle device that can be used to display captured images around the vehicle in different display modes; and a server device that can communicate with the first in-vehicle device and the second in-vehicle device. The first in-vehicle device includes (a-1) an acquisition unit that acquires a current vehicle position, (a-2) a vehicle position when the user switches the display mode of the captured image, and the switched state. And (b) the server device obtains the vehicle information from the first in-vehicle device of a plurality of vehicles. And (b-2) based on the vehicle information , A specifying means for specifying a representative aspect that is a display aspect having the highest use frequency at the position of the vehicle indicated by the vehicle information, and (b-3) the position of the vehicle and the representative aspect of the position are stored in association with each other. And (b-4) transmitting means for transmitting the representative aspect to the second in-vehicle device, (c) the second in-vehicle device is (c-1) the current vehicle position. And (c-2) display means for displaying the photographed image in the representative mode corresponding to the current position of the vehicle.

  Further, the invention of claim 2 is an information processing system, which is used in a vehicle and capable of displaying captured images around the vehicle in different display modes, and a server device capable of communicating with the in-vehicle device. (A) The in-vehicle device includes an acquisition unit that acquires a current vehicle position, a vehicle position when the user switches the display mode of the captured image, and the display mode after switching. Transmission means for transmitting vehicle information including to the server device, (b) the server device based on the vehicle information, acquisition means for acquiring the vehicle information from the in-vehicle devices of a plurality of vehicles, Specifying means for specifying a representative aspect which is a display aspect having the highest use frequency at the position of the vehicle indicated by the vehicle information.

  Further, the invention of claim 3 is an information processing system, which is used in a vehicle and capable of displaying captured images around the vehicle in different display modes; and a server device capable of communicating with the in-vehicle device; (A) The server device stores the vehicle position in association with a representative mode that is the display mode having the highest use frequency among the different display modes at the position; (B) the in-vehicle device acquires the current vehicle position, and the imaging in the representative mode corresponding to the current vehicle position. Display means for displaying an image.

  According to a fourth aspect of the present invention, there is provided a server device capable of communicating with an in-vehicle device that is used in a vehicle and can display captured images around the vehicle in different display modes. An acquisition means for acquiring vehicle information including the position of the vehicle when switched and the display mode after switching from the in-vehicle device of a plurality of vehicles, and the vehicle information indicated by the vehicle information based on the vehicle information Specifying means for specifying a representative aspect which is a display aspect having the highest use frequency at the position.

  Further, the invention of claim 5 is a server device that can be used in a vehicle and can communicate with an in-vehicle device capable of displaying captured images of the periphery of the vehicle in different display modes, wherein the position of the vehicle Storage means for associating and storing a representative aspect that is the most frequently used display aspect among different display aspects, and transmission means for transmitting the representative aspect to the in-vehicle device.

  The invention according to claim 6 further includes an acquisition unit that acquires a position of a vehicle in which the in-vehicle device is used in the server device according to claim 5, and the in-vehicle device is used as the transmission unit. The representative aspect corresponding to the position of the vehicle is transmitted.

  According to a seventh aspect of the present invention, in the server device according to the fifth aspect, the transmission unit transmits all the representative aspects stored in the storage unit at a time.

  The invention of claim 8 is an in-vehicle device that is used in a vehicle and is capable of displaying captured images around the vehicle in different display modes, the acquisition means for acquiring the current position of the vehicle, and the captured image Transmission means for transmitting vehicle information including the position of the vehicle when the user switches the display mode and the display mode after the switching to a predetermined server device.

  The invention of claim 9 is an in-vehicle device that is used in a vehicle and can display captured images of the periphery of the vehicle in different display modes, the first acquisition means for acquiring the current vehicle position, and the vehicle And a second acquisition means for acquiring the representative aspect from a server device that associates and stores the position and the representative aspect that is the display aspect having the highest use frequency among the different display aspects at the position, and the current vehicle Display means for displaying the photographed image in the representative mode corresponding to the position.

  According to a tenth aspect of the present invention, in the in-vehicle device according to the ninth aspect, the second acquisition unit acquires the representative aspect corresponding to the position of the vehicle acquired by the first acquisition unit.

  The invention according to claim 11 is the vehicle-mounted device according to claim 9, further comprising storage means for storing the representative aspect, wherein the second acquisition means stores all the representatives stored in the storage means. A mode is acquired at once and stored in the storage means.

  Further, the invention of claim 12 is an information processing system, which is a display means used in a vehicle and capable of displaying captured images around the vehicle in different display modes, and an acquisition means for acquiring the current position of the vehicle. And based on vehicle information including the position of the vehicle when the user switches the display mode of the captured image and the display mode after switching, the frequency of use is highest at the position of the vehicle indicated by the vehicle information A specifying unit that specifies a representative mode that is a display mode; a storage unit that stores the vehicle position and the representative mode of the position in association with each other; and the display unit is configured to store the current vehicle position. The photographed image is displayed in the representative mode corresponding to.

  According to the first aspect of the present invention, by displaying the captured image in a representative manner corresponding to the current vehicle position, the optimal display of the captured image at a point corresponding to the position of the vehicle on which the plurality of vehicles actually traveled is performed. Aspect can be acquired. Moreover, the optimal display mode of the captured image can be provided to the user who travels for the first time at the point.

  According to the invention of claim 2, a plurality of vehicles are identified by specifying a representative aspect that is the display aspect having the highest use frequency at the position of the vehicle indicated by the vehicle information, based on the vehicle information acquired by the acquisition means. It is possible to acquire the optimum display mode of the captured image at a point corresponding to the position of the vehicle that actually traveled.

  According to the invention of claim 3, by displaying the photographed image in a representative manner corresponding to the current vehicle position, the optimum display manner of the photographed image for a user who travels for the first time at a point corresponding to the vehicle position. Can provide.

  According to the invention of claim 4, a plurality of vehicles are identified by specifying a representative aspect that is the display aspect having the highest use frequency at the position of the vehicle indicated by the vehicle information, based on the vehicle information acquired by the acquisition means. It is possible to acquire the optimum display mode of the captured image at a point corresponding to the position of the vehicle that actually traveled.

  According to the invention of claim 5, by transmitting the representative mode to the in-vehicle device, it is possible to provide the optimum display mode of the captured image to the user who travels for the first time at the point corresponding to the position of the vehicle.

  According to the sixth aspect of the present invention, the transmission means transmits a representative aspect corresponding to the position of the vehicle used in the in-vehicle device, so that the user who travels for the first time at the point corresponding to the position of the vehicle can receive the captured image. An optimal display mode can be provided.

  Further, according to the invention of claim 7, the transmission means transmits all the representative aspects stored in the storage means at a time so that the optimum display aspect of the photographed image for each of the plurality of points can be displayed to the user of the vehicle. Can be provided.

  According to the invention of claim 8, by transmitting vehicle information including the position of the vehicle when the user switches the display mode of the captured image and the display mode after switching to a predetermined server device. The server apparatus can be provided with a display mode of the captured image at a point corresponding to the position of the vehicle on which the plurality of vehicles actually traveled.

  According to the ninth aspect of the present invention, the captured image is displayed in a representative manner corresponding to the current position of the vehicle, so that the optimum display mode of the photographed image is displayed for a user who travels for the first time at a point corresponding to the position of the vehicle. Can provide.

  According to the invention of claim 10, the second acquisition means travels for the first time at a point corresponding to the position of the vehicle by acquiring a representative aspect corresponding to the position of the vehicle acquired by the first acquisition means. It is possible to provide the user with an optimal display mode of the captured image. Moreover, since it is not necessary to store the representative aspects of the captured images at a plurality of points in advance, the capacity stored in the memory of the in-vehicle device can be reduced.

  According to the invention of claim 11, the second acquisition unit acquires all the representative modes stored in the storage unit at a time and stores them in the storage unit, whereby the representative modes of the captured images at a plurality of points are stored. Can be acquired in advance.

  Further, according to the invention of claim 12, the display means displays the captured image in a display mode corresponding to the current position of the vehicle, so that the user who travels for the first time at the point corresponding to the position of the vehicle can display the captured image. An optimal display mode can be provided.

FIG. 1 is a diagram illustrating an overview of an information processing system that performs communication in an image display mode. FIG. 2 is a block diagram of the in-vehicle device. FIG. 3 is a diagram illustrating a position where the in-vehicle camera is arranged in the vehicle. FIG. 4 is a diagram for explaining a method of generating a composite image. FIG. 5 is a block diagram of the server device. FIG. 6 is a block diagram of the in-vehicle device. FIG. 7 is a diagram illustrating the transition of the operation mode of the in-vehicle device. FIG. 8 is a diagram showing the transition of the display mode in the front mode. FIG. 9 is a diagram illustrating a region corresponding to an image displayed on the display device. FIG. 10 is a diagram illustrating vehicle information stored in the storage unit of the display device. FIG. 11 is a diagram illustrating the aggregated data stored in the data storage unit of the server device. FIG. 12 is a diagram illustrating point data stored in the data storage unit of the server device. FIG. 13 is a flowchart illustrating a process of transmitting vehicle information from the in-vehicle device to the server device. FIG. 14 is a process flowchart for transmitting individual point data from the server device to the in-vehicle device. FIG. 15 is a process flowchart for transmitting individual point data from the server device to the in-vehicle device. FIG. 16 is a block diagram of the in-vehicle device. FIG. 17 is a processing flowchart for transmitting point data from the server device to the in-vehicle device. FIG. 18 is a process flowchart for transmitting point data from the server device to the in-vehicle device.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Embodiment 1>
<1. Configuration>
<1-1. Overall system overview>
FIG. 1 is a diagram illustrating an overview of an information processing system 130 that performs communication of information related to an image display mode. The information processing system 130 includes an in-vehicle device 120a, 120b, 120c, 120d mounted on a plurality of vehicles such as vehicles A1a, A1b, A1c, A1d, and a server device 9 arranged at a predetermined center. Is done.

  Vehicle-mounted devices 120a, 120b, and 120c (hereinafter also referred to as “transmission-side vehicle-mounted device 120”) mounted on vehicles A1a, A1b, and A1c (hereinafter also referred to as “vehicle A1”) such as automobiles are display devices. Communication with the server device 9 is performed via a communication unit (for example, the communication unit 23 of the display device 20 shown in FIG. 2).

  The server device 9 acquires information on the display mode of the captured image displayed on the display (for example, “display 24” illustrated in FIG. 2) of the transmission-side in-vehicle device 120 through communication with the transmission-side in-vehicle device 120, and the display mode Is processed and stored. And the server apparatus 9 transmits the information regarding the display mode of the picked-up image in the point corresponding to the position where vehicle A1d travels to the receiving side vehicle-mounted apparatus 120d.

  The in-vehicle device 120d displays an image captured by using at least one of a plurality of cameras (cameras 51, 52, 53, and 54 shown in FIG. 4) on the display (for example, according to information on the display mode acquired from the server device 9). This is displayed on the display 24) shown in FIG. Hereinafter, this in-vehicle device 120d is also referred to as “reception-side in-vehicle device 120d”. Thereby, the optimal display mode of the captured image at the point corresponding to the position of the vehicle on which the plurality of vehicles actually traveled can be acquired. Moreover, the optimal display mode of the captured image can be provided to the user who travels for the first time at the point.

<1-2. In-vehicle device on the transmission side>
FIG. 2 is a block diagram of the in-vehicle devices 120a, 120b, and 120c. The transmission-side in-vehicle device 120 has a function of shooting the surroundings of the vehicles A1a, A1b, and A1c to generate an image, and outputting the generated image to the display device 20 in the vehicle interior. A user (typically a driver) of the vehicle A1 can grasp the state of the periphery of the vehicle almost in real time by using the transmission-side in-vehicle device 120.

  As shown in FIG. 2, the in-vehicle device 120 on the transmission side captures the display device 20, the image processing device 100 that generates a peripheral image indicating the periphery of the vehicle and outputs image information to the display device 20, and the periphery of the vehicle. And a photographing unit 5 having a camera to be used.

  The display device 20 includes a navigation device having a display for displaying navigation guidance, an audio device having a display for displaying music title information, television images, and the like, a smartphone that can be carried by the user and brought into the vehicle A1, and the like. It is.

  The display device 20 includes a control unit 21 that controls the entire device, a storage unit 22 that stores processing data of the control unit 21, a communication unit 23 that communicates with the server device 9, and a touch panel function. And a display 24 such as a liquid crystal display and an operation unit 25 operated by a user. And the display apparatus 20 is installed in the instrument panel of vehicle A1, etc. so that the screen of the display 24 can be visually recognized from a user. Various instructions from the user are received by the operation unit 25 and the display 24 as a touch panel.

  The control unit 21 is a computer including a CPU, a RAM, a ROM, and the like, and various functions including a navigation function are realized by the CPU performing arithmetic processing according to a predetermined program.

  The control unit 21 mainly has functions of a display control unit 201 and a position deriving unit 202. The display control unit 201 controls a display mode of a captured image of the in-vehicle camera displayed on the display 24 in accordance with a user operation from the operation unit 25 or the like. That is, control is performed to display captured images around the vehicle A1 in different display modes.

  The position deriving unit 202 derives latitude / longitude information corresponding to the position of the vehicle A1. The position information of the vehicle A1 is derived when, for example, the user switches the display form of the captured image of the in-vehicle camera displayed on the display 24. Such a position of the vehicle A1 is realized by receiving a radio wave signal from a GPS (Glpbal Positioning System) satellite and processing the received signal by the control unit 21. GPS is a system that measures the position of the vehicle A1 based on radio waves from at least three or more GPS satellites that orbit the earth.

  The storage unit 22 stores a program for the control unit 1 to control the display device 20. The storage unit 22 is information including positional information of the vehicle A1 when the user switches the display mode of the captured image of the in-vehicle camera displayed on the display 24 and the display mode of the captured image after the switching by the user. Certain vehicle information (for example, vehicle information TA1, TA2, TA3 shown in FIG. 10) is stored. That is, the switching operation of the user's captured image at the travel point for each vehicle is a trigger, and the display mode of the captured image after switching and the point information where the switching has been performed are stored in the storage unit 22.

  The communication unit 23 is an interface that performs communication between the display device 20 and the server device 9, and the vehicle information TA 1, TA 2, TA 3 (hereinafter also referred to as “vehicle information TA”) stored in the storage unit 22. Send to. Thereby, the optimal display mode of the captured image at the point corresponding to the position of the vehicle on which the plurality of vehicles actually traveled can be acquired. The transmission of the vehicle information TA to the server device 9 is performed when a predetermined number of data is accumulated in the storage unit 22.

  The vehicle information TA may be transmitted to the server device 9 via the communication unit 23 when the vehicle information TA is acquired without storing the vehicle information TA in the storage unit 22. That is, when the display mode of the captured image displayed on the display 24 is switched by a user operation, the vehicle information TA associated with the switching may be transmitted to the server device 9.

  The display 24 is a display unit that displays various images, and normally displays an image based on the function of the display device 20 alone under the control of the display control unit 201. And the surrounding image which shows the mode of the periphery of vehicle A1 produced | generated with the image processing apparatus 100 on the predetermined conditions is displayed. The display 24 displays a bird's-eye view image of the vehicle A1 and the surrounding area of the vehicle A1. Furthermore, the bird's-eye view image displayed on the display 24 shows the peripheral area almost in real time. Thereby, the display device 20 also functions as a device that receives and displays the peripheral image generated by the image processing device 100.

  The operation unit 25 is operated by a user to realize various functions of the display device 20, and is, for example, a hard switch provided in the display device 20. The display mode of the captured image of the in-vehicle camera displayed on the display 24 can be switched by the user operating the operation unit 25.

  The image processing apparatus 100 is an ECU (Electronic Control Unit) whose main body 10 has a function of generating a peripheral image, and is arranged at a predetermined position of the vehicle A1. The image processing apparatus 100 functions as an image generation apparatus that generates a composite image viewed from a virtual viewpoint based on a captured image obtained by photographing the periphery of the vehicle A1 with the photographing unit 5. The image processing apparatus 100 also functions as an apparatus that processes an image captured by one camera of the imaging unit 5. The plurality of in-vehicle cameras 51, 52, 53, and 54 provided in the photographing unit 5 are arranged at appropriate positions of the vehicle A1 different from the main body unit 10, but will be described in detail later.

  The main body 10 of the image processing apparatus 100 mainly includes a control unit 1 that controls the entire apparatus, and an image processing unit 3 that processes a captured image acquired by the imaging unit 5 to generate a peripheral image for display. Yes.

  Various instructions from the user received by either the operation unit 25 of the display device 20 or the display 24 are received by the communication unit 41 as control signals and input to the control unit 1. As a result, the image processing apparatus 100 can operate in response to a user operation on the display device 20.

  The image processing apparatus 100 also includes a switching button 301. When the switch button 301 is operated by the user, a signal indicating the user's instruction is input to the control unit 1, and an image of a display mode in a front mode (for example, front mode M2 shown in FIG. 8) described later is displayed on the display 24. Is displayed. Note that the switching button 301 may be provided on the display device 20 as a hard button. Further, the switching button 301 may be displayed on the display 24 of the display device 20 as a touch panel button operated by a user touching the display 24 with a finger or the like.

  The image processing unit 3 is configured as a hardware circuit capable of various types of image processing, and includes an image acquisition unit 31, a composite image generation unit 32, and an image transmission unit 33 as main functions.

  The image acquisition unit 31 acquires a plurality of captured images captured by the in-vehicle cameras 51, 52, 53, and 54 of the imaging unit 5 that captures the periphery of the vehicle A1.

  The composite image generation unit 32 generates a composite image viewed from an arbitrary virtual viewpoint around the vehicle, based on a plurality of captured images acquired by the in-vehicle cameras 51, 52, 53, and 54. A method by which the composite image generation unit 32 generates a composite image viewed from a virtual viewpoint will be described later.

  Further, the image transmission unit 33 displays at least one of the composite image generated by the composite image generation unit 32 and other images (for example, images taken by one camera) via the communication unit 41. Send to. Thereby, a peripheral image showing the periphery of the vehicle is displayed on the display 24 of the display device 20.

  The control unit 1 is a computer including a CPU, a RAM, a ROM, and the like, and various control functions are realized by the CPU performing arithmetic processing according to a predetermined program. An image control unit 101 shown in the drawing is a part of the functions of the control unit 1 realized in this way.

  The image control unit 101 controls image processing executed by the image processing unit 3. For example, the image control unit 101 instructs various parameters necessary for generating a composite image generated by the composite image generation unit 32. Note that such an instruction for parameters is given by the image control unit 101 when the control unit 1 receives, for example, a user operation instruction signal associated with switching of the display mode of a captured image.

  The main body 10 of the image processing apparatus 100 further includes a nonvolatile memory 40, a communication unit 41, and a signal input unit 42, which are connected to the control unit 1.

  The nonvolatile memory 40 is configured by a flash memory or the like that can maintain stored contents even when the power is turned off. The nonvolatile memory 40 stores vehicle type information 4a. The vehicle type information 4a includes data corresponding to the type of vehicle that is required when the composite image generation unit 32 generates a composite image.

  The communication unit 41 is an interface for the image processing apparatus 100 to communicate with the display device 20.

  The signal input unit 42 is an interface when signals are input from various devices provided in the vehicle A1. A signal from the outside of the transmission-side in-vehicle device 120 is input to the control unit 1 via the signal input unit 42. Specifically, signals indicating various information are input to the control unit 1 from the shift sensor 81, the vehicle speed sensor 82, the direction indicator 83, the steering angle sensor 84, and the like.

  From the shift sensor 81, the operation position of the shift lever of the transmission of the vehicle A1, that is, "P (parking)", "D (forward)", "N (neutral)", "R (reverse)", etc. The shift position is input. From the vehicle speed sensor 82, the traveling speed (km / h) of the vehicle A1 at that time is input.

  From the direction indicator 83, a turn signal indicating a direction instruction based on the operation of the blinker switch, that is, a direction instruction intended by the driver of the vehicle A1 is input. When the turn signal switch is operated, a turn signal is generated, and the turn signal indicates the operated direction (left direction or right direction). When the turn signal switch is in the neutral position, the turn signal is turned off.

  From the steering angle sensor 84, the operation direction of the steering wheel operated by the user is input to the control unit 1. Specifically, the steering angle sensor 84 is inputted with the rotation direction, rotation angle, and rotation speed of the steering wheel operated by the user to the control unit 1 via the signal input unit 42.

<1-3. Shooting Department>
Next, the photographing unit 5 of the image processing apparatus 100 will be described in detail. The photographing unit 5 is electrically connected to the control unit 1 and operates based on a signal from the control unit 1.

  The imaging unit 5 includes a front camera 51, a back camera 52, a left side camera 53, and a right side camera 54 that are in-vehicle cameras. Each of these vehicle-mounted cameras 51, 52, 53, and 54 includes an image sensor such as a CCD or a CMOS and electronically acquires an image.

  FIG. 3 is a diagram illustrating positions where the in-vehicle cameras 51, 52, 53, and 54 are arranged in the vehicles A1 and A1d. In the following description, the vehicle A1 will be described as an example. Also, when indicating the direction and orientation, the three-dimensional XYZ orthogonal coordinates shown in the figure are used as appropriate. The XYZ axes are fixed relative to the vehicle A1. Here, the X-axis direction is along the left-right direction of the vehicle A1, the Y-axis direction is along the straight traveling direction (front-rear direction) of the vehicle A1, and the Z-axis direction is along the vertical direction. For convenience, the + X side is the right side of the vehicle A1, the + Y side is the rear side of the vehicle A1, and the + Z side is the upper side.

  The front camera 51 is provided in the vicinity of the license plate mounting position at the front end of the vehicle A1, and its optical axis 51a is directed in the straight traveling direction of the vehicle A1 (on the Y side in the Y axis direction in plan view). The back camera 52 is provided in the vicinity of the license plate mounting position at the rear end of the vehicle A1, and its optical axis 52a is directed in the reverse direction of the straight traveling direction of the vehicle A1 (on the + Y side in the Y-axis direction in plan view). Yes. The left side camera 53 is provided on the left door mirror 93, and the right side camera 54 is provided on the right door mirror 94. The optical axis 53a is directed to the outside along the left direction of the vehicle A1, and the optical axis 54a is directed along the right direction of the vehicle A1 (X-axis direction in plan view). Note that the mounting position of the front camera 51 and the back camera 52 is preferably approximately the center in the left and right, but may be slightly shifted in the left and right directions from the center in the left and right.

  As the lenses of these in-vehicle cameras 51, 52, 53, and 54, fish-eye lenses and the like are employed, and the in-vehicle cameras 51, 52, 53, and 54 have an angle of view α of 180 degrees or more. Therefore, by using the four in-vehicle cameras 51, 52, 53, and 54, it is possible to capture the entire periphery of the vehicle A1.

<1-3. Image conversion processing>
Next, the composite image generation unit 32 of the image processing unit 3 generates a composite image that shows a state in which the periphery of the vehicles A1 and A1d is viewed from an arbitrary virtual viewpoint based on the plurality of captured images obtained by the imaging unit 5. The method will be described. When generating the composite image, the vehicle type information 4a stored in advance in the nonvolatile memory 40 is used. FIG. 4 is a diagram for explaining a method of generating a composite image. In the following description of the image conversion process, the process of the vehicle A1 will be described as an example, but the same image conversion process is performed for the vehicle A1d.

  When photographing is simultaneously performed by the front camera 51, the back camera 52, the left side camera 53, and the right side camera 54 of the photographing unit 5, four images that respectively indicate the front, rear, left side, and right side of the vehicle A1. Captured images P1 to P4 are acquired. That is, the four captured images P1 to P4 acquired by the capturing unit 5 include information indicating the entire periphery of the vehicle A1 at the time of capturing.

  Next, each pixel of the four captured images P1 to P4 is projected onto a three-dimensional curved surface SP in a virtual three-dimensional space. The three-dimensional curved surface SP has, for example, a substantially hemispherical shape (a bowl shape), and a center portion (a bottom portion of the bowl) is defined as a position where the vehicle A1 exists. A correspondence relationship is determined in advance between the positions of the pixels included in the captured images P1 to P4 and the positions of the pixels of the three-dimensional curved surface SP. For this reason, the value of each pixel of the solid curved surface SP can be determined based on this correspondence and the value of each pixel included in the captured images P1 to P4.

  The correspondence between the positions of the pixels of the captured images P1 to P4 and the positions of the pixels of the three-dimensional curved surface SP is based on the arrangement of the four in-vehicle cameras 51, 52, 53, and 54 in the vehicle A1 (the distance between each other, the ground height, Depending on the optical axis angle, etc.). For this reason, table data indicating this correspondence relationship is included in the vehicle type information 4 a stored in the nonvolatile memory 40.

  Further, polygon data indicating the shape and size of the vehicle body included in the vehicle type information 4a is used, and a vehicle image that is a polygon model indicating the three-dimensional shape of the vehicle A1 is virtually provided. The configured vehicle image is arranged in a substantially hemispherical center portion defined as the position of the vehicle A1 in the three-dimensional space in which the three-dimensional curved surface SP is set.

  Further, the virtual viewpoint VP is set by the control unit 1 for the three-dimensional space where the solid curved surface SP exists. The virtual viewpoint VP is defined by the viewpoint position and the viewing direction, and is set at an arbitrary viewpoint position corresponding to the periphery of the vehicle A1 in this three-dimensional space toward an arbitrary viewing direction.

  Then, according to the set virtual viewpoint VP, a necessary area on the three-dimensional curved surface SP is cut out as an image. The relationship between the virtual viewpoint VP and a necessary area in the three-dimensional curved surface SP is determined in advance, and is stored in advance in the nonvolatile memory 40 or the like as table data. That is, an image area is cut out according to the set virtual viewpoint VP, rendering is performed on a vehicle image composed of polygons, and the resulting two-dimensional vehicle image is superimposed on the cut out image. Is done. Thereby, the composite image which shows a mode that the periphery of vehicle A1 and vehicle A1 was seen from arbitrary virtual viewpoints is produced | generated.

  For example, in the case where the virtual viewpoint VP1 having a viewpoint position just above the center of the position of the vehicle A1 and a visual field direction substantially directly below is set, the vehicle A1 ( Actually, a composite image CP1 showing the vehicle image) and the surroundings of the vehicle A1 is generated. Also, as shown in the figure, when the virtual viewpoint VP2 in which the viewpoint position is the left rear of the position of the vehicle A1 and the visual field direction is substantially in front of the vehicle A1, the entire periphery from the left rear of the vehicle A1 is set. As seen, a composite image CP2 showing the vehicle A1 (actually a vehicle image) and the surroundings of the vehicle A1 is generated.

  In the case of actually generating a composite image, it is not necessary to determine the values of all the pixels of the three-dimensional curved surface SP, and only the values of the pixels in the area necessary corresponding to the set virtual viewpoint VP are photographed. By determining based on the images P1 to P4, the processing speed can be improved.

<1-4. Server device>
FIG. 5 is a block diagram of the server device 9. The server device 9 processes data including the display mode of the captured image collected from the transmission-side in-vehicle device 120 of the vehicle A1, and transmits data related to the display mode to the reception-side in-vehicle device 120d of the vehicle A1d. The server device 9 mainly includes a control unit 91, a data recording unit 92, and a communication unit 95.

  The control unit 91 is a computer including a CPU, a RAM, a ROM, and the like, and various control functions are realized by the CPU performing arithmetic processing according to a predetermined program. The acquisition unit 911 and the specifying unit 912 shown in the drawing are a part of the functions of the control unit 1 realized in this way.

  The acquisition unit 911 acquires vehicle information TA transmitted from the transmission-side in-vehicle device 120. That is, the vehicle information TA including the position information of the vehicle A1 (latitude / longitude information of the vehicle A1) when the user switches the display mode of the captured image and the display mode of the captured image after the switching by the user is acquired.

  The specifying unit 912 specifies a representative aspect that is a display aspect having the highest use frequency at a point corresponding to the position of the vehicle A1 in the vehicle information TA. That is, when the user switches the display mode of the captured image displayed on the display 24, the display mode with the highest number of uses is specified as the representative mode among the display modes of the captured image after the switching by the user. Thereby, the optimal display mode of the captured image at a point corresponding to the position of the vehicle on which the plurality of vehicles actually traveled can be acquired.

  The data recording unit 92 records data including total data (for example, total data TD shown in FIG. 11) and point data (for example, point data TP shown in FIG. 12). The total data TD is data obtained by totaling the vehicle information TA transmitted from the plurality of vehicles A1 and acquired by the acquisition unit 911. In other words, the total data TD is accumulated in the vehicle information TA including the positional information of the vehicle A1 when the user switches the captured image displayed on the display 24 and the display mode of the captured image after the switching by the user. is there.

  The point data TP is data in which the specifying unit 912 specifies a representative aspect of the captured image at the point corresponding to the position of the vehicle A1 based on the data of the total data TD. That is, the point data TP is the display mode having the highest use frequency among the display modes of the captured image after the switching by the user at the position of the vehicle A1 when the user switches the display mode of the captured image displayed on the display 24. Is accumulated. In this way, it is possible to acquire the optimum display mode of the captured image at the point corresponding to the position of the vehicle on which the plurality of vehicles actually traveled.

  The communication unit 95 is an interface for the server device 9 to transmit / receive information to / from the transmission-side in-vehicle device 120 and the reception-side in-vehicle device 120d. Then, by transmitting the point data of the point corresponding to the position of the vehicle A1d from the server device 9 to the receiving-side in-vehicle device 120d via the communication unit 95, it is optimal for the user who travels for the first time at the point corresponding to the position of the vehicle. A display mode can be provided. The point data transmitted from the server device 9 to the receiving-side in-vehicle device 120d is data of a point corresponding to the position of the vehicle A1d in the data recorded in the point data TP, and will be described later as individual point data (for example, FIG. 12 is individual point data tp1).

<1-5. In-vehicle device on the receiving side>
FIG. 6 is a block diagram of the receiving-side in-vehicle device 120d. The receiving-side in-vehicle device 120d receives, from the server device 9, information on the representative aspect of the point corresponding to the position where the vehicle 1d on which the receiving-side in-vehicle device 120d is mounted travels. Note that the configuration of the reception-side in-vehicle device 120d is substantially the same as the configuration of the transmission-side in-vehicle device 120 that transmits the vehicle information TA to the server device 9, but a part thereof is different. Differences will be described below.

The transmission-side in-vehicle device 120 mounted on the vehicle A1 stores the vehicle information TA to be transmitted to the server device 9 in the storage unit 22 of the display device 20, whereas the reception-side in-vehicle device 120d is connected to the server device 9 from the server device 9. The acquired individual point data tp1 is recorded. That is, the receiving-side in-vehicle device 120d acquires the individual point data tp1 including the representative aspect of the point corresponding to the position of the vehicle A1d from the point data TP of the server device 9. Thus, the display control unit 201 controls the display mode of the captured image displayed on the display 24 according to the representative mode. Thereby, the optimal display mode of a picked-up image can be provided to the user who travels for the first time at a point corresponding to the position of the vehicle. Moreover, since it is not necessary to store the representative aspects of a plurality of points in advance, it is possible to reduce the capacity stored in the storage part 22d of the display device 20d, compared to the case where the representative aspects of a plurality of points are stored in the storage unit 22d. .

  In addition, when transmitted from the server device 9, the display mode of the captured image displayed on the display 24 according to the representative mode of the individual spot data tp 1 is stored in the storage unit 22 d without storing the individual spot data tp 1. May be controlled.

<1-6. Operation mode>
Next, operation modes of the transmission-side in-vehicle device 120 and the reception-side in-vehicle device 120d will be described. FIG. 7 is a diagram illustrating transition of operation modes of the transmission-side in-vehicle device 120 and the reception-side in-vehicle device 120d. The transmission-side in-vehicle device 120 and the reception-side in-vehicle device 120d have four operation modes: a navigation mode M0, a surrounding confirmation mode M1, a front mode M2, and a back mode M3. These operation modes can be switched by the control of the control unit 1 according to the user's operation and the traveling state of the vehicles A1 and A1d. In the following description of the operation mode, the processing of the vehicle A1d will be described as an example, but the same image conversion processing is performed for the vehicle A1.

The navigation mode M0 is an operation mode in which a map image for navigation guidance is displayed on the display 24 by the function of the display device 20d. In the navigation mode M0, the functions of the image processing apparatus 100 are not used, and various displays are performed using the functions of the display device 20d alone. For this reason, when the display device 20d has a function of receiving and displaying television broadcast radio waves, a television broadcast screen may be displayed instead of the map image for navigation guidance.

  On the other hand, in the surrounding confirmation mode M1, the front mode M2, and the back mode M3, a display image that shows the state of the surroundings of the vehicle A1d in real time is displayed on the display 24 by using the function of the image processing apparatus 100. This is an operation mode.

  The surrounding confirmation mode M1 is an operation mode in which an animation expression is performed such that the surroundings of the vehicle A1d circulate around the vehicle A1d while looking down. The front mode M2 is an operation mode for displaying a display image that mainly indicates the front or side of the vehicle A1d that is required when traveling forward. Further, the back mode M3 is an operation mode for displaying a display image mainly showing the rear of the vehicle A1d that is required at the time of reverse.

  When the receiving-side in-vehicle device 120d is activated, the surrounding confirmation mode M1 is first set. In the case of the surrounding confirmation mode M1, when a predetermined time (for example, 6 seconds) elapses after the animation expression that makes a round around the vehicle A1d is made, the mode is automatically switched to the front mode M2. Further, in the case of the front mode M2, for example, when the switching button 301 is continuously pressed for a predetermined time or more while the traveling speed is 0 km / h (stopped state), the surrounding confirmation mode M1 is switched. Note that the surrounding confirmation mode M1 may be switched to the front mode M2 in accordance with a predetermined instruction from the driver.

  Further, in the case of the front mode M2, when the traveling speed becomes, for example, 10 km / h or more, the mode is switched to the navigation mode M0. Conversely, when the traveling speed input from the vehicle speed sensor 82 is less than 10 km / h in the navigation mode M0, the mode is switched to the front mode M2.

  When the traveling speed of the vehicle A1d is relatively high, the front mode M2 is canceled in order to concentrate the driver on traveling. On the other hand, when the traveling speed of the vehicle A1d is relatively low, the driver operates in consideration of the situation around the vehicle A1d. Specifically, the driver enters an intersection with poor visibility, changes direction, or approaches the vehicle. There are a lot of scenes that do. For this reason, when the traveling speed is relatively low, the navigation mode M0 is switched to the front mode M2. When switching from the navigation mode M0 to the front mode M2, a condition that there is an explicit operation instruction from the driver may be added to the condition that the traveling speed is less than 10 km / h.

  In the navigation mode M0, when the switching button 301 is continuously pressed for a predetermined time or more in a state where the traveling speed is 0 km / h (stopped state), for example, the surrounding confirmation mode M1 is switched. Then, when a predetermined time (for example, 6 seconds) elapses after the animation expression that goes around the vehicle A1d is made, the mode automatically switches to the navigation mode M0 that is the original mode.

  In the navigation mode M0 or the front mode M2, when the position of the shift lever input from the shift sensor 81 is “R (reverse)”, the mode is switched to the back mode M3. That is, when the transmission of the vehicle A1d is operated to the “R (reverse)” position, since the vehicle A1d is in the reverse state, the vehicle is switched to the back mode M3 mainly showing the rear of the vehicle A1d.

  On the other hand, in the case of the back mode M3, when the position of the shift lever is other than “R (reverse)”, the mode is switched to the navigation mode M0 or the front mode M2 based on the traveling speed at that time. That is, if the traveling speed is 10 km / h or more, the mode is switched to the navigation mode M0, and if the traveling speed is less than 10 km / h, the mode is switched to the front mode M2. The above-described mode switching conditions (for example, the traveling speed is 0 km / h, the traveling speed is 10 km / h or more, etc.) are examples, and other conditions may be used.

  Next, a display mode around the vehicle A1d in the front mode M2 among these four modes will be described in detail. FIG. 8 is a diagram showing transition of display modes in the front mode M2. FIG. 9 is a diagram showing an area corresponding to an image displayed on the display 24. In addition, the area | region (R1-R10) of the periphery of vehicle A1d of FIG. 9 has shown the area | region around the vehicles A1 and A1d of the solid curved surface SP demonstrated previously using FIG. Each of the regions R1 to R10 is a rectangular region of a set of a plurality of pixels.

  In the front mode M2, there are three display modes: a traveling bird's-eye view mode M21, an own vehicle confirmation mode M22, and a side camera mode M23, and these display modes differ in the types of captured images displayed on each other. Of these screens, the M21, M22, and M23 screens display a field-of-view guide 90 that indicates the field-of-view range in each display mode, and which area around the vehicle A1d is displayed to the user. As shown.

  The display modes of the modes M21 to M23 are switched by the control of the control unit 1 in the order of the traveling bird's-eye view mode M21, the own vehicle confirmation mode M22, and the side camera mode M23 every time the user presses the switching button 301.

  In the traveling bird's-eye view mode M21, a composite image FP1 showing the state of the vehicle A1d viewed from the virtual viewpoint VP1 with the viewpoint position being directly above the center of the position of the vehicle A1d and the visual field direction being substantially directly below, and the photographing of the front camera 51 This is a display mode in which a screen including the front image FP2 obtained by the above is displayed on the display 24. That is, in the traveling bird's-eye view mode M21, there are two images, a composite image FP1 corresponding to the areas R1 to R10 around the vehicle A1d shown in FIG. 9 and a front image FP2 showing the front of the vehicle A1d that is a photographed image of the front camera 51. The image is shown on the same screen.

  In the traveling bird's-eye view mode M21, such two composite images FP1 and FP2 can be viewed, so that the user can confirm at a glance the situation in front of the traveling direction of the vehicle A1d along with the entire periphery of the vehicle A1d. . It can be said that the traveling bird's-eye view mode M21 is a display mode that can be used with high versatility in various scenes during forward movement.

  The own vehicle confirmation mode M22 includes a front image FP3 obtained by photographing with the front camera 51, and a virtual viewpoint VP2 in which the viewpoint position is substantially rearward of the position of the vehicle A1d and the visual field direction is substantially forward in the vehicle A1. This is a display mode in which a screen including the composite image FP4 showing the state of the seen vehicle A1d is displayed on the display 24. That is, in the own vehicle confirmation mode M22, two images of the front image FP3 showing the front of the vehicle A1d and the composite image FP4 corresponding to the peripheral areas R1 to R10 of the vehicle A1d are shown on the same screen.

  In this way, the composite image FP4 in the vehicle confirmation mode M22 corresponds to the same region (R1 to R10) as the composite image FP1 in the traveling bird's-eye view mode M21 in the region shown in FIG. The difference between the composite images FP1 and FP4 is the viewpoint position and viewpoint direction of the virtual viewpoint. As described above, the viewpoint position of the composite image FP1 is a position directly above the center of the position of the vehicle A1d, and the visual field direction is substantially directly below. Further, the viewpoint position of the composite image FP4 is a substantially rear position of the position of the vehicle A1d, and the visual field direction is substantially the front of the vehicle A1.

  Further, the front image FP3 in the vehicle confirmation mode M22 has a wider field of view in the left-right direction than the front image FP2 in the traveling bird's-eye view mode M21. For this reason, it is possible to confirm an object that exists forward and in the left-right direction from the front end of the vehicle A1d that is likely to become a blind spot when entering an intersection with poor visibility.

  Further, the composite image FP4 in the own vehicle confirmation mode M22 is moved to the rear of the vehicle A1d as compared with the composite image FP1 in the traveling bird's-eye view mode M21, so the region indicating the rear of the vehicle A1d is Although narrowed, the side of the vehicle A1d can be easily confirmed. For this reason, when passing the oncoming vehicle, the clearance with the oncoming vehicle can be easily confirmed.

  Since the two images FP3 and FP4 can be viewed in the own vehicle confirmation mode M22, the user needs careful driving such as when entering an intersection with poor visibility or when passing the oncoming vehicle. The situation of the area to be confirmed can be confirmed at a glance.

  The side camera mode M23 is a display mode in which a screen including side images FP5 and FP6 obtained by photographing with the left side camera 53 and the right side camera 54 is displayed on the display 24. The side images FP5 and FP6 show only the outside of the front fender 97 that tends to be a blind spot from the driver's seat.

  Since the two images FP5 and FP6 can be viewed in the side camera mode M23, the user can easily check the situation of the area to be confirmed when performing the width adjustment to bring the vehicle body to the end of the road. Can be confirmed.

<1-7. Database>
Next, vehicle information TA1 stored in the vehicle-mounted device 120a of the vehicle A1a, vehicle information TA2 stored in the vehicle-mounted device 120b of the vehicle A1b, vehicle information TA3 stored in the vehicle-mounted device 120c of the vehicle A1c, and the server device 9 The total data TD and the point data TP recorded in the data recording unit 92 will be described in detail.

  FIG. 10 is a diagram showing vehicle information TA1, TA2, and TA3. The vehicle information TA1 is data acquired when the user switches the display mode of the captured image displayed on the display 24 in the vehicle A1a, the vehicle information TA2 in the vehicle A1b, and the vehicle information TA3 in the vehicle A1c. Each data has items of “user ID”, “information acquisition date / time”, “position information”, “mode information”, and “image information”.

  “User ID” is an identification number of each vehicle. “Information acquisition date / time” is information on the date / time when the user switches the display mode of the captured image displayed on the display 24. “Position information” is information on a point corresponding to the position of the vehicle when the user switches the display mode of the captured image displayed on the display 24 (for example, latitude / longitude information of the vehicle A1).

  “Mode information” is information on a mode in which a captured image is displayed on the display 24 which is a display mode after the user switches the display mode of the captured image. In the vehicle information TA1 and TA2, the value of the own vehicle confirmation mode M22 is recorded, and in the vehicle information TA3, the value of the side camera mode M23 is recorded.

  “Image information” is at least one piece of information of a captured image and an area corresponding to an image displayed on the display 24 in each mode. In the vehicle information TA1 and TA2, information on areas R1 to R10 corresponding to the composite image FP1 and information on the front camera image corresponding to the image FP2 is recorded. In the vehicle information TA3, information of a left side camera image corresponding to the image FP5 and a right side camera image corresponding to the image FP6 is recorded.

  FIG. 11 is a diagram showing the total data TD recorded in the data recording unit 92 of the server device 9. Aggregated data TD is data obtained by aggregating vehicle information TA transmitted from a plurality of vehicles A1, and includes the same items (“user ID”, “information acquisition date / time”, “position information”, “mode” as the items of vehicle information TA. Information ”and“ image information ”). Note that the three pieces of data shown in the aggregate data TD in FIG. 11 are vehicle information having the same or nearby position information. In addition, the total data TD can total not only the position where the position information is the same and the vicinity but also the position information of different positions from a plurality of vehicles.

  FIG. 12 is a diagram illustrating the point data TP recorded in the data recording unit 92 of the server device 9. The point data TP is used by the specifying unit 912 when the vehicle unit information of the position information in the total data TD that is the same or in the vicinity is acquired by the acquiring unit 911 of the server device 9 exceeding a predetermined number. This is data in which the highest display mode is specified as a representative mode and the representative mode is recorded in association with position information.

  Specifically, as shown in FIG. 12, the point data TP has items of “position information”, “mode information”, and “image information”. Note that the data for each point recorded in the point data TP is the individual point data tp, and a plurality of individual point data tp1 and tp2 are recorded in the point data TP in FIG. “Position information” is information on a point corresponding to the position of the vehicle when the user switches the display mode of the captured image displayed on the display 24.

  The “mode information” is displayed on the display 24 after the user switches the display mode of the photographed image at the position in association with the position information when the user switches the display mode of the photographed image displayed on the display 24. This mode information is the most frequently used mode information. That is, the value of the mode information with the highest use frequency of the user is recorded among any of the data having the same position information and the vicinity in the total data TD.

  “Image information” is information corresponding to the mode information, and the display mode of the captured image displayed on the display 24 is associated with the position information when the user switches, and the display mode of the captured image at the position is displayed by the user. Of the image information displayed on the display 24 after switching, the image information is the most frequently used. That is, the image information with the highest use frequency of the user is recorded among any of the data having the same position information and the vicinity in the total data TD.

<1-8. Processing flowchart>
FIG. 13 is a process flowchart for transmitting the vehicle information TA from the transmission-side in-vehicle device 120 to the server device 9. When the mode switching operation of the captured image displayed on the display 24 is performed by the user (Yes in Step S101), the process proceeds to Step S102. When the mode switching operation by the user is not performed (No in step S101), the transmission-side in-vehicle device 120 continues to display the captured image on the display 24 in the current mode.

  In step S102, the transmission-side in-vehicle apparatus 120 displays a captured image in a mode corresponding to the user switching operation on the display 24 (step S102), and the process proceeds to step S103.

  In step S103, when the captured image in the mode corresponding to the user switching operation is displayed for, for example, more than 3 seconds, that is, when the user has not performed the mode switching operation for more than 3 seconds (Yes in step S103). ) Proceeds to the process of step S104. If the user switches to another mode within 3 seconds (No in step S103), the process returns to step S102, and a captured image in another mode corresponding to the user's switching operation is transmitted. The in-vehicle device 120 displays on the display 24. Further, the time (3 seconds) for determining the presence / absence of the mode switching operation can be changed to another time (for example, 2 seconds, 5 seconds, etc.).

  In step S104, the position deriving unit 202 of the display device 20 derives the position of the vehicle A1 when the user switches the display mode of the captured image displayed on the display 24 (step S104), and the process proceeds to step S105. .

  In step S105, vehicle information TA, which is information including the positional information of the vehicle A1 when the user switches the display mode of the captured image displayed on the display 24 and the display mode of the captured image after the switching by the user, is stored in the server. It transmits to the apparatus 9 (step S104).

  In the server device 9, the acquisition unit 911 acquires the vehicle information TA transmitted from the transmission-side in-vehicle device 120 (step S201), and records the vehicle information TA in the data recording unit 92 as aggregated data TD (step S202). .

  FIG. 14 and FIG. 15 are process flowcharts for transmitting the individual point data tp1 from the server device 9 to the receiving in-vehicle device 120d. The position deriving unit 202 of the display device 20d in the receiving-side in-vehicle device 120d derives the position of the vehicle A1d as the vehicle A1d travels (step S301), and transmits position data corresponding to the position of the vehicle A1d to the server device 9. (Step S302).

  The server device 9 acquires position data from the receiving-side in-vehicle device 120d (step S401), and proceeds to the process of step S402.

  In step S402, when the point data corresponding to the position of the vehicle A1 in the position data acquired from the receiving-side in-vehicle device 120d is recorded in the point data TP of the data recording unit 92 of the server device 9 (Yes in step S402). Then, individual point data tp1, which is point data corresponding to the position of the vehicle A1d, is transmitted from the point data TP to the receiving-side in-vehicle device 120d (step S403). That is, mode information that is a representative mode of position information corresponding to the position of the vehicle A1d and information including image information are transmitted from the point data TP to the receiving-side in-vehicle device 120d. Note that if the individual spot data tp corresponding to the position of the vehicle A1d is not recorded in the spot data TP of the data recording unit 92 (No in step S402), the process ends.

  The in-vehicle device 120d on the receiving side acquires the individual point data tp1 from the server device 9 (step S303), and proceeds to the process of step S304.

  In step S304, the display control unit 201 switches the mode for displaying the captured image based on at least one of the mode information and the image information recorded in the individual point data tp1 (step S304), and the process proceeds to step S305. Since the mode is switched to the mode with the highest use frequency in this way, the situation around the vehicle can be confirmed in a mode that is optimal for the traveling environment at the position of the vehicle A1d without complicated operations.

  In step S305, when the position of the vehicle A1d is not a position that has passed the point of the individual point data tp1 (step S305 is No), the process proceeds to step S307. When the position of the vehicle A1d is a position that has passed the point of the individual point data tp (Yes in step S305), the display control unit 201 performs control to return to the mode before switching by the user before switching the mode. Is displayed on the display 24 (step S306).

  In step S307, when the mode switching operation is performed by the user's operation before the vehicle A1d passes the point corresponding to the individual point data tp1 (step S307 is Yes), the process proceeds to step S308. In addition, when the mode switching operation by a user's operation is not performed (step S307 is No), it returns to the process of step S305.

  In step S308, a captured image in a mode corresponding to the user switching operation is displayed on the display 24 (step S308), and the process proceeds to step S309.

  In the process of step S309, if the user's mode switching operation is not performed for more than 3 seconds (step S309 is Yes), the process proceeds to step S310. If the user performs a switching operation to another mode within 3 seconds (No in Step S309), the process returns to Step S308 to display a captured image of another mode corresponding to the user switching operation on the display 24. To display.

  In step S310, the position deriving unit 202 of the display device 20d derives the position of the vehicle A1d when the user switches the display mode of the captured image displayed on the display 24 (step S310), and the process proceeds to step S311. .

  In step S311, vehicle information that is information including the positional information of the vehicle A1d when the user switches the display mode of the captured image of the in-vehicle camera displayed on the display 24 and the display mode of the captured image after the switching by the user. Is transmitted to the server device 9 (step S310).

In the server device 9, the acquisition unit 911 acquires vehicle information from the receiving-side in-vehicle device 120d (step S404), and records the vehicle information in the data recording unit 92 as aggregated data TD (step S405). As described above, the information on the representative aspect of the point corresponding to the position of the vehicle A1d is acquired from the server device 9, and a new operation associated with the switching operation of the user is performed also from the receiving in-vehicle device 120d that switches the image displayed on the display 24. Vehicle information can be transmitted to the server device 9. Then, by accumulating the data transmitted to the server device 9, it is possible to accumulate vehicle information for deriving an optimal display mode of the captured image at the point.
<Second Embodiment>
Next, a second embodiment will be described. The differences between the second embodiment and the first embodiment are as follows. In 1st Embodiment, when the server apparatus 9 transmits the point data corresponding to the position of vehicle A1d to the receiving side vehicle-mounted apparatus 120d, it is a receiving side vehicle-mounted apparatus among the display modes for every some point of the point data TP. The individual spot data tp1 including the representative aspect of the spot corresponding to the position of the vehicle A1d received from 120d was transmitted.

  On the other hand, in 2nd Embodiment, regardless of the position of vehicle A1d, the display mode of all the points is transmitted to vehicle A1d at once among the display modes for every some point of point data TP. Hereinafter, a second embodiment will be described. In the second embodiment, the system configuration is substantially the same as in the first embodiment, and the processing is different. Below, it demonstrates centering around difference.

<2-1. System configuration>
FIG. 16 is a block diagram of the receiving-side in-vehicle device 121d. The same data as the point data TP recorded in the data recording unit 92 of the server device 9 is recorded in the storage unit 22d of the display device 20d. That is, the point data TP, which is data of all representative aspects, is transmitted from the server device 9 to the receiving-side in-vehicle device 121d at a predetermined timing and stored in the storage unit 22d. Since the representative aspects of all the points are recorded in advance as described above, the reception-side in-vehicle device 121d alone corresponds to the position of the vehicle A1d without performing communication with the server device 9 according to the position of the vehicle A1d. The display mode of the display 24 can be automatically switched according to the representative mode of the point. The other configuration of the receiving-side in-vehicle device 121d is the same as that of the receiving-side in-vehicle device 120d shown in FIG.

<2-2. Processing flowchart>
17 and 18 are process flowcharts for transmitting the point data TP from the server device 9 to the receiving-side in-vehicle device 121d. The processes of FIGS. 17 and 18 are different from the processes of FIGS. 14 and 15 described in the first embodiment.

  The server device 9 transmits all the point data TPs that are all representative aspects recorded in the data recording unit 92 to the receiving-side in-vehicle device 121d at a time (step S601). That is, the representative mode data of all points recorded in the point data TP is transmitted from the server device 9 to the receiving-side in-vehicle device 121d.

  The reception-side in-vehicle device 121d acquires the spot data TP transmitted from the server device 9 (step S501), and proceeds to the process of step S301.

  In step S301, the position deriving unit 202 derives the position of the vehicle A1d (step S301), and the process proceeds to step S502.

  In step S502, when the point corresponding to the position of the vehicle A1d is present in the point data TP stored in the storage unit 22d of the display device 20d (step S502 is Yes), the process proceeds to step S503.

  In step S503, the display control unit 201 switches the mode for displaying the captured image based on the spot data TP stored in the storage unit 22d (step S503), and the process proceeds to step S305.

Note that if the point data corresponding to the position of the vehicle A1d does not exist in the point data TP (Yes in Step S502), the mode displayed on the display 24 by the display control unit 201 is not switched and Step S301 is performed. Returning to the process, the position deriving unit 202 derives the position of the vehicle A1d (step S301).
<Modification>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible. Below, such a modification is demonstrated. In addition, all the forms including the form demonstrated in the said embodiment and the form demonstrated below are combinable suitably.

  In the above-described embodiment, the transmission-side in-vehicle device 120 transmits vehicle information TA to the server device 9 for one point, and the reception-side in-vehicle device 120d that acquires the representative mode transmitted from the server device 9 displays the representative mode. It has been described that mode switching is performed. In addition to this, different combinations are possible for the in-vehicle device that transmits vehicle information to the server device 9 and the in-vehicle device that acquires the representative form from the server device 9.

  For example, the vehicle-mounted devices 120b, 120c, and 120d may transmit vehicle information to the server device 9 at one point, and the vehicle-mounted device 120a may acquire a representative aspect transmitted from the server device 9. The vehicle-mounted devices 120a, 120b, and 120c transmit vehicle information to the server device 9 at one point, and the vehicle A1a equipped with the vehicle-mounted device 120a travels at a different timing from the vehicle information transmitted at the same point. In this case, the in-vehicle device 120a may acquire the representative aspect transmitted from the server device 9.

  Moreover, in the said embodiment, one vehicle-mounted apparatus may be provided with the function of both the transmission side vehicle-mounted apparatus 120 mentioned above and the receiving side vehicle-mounted apparatus 120d.

  In the above embodiment, some of the functions described as being realized by the transmission-side in-vehicle device 120 and the reception-side in-vehicle device 120d may be realized by the server device 9.

  In the above embodiment, some of the functions described as being realized by the server device 9 may be realized by any one of the transmission-side in-vehicle device 120 and the reception-side in-vehicle device 120d.

  In the above embodiment, the “mode information” and “image information” described as items indicating the display mode (representative mode) in the vehicle information TA, the total data TD, and the point data TP are displayed. Only information that can discriminate the type of mode (representative mode) may be recorded. That is, only “mode information” may be recorded, or only “image information” may be recorded. Further, information on the virtual viewpoint of the composite image may be recorded as information indicating the display mode. That is, information on the viewpoint position of the virtual viewpoint and the viewing direction may be recorded.

  In the above-described embodiment, any or all of the signals described as being input to the control unit 1 of the image processing apparatus 100 via the signal input unit 42 are input to the display device 20. It may be. In this case, the signal may be input to the control unit 1 of the image processing apparatus 100 via the communication unit 41.

  In the above embodiment, the type of the on-vehicle device mounted on the vehicle is recorded in advance in the data recording unit 92 of the server device 9, and the representative mode corresponding to the type of the image system mounted on the vehicle is shown. Information may be transmitted from the server device 9 to the in-vehicle device. That is, at least one piece of mode information that differs depending on the type of the in-vehicle device and image information that differs depending on the type of the in-vehicle device may be transmitted from the server device 9 according to the type of the in-vehicle device.

DESCRIPTION OF SYMBOLS 1 ... Control part 3 ... Image processing part 5 ... Image pick-up part 10 ... Main part 20 ... Display apparatus

Claims (12)

An information processing system,
A first in-vehicle device that is used in a vehicle and is capable of displaying captured images around the vehicle in different display modes;
A second in-vehicle device that is used in a vehicle and is capable of displaying captured images around the vehicle in different display modes;
A server device capable of communicating with the first in-vehicle device and the second in-vehicle device;
With
(A) The first in-vehicle device is:
(A-1) acquisition means for acquiring the current vehicle position;
(A-2) Transmitting means for transmitting vehicle information including the position of the vehicle when the user switches the display mode of the captured image and the display mode after switching to the server device;
With
(B) The server device
(B-1) acquisition means for acquiring the vehicle information from the first in-vehicle device of a plurality of vehicles;
(B-2) Based on the vehicle information, specifying means for specifying a representative aspect that is a display aspect having the highest use frequency at the position of the vehicle indicated by the vehicle information;
(B-3) storage means for storing the position of the vehicle and the representative form of the position in association with each other;
(B-4) transmitting means for transmitting the representative aspect to the second in-vehicle device;
With
(C) The second in-vehicle device is
(C-1) acquisition means for acquiring the current position of the vehicle;
(C-2) display means for displaying the photographed image in the representative mode corresponding to the current vehicle position;
An information processing system comprising: An information processing system,
An in-vehicle device that is used in a vehicle and can display captured images of the periphery of the vehicle in different display modes;
A server device capable of communicating with the in-vehicle device;
With
(A) The in-vehicle device is:
Obtaining means for obtaining the current vehicle position;
Transmitting means for transmitting vehicle information including the position of the vehicle when the user switches the display mode of the captured image and the display mode after switching to the server device;
With
(B) The server device
Obtaining means for obtaining the vehicle information from the in-vehicle devices of a plurality of vehicles;
Based on the vehicle information, specifying means for specifying a representative aspect that is a display aspect having the highest use frequency at the position of the vehicle indicated by the vehicle information;
An information processing system comprising: An information processing system,
An in-vehicle device that is used in a vehicle and can display captured images of the periphery of the vehicle in different display modes;
A server device capable of communicating with the in-vehicle device;
With
(A) The server device
Storage means for storing the position of the vehicle in association with a representative aspect that is the display aspect having the highest use frequency among the different display aspects at the position;
Transmitting means for transmitting the representative aspect to the in-vehicle device;
With
(B) The in-vehicle device is
Obtaining means for obtaining the current vehicle position;
Display means for displaying the captured image in the representative mode corresponding to the current vehicle position;
An information processing system comprising: A server device capable of communicating with an in-vehicle device that is used in a vehicle and can display captured images around the vehicle in different display modes,
Acquisition means for acquiring vehicle information including the position of the vehicle when the user switches the display mode of the captured image and the display mode after switching from the in-vehicle device of a plurality of vehicles;
Based on the vehicle information, specifying means for specifying a representative aspect that is a display aspect having the highest use frequency at the position of the vehicle indicated by the vehicle information;
A server device comprising: A server device capable of communicating with an in-vehicle device that is used in a vehicle and can display captured images around the vehicle in different display modes,
Storage means for storing the position of the vehicle in association with a representative aspect that is the display aspect having the highest use frequency among the different display aspects at the position;
Transmitting means for transmitting the representative aspect to the in-vehicle device;
A server device comprising: The server device according to claim 5,
Obtaining means for obtaining a position of a vehicle in which the in-vehicle device is used;
Further comprising
The server device according to claim 1, wherein the transmitting unit transmits the representative aspect corresponding to a position of a vehicle where the in-vehicle device is used. The server device according to claim 5,
The server device is characterized in that the transmission unit transmits all the representative aspects stored in the storage unit at a time. An in-vehicle device that can be used in a vehicle and can display captured images around the vehicle in different display modes,
Obtaining means for obtaining the current vehicle position;
Transmitting means for transmitting vehicle information including the position of the vehicle when the user switches the display mode of the captured image and the display mode after switching to a predetermined server device;
A vehicle-mounted device comprising: An in-vehicle device that can be used in a vehicle and can display captured images around the vehicle in different display modes,
First acquisition means for acquiring a current vehicle position;
Second acquisition means for acquiring the representative aspect from a server device that associates and stores the position of the vehicle and the representative aspect that is the display aspect having the highest use frequency among the different display aspects at the position;
Display means for displaying the captured image in the representative mode corresponding to the current vehicle position;
A vehicle-mounted device comprising: The in-vehicle device according to claim 9,
The in-vehicle device, wherein the second acquisition unit acquires the representative aspect corresponding to the position of the vehicle acquired by the first acquisition unit. The in-vehicle device according to claim 9,
Storage means for storing the representative aspect;
Further comprising
The second acquisition means acquires all the representative aspects stored in the storage means at a time,
An in-vehicle device that is stored in the storage means. An information processing system,
Display means used in a vehicle and capable of displaying captured images around the vehicle in different display modes;
Obtaining means for obtaining the current vehicle position;
Based on vehicle information including the position of the vehicle when the user switches the display mode of the captured image and the display mode after switching, the display mode with the highest use frequency at the position of the vehicle indicated by the vehicle information A specifying means for specifying a representative aspect of:
Storage means for storing the vehicle position and the representative form of the position in association with each other;
With
The display means displays the captured image in the representative form corresponding to the current vehicle position;
An information processing system characterized by

Images