TW202215365A - Image processing apparatus and image display apparatus - Google Patents

Abstract

An image processing apparatus according to an aspect of the present invention comprises: an information acquisition means that acquires driver posture information indicating the posture of a driver of a vehicle, and vehicle state information indicating the state of the vehicle; a first image generation means that, regarding some vehicle and a driver thereof, generates a first composite image by superimposing a first driver image based on the driver posture information and a first vehicle image based on the vehicle state information; a second image generation means that, regarding a vehicle traveling independently of the some vehicle and a driver thereof, generates a second composite image by superimposing a second driver image based on the driver posture information and a second vehicle image based on the vehicle state information; and an association means that associates the first composite image and the second composite image on the basis of predetermined information.

Description

Image processing device and image display device

The present invention mainly relates to image processing apparatuses.

Patent Document 1 describes an image processing technique in which a virtual viewpoint is set in an image representing a situation of a vehicle and its surroundings, and the image can be visually recognized while changing the virtual viewpoint. In Patent Document 1, an example of the application of this technology is for theft prevention. [Prior Art Literature] [Patent Literature]

[Patent Document 1] Japanese Patent Laid-Open No. 2015-76062

(The problem that the invention intends to solve)

Further technical improvement is required when further diversification of image applications can be obtained by the above-mentioned image processing technology.

The present invention is intended to be an illustration that can be achieved relatively easily by the above-mentioned image processing technology to obtain a variety of image applications. (Technical means to solve problems)

An aspect of the present invention relates to an image processing device, characterized in that it includes: information acquisition means for acquiring driver posture information indicating a posture of a driver of a vehicle, and vehicle state information indicating a state of the vehicle; A first image generating means for generating a first composite image of a vehicle and its driver by superimposing a first driver image based on the driver posture information and a first vehicle image based on the vehicle state information image; a second image generating means for generating a second driver image based on the above-mentioned driver posture information and a second vehicle image based on the above-mentioned vehicle state information for a vehicle and its driver that are traveling independently of the above-mentioned vehicle. superimposing the images to generate a second composite image; and associating means for relating the first composite image and the second composite image based on predetermined information. (Compared to the efficacy of the prior art)

According to the present invention, it is possible to facilitate the diversification of the use of images obtained by using the above-mentioned image processing technology.

Other features and advantages of the present invention will be clarified by the following description with reference to the accompanying drawings. In addition, in the drawings, the same or the same components are denoted by the same reference numerals.

Hereinafter, the embodiment will be described in detail with reference to the drawings. Furthermore, the following embodiments are not intended to limit the scope of the patent application, and all combinations of features described in the embodiments are not necessarily required for the present invention. Two or more of the plural features described in the embodiments may be arbitrarily combined. In addition, the same or the same structure is attached|subjected to the same element number, and the repeated description is abbreviate|omitted.

(Example of image display system) FIG. 1 is a schematic diagram showing a configuration example of an image display system SY according to the embodiment. The image display system SY includes a vehicle 1, an image processing device 2, and a terminal 3, and these can communicate with each other via a network N in this embodiment.

The vehicle 1 is a saddle-riding type vehicle in this embodiment. Here, a saddle-riding vehicle refers to a type in which the driver rides across the vehicle body. In addition to typical two-wheeled vehicles (including scooter-type vehicles), the concept also includes, for example, three-wheeled vehicles (front wheel and rear two wheels, or Vehicles with the first two wheels and the rear wheel), and all-terrain support vehicles (ATVs) like four-wheel off-road vehicles. Furthermore, as another embodiment, the vehicle 1 may be a passenger vehicle. The vehicle 1 includes a camera device 11A, a camera device 11B, a vehicle state detection device 12 , a vehicle position identification device 13 , and a communication device 14 .

As illustrated in FIG. 2 , a plurality of camera devices 11A are provided in the peripheral portion of the vehicle body so as to capture an image showing a situation around the vehicle 1 , and the plurality of camera devices 11A include the vehicle in the shooting area. 1 The way of the global area of the periphery is set. That is, the plurality of imaging devices 11A are installed so that the imaging regions of the two adjacent imaging devices 11A partially overlap each other. In the figure, although the pointing direction of the imaging device 11A is schematically shown by a dotted line, the actual detection range of the imaging device 11A is set to be larger than that shown in the figure.

The imaging device 11B is installed in the front and rear of the driver's seat so that the driver can be photographed from the front and the rear, respectively. In the figure, similarly to the imaging device 11A, the pointing direction of the imaging device 11B is schematically shown by a dotted line, but the actual detection range of the imaging device 11B is set to be larger than the illustrated range. Although the relevant details will be described later, the driver's posture (including behavior, etc.) and attached appearance (including clothing, etc.) can be photographed by this method.

The imaging devices 11A and 11B may use well-known cameras composed of a CCD (Charge Coupled Device)/CMOS (Complementary Metal Oxide Semiconductor) image sensor or the like. In addition, in the present embodiment, as an example, a single-lens camera is used in order to reduce the cost required for the imaging devices 11A and 11B.

Referring again to FIG. 1 , the vehicle state detection device 12 is provided at each part of the vehicle body so as to be able to detect the state of the vehicle 1 . In the present embodiment, the state of the vehicle 1 includes the vehicle speed, the steering angle (or steering angle), the posture of the vehicle body, and the state of the lamp body (headlight, taillight, turn signal, etc.). The vehicle state detection device 12 may also be represented as a state detection device or only as a detection device.

The vehicle speed is detected, for example, based on the number of revolutions of the wheels per unit time, which can be realized by using a well-known speed sensor. The steering angle can be detected, for example, from the direction of the steering wheel relative to the vehicle body (or the direction of the handlebar relative to the vehicle body), which can be achieved by using a well-known steering angle sensor. The posture of the vehicle body can be detected, for example, according to the orientation of the vehicle body with respect to the direction of gravity, which can be realized by using a well-known acceleration sensor. Also, the state of the lamp body can be detected based on, for example, the conduction state of the light source, and this can be realized by using a well-known galvanometer.

The vehicle position specifying means 13 specifies the position of the vehicle 1 on the travel path. The travel path indicates a road on which the vehicle 1 is currently traveling, and the vehicle position specifying means 13 indicates the position of the vehicle 1 on the map data. The vehicle position specific device 13 may typically use a GPS (Global Positioning System) sensor. The vehicle location specific device 13 may also be represented as a location specific device or only as a specific device.

The communication device 14 transmits the photographing results of the camera devices 11A and 11B, the detection results of the vehicle state detection device 12 , and the identification results of the vehicle position identification device 13 to the image processing device 2 via the network N. The communication device 14 may be represented by a signal transmitting/receiving device or the like, and may be represented only by a signal transmitting device in this embodiment.

The photographing result of the imaging device 11A displays information (hereinafter referred to as vehicle surrounding information 90A) indicating the situation around the vehicle 1 . In this embodiment, the vehicle peripheral information 90A is set as image information or image data.

The photographing result of the imaging device 11B is displayed as information indicating the driver's posture (hereinafter referred to as driver posture information 90B). In this embodiment, the driver posture information 90B is set as image information or image data.

The detection result of the vehicle state detection device 12 is displayed as information indicating the state of the vehicle 1 (hereinafter referred to as vehicle state information 90C). In the present embodiment, the vehicle state information 90C is a signal group representing the vehicle speed, the steering angle, the posture of the vehicle body, and the state of the lamp body.

Information (hereinafter referred to as vehicle position information 90D) indicating the position of the vehicle 1 on the travel path is displayed according to the identification result of the vehicle position identification device 13 . In the present embodiment, the vehicle position information 90D can be obtained from GPS, and is set as a signal group representing the coordinates on the map data.

As shown in FIG. 1 , the above-mentioned information 90A to 90D are transmitted from the vehicle 1 to the image processing device 2 via the network N.

The image processing device 2 includes a communication unit 21 and an arithmetic unit 22 . The communication unit 21 enables the image processing device 2 to communicate with each of the vehicle 1 and the terminal 3 via the network N. The details of the arithmetic unit 22 will be described later, but it performs predetermined arithmetic processing including image processing. In the present embodiment, the computing unit 22 is a processor including a CPU (Central Processing Unit) and a memory, and the function thereof can be realized by executing a predetermined program. That is, the program only needs to be read through a network or a storage medium and executed on a computer.

Furthermore, as another embodiment, the computing unit 22 may be constituted by a semiconductor device such as a PLD (programmable logic device), an ASIC (application specific integrated circuit) or the like. That is, the function of the computing unit 22 can be realized by either hardware or software.

In the present embodiment, the terminal 3 is a mobile terminal (for example, a smartphone), and includes a communication unit 31 , an operation unit 32 , and a display unit 33 . The user of the terminal 3 may be the driver of the vehicle 1 or a third party different from the driver. The communication unit 31 is configured to allow the terminal 3 to communicate with the image processing apparatus 2 via the network N. The operation unit 32 is configured to accept the user's operation input, and for example, a known operation panel such as a touch sensor type operation panel and a button/switch type operation panel can be used for the operation unit 32 . Moreover, the display part 33 is set so that an image can be displayed, and a well-known display, such as a liquid crystal display and an organic EL (Electroluminescence) display, can be used for the display part 33, for example. The operation unit 32 and the display unit 33 are provided as one unit (touch panel type display) in this embodiment, but may be provided separately in another embodiment.

Although the details will be described later, in this image display system SY, the vehicle 1 can communicate with the image processing device 2 , and the photographing results of the camera devices 11A and 11B and the detection results of the vehicle state detection device 12 are recorded. , and the specific result of the vehicle position specific device 13 is sent to the image processing device 2 . The image processing device 2 generates a composite image (hereinafter referred to as composite image 90X) by performing predetermined image processing by the arithmetic unit 22 based on these, and transmits the composite image 90X to the terminal 3 . The terminal 3 functions as an image display device, and the user can visually recognize the composite image 90X on the display unit 33 while operating and inputting the terminal 3 to the operation unit 32 .

(Example of image processing method) FIG. 3 is a flowchart showing an example of an image processing method for generating the composite image 90X. The contents of this flowchart are mainly executed by the arithmetic unit 22, and the outline thereof is to generate the composite image 90X based on the above-mentioned information 90A to 90D. In addition, although this flowchart is set to be executed after the use of the vehicle 1 (when not running), it may be executed when the vehicle 1 is used (while running).

In step S1000 (hereinafter abbreviated as "S1000", the same applies to other steps described later), the vehicle peripheral information 90A is acquired. As described above, the vehicle surrounding information 90A can be obtained by a plurality of camera devices 11A, and the plurality of camera devices 11A are installed in the peripheral portion of the vehicle body such that the shooting area includes the entire area around the vehicle 1 . Thereby, the vehicle surrounding information 90A can represent the situation of the whole area around the vehicle 1, that is, image information or image data representing a so-called panoramic view (360-degree panoramic view) can be obtained. Although the details will be described later, an image showing a situation around the vehicle 1 corresponding to the vehicle surrounding information 90A can be generated as the vehicle surrounding image 90P. Therefore, the image processing using the vehicle peripheral information 90A can be implemented relatively simply by using the spherical coordinate system.

In step S1010, the driver posture information 90B is obtained. As described above, the driver's posture information 90B can be obtained by a pair of camera devices 11B arranged in front of and behind the driver's seat, and the pair of camera devices 11B are arranged to photograph the driving from the front and the rear people. Therefore, the driver posture information 90B can be obtained as image information or image data representing the driver's driving state, such as the driver's posture (in the case of a still image), behavior (in the case of a moving image), etc. Appearance (eg, physique, clothing (wear, helmet, etc.)) can also be additionally indicated. The details will be described later, but the image of the driver corresponding to the driver posture information 90B is generated as the driver image 90Q. Therefore, the image processing of the driver posture information 90B can be performed relatively simply by using a three-dimensional coordinate system based on a predetermined human body model.

In step S1020, the vehicle state information 90C is obtained. As mentioned above, the vehicle state information 90C can be obtained by detecting the state of the vehicle 1 by the vehicle state detection device 12 installed in each part of the vehicle body, and the state of the vehicle 1 includes vehicle speed, steering angle, and posture, and the state of the lamp body. Although the details will be described later, the image of the vehicle 1 corresponding to the vehicle state information 90C generates the vehicle image 90R. Therefore, the image processing on the vehicle image 90R can be performed relatively simply by using the three-dimensional coordinate system according to the corresponding vehicle model.

In step S1030, images 90P to 90R are respectively generated according to the above-mentioned information 90A to 90C. That is, the vehicle surrounding image 90P is generated from the vehicle surrounding information 90A, the driver image 90Q is generated from the driver posture information 90B, and the vehicle image 90R is generated from the vehicle state information 90C.

In step S1040, the images 90P, 90Q, and 90R are superimposed to generate a composite image 90X. As described above, in this embodiment, the vehicle surrounding image 90P is processed by the spherical coordinate system, and the driver image 90Q and the vehicle image 90R are processed by the three-dimensional coordinate system.

Here, the three-dimensional coordinate system can typically use the distance x from the coordinate center to the front-rear direction of the object's vehicle body, the distance y from the coordinate center to the left-right direction of the object's vehicle body, and the coordinate center to the object's vehicle body. The distance z is represented by coordinates (x, y, z). In addition, the spherical coordinate system can typically use the distance r from the coordinate center to the object, the angle θ formed by the line passing through the coordinate center to the object and the vertical direction of the vehicle body, and the line passing through the coordinate center-object and the vehicle body. The angle ϕ formed by the front and rear directions is represented by coordinates (r, θ, ϕ).

FIG. 4A is a schematic diagram showing a vehicle surrounding image 90P. The vehicle surrounding image 90P is processed using a spherical coordinate system, and is drawn at a distance r from the coordinate center. In other words, the vehicle surrounding image 90P as a panoramic view can be drawn in the shape of the inner wall of a sphere of radius r. The above-mentioned r may be set so as to be a position outside the vehicle 1 .

FIG. 4B is a schematic diagram showing the driver image 90Q. The driver image 90Q is processed using a three-dimensional coordinate system, which can depict the head, shoulders, torso (chest and abdomen), waist, arms (upper arm and forearm), Hands, feet (thighs and calves), feet, etc. It is also possible to further describe the clothing incidentally.

FIG. 4C is a schematic diagram showing a vehicle image 90R. The vehicle image 90R is processed using a three-dimensional coordinate system, for example, an image of the vehicle 1 can be drawn according to the state of the vehicle state information 90C (information representing vehicle speed, steering angle, posture of the vehicle body, and state of the lamp body). For example, regarding the vehicle 1 during cornering, the vehicle image 90R may be drawn in a posture where the vehicle body is inclined.

Here, since the vehicle state information 90C includes information indicating the posture of the vehicle body, the photographing results of the camera devices 11A and 11B can also be corrected according to the degree of inclination of the vehicle body. For example, when the image 90P around the vehicle is acquired by the camera 11A while the vehicle body is inclined at the inclination angle λ1, the image 90P can be rotated by the angle λ1. The correction processing performed on the photographing results of the imaging devices 11A and 11B may be performed in the image processing device 2 or in the vehicle 1 .

FIG. 4D is a schematic diagram showing the composite image 90X. The images 90P to 90R may be synthesized so that the coordinate centers, distances, and directions may match. In addition, although the coordinate center is set to the position directly above the seat in this embodiment, it may be other positions (for example, any position of the vehicle body) in other embodiments.

Referring again to FIG. 3 , in step S1050 , vehicle position information 90D is acquired, and information indicating the vehicle is assigned to composite image 90X, images 90P to 90R constituting composite image 90X, and information 90A to 90C for generating these 1. Vehicle position information 90D of the position on the travel path. Although the details will be described later, the composite image 90X (or the information 90A to 90C and/or the images 90P to 90R used in the generation of the composite image 90X) can be combined with other composite images 90X (or The information 90A- 90C and/or the images 90P- 90R) used in its generation are associated.

In step S1060 , the composite image 90X is sent to the terminal 3 . The user of the terminal 3 can display the composite image 90X on the display unit 33 from a viewpoint (hereinafter referred to as a virtual viewpoint) from an arbitrary position by operating input to the operation unit 32 . In addition, the user can also perform zoom in or zoom out (zoom out) of the composite image 90X by operating input to the operation unit 32 .

To summarize the above, in steps S1000 to S1020, the computing unit 22 functions as an information acquisition unit that acquires the vehicle surrounding information 90A, the driver posture information 90B, and the vehicle state information 90C. In steps S1030 to S1040 , the computing unit 22 generates an image that generates the vehicle surrounding image 90P, the driver image 90Q, and the vehicle image 90R, and superimposes these images 90P to 90R to generate the composite image 90X. Department functions. In S1050, the computing unit 22 functions as an association unit that associates a certain composite image 90X with any one of the plurality of composite images 90X generated in the past using the vehicle position information 90D. In addition, in step S1060, the arithmetic unit 22 functions as a signal transmission unit that transmits the composite image 90X.

Furthermore, in the description of the above steps S1000 to S1060, the acquisition of the information 90A to 90D means that the CPU of the computing unit 22 reads the information 90A to 90D from the memory, and the image processing device 2 can automatically The vehicle 1 collectively receives the information 90A to 90D.

(Example of composite image) FIG. 5A shows an example of a composite image 90X when the operation unit 32 and the display unit 33 are integrated into a touch-panel display, and FIG. 5B shows another example of the composite image 90X (an imaginary viewpoint different from that of FIG. 5A ) Example of composite image 90X below). Icons 8 a and 8 b for changing a virtual viewpoint, an icon 8 c for zooming in, and an icon 8 d for zooming out are displayed on the display unit 33 as a part of the operation unit 32 . By performing a predetermined operation input (such as a single-point touch (tap) operation, a swipe (swipe) operation, a fast drag (flick) operation, etc.) on the icons 8a, the user can view the icons from a desired imaginary point of view. Identify the situation of the vehicle 1 and its surroundings.

When the virtual viewpoint is changed, the size of the vehicle image 90R in the composite image 90X is changed, and the size of the driver image 90Q is also changed, so the composite image with the change is observed. Results like 90X will cause less dissonance. On the other hand, since the vehicle surrounding image 90P is processed using the spherical coordinate system, when the virtual viewpoint is changed, the size of the vehicle surrounding image 90P in the composite image 90X can be maintained.

Furthermore, in the image processing of the vehicle surrounding image 90P, by setting the distance r in the spherical coordinate system to a relatively large value, it is possible to reduce the sense of incongruity with the change of the vehicle surrounding image 90P after changing the virtual viewpoint ( such as deformation). In this case, by using the camera device 11A with a relatively large number of pixels, the vehicle surrounding image 90P can be clearly displayed.

In the above-described step S1040 (generation of the composite image 90X), the images 90P to 90R are required to represent approximately the same time as each other. Therefore, in the present embodiment, the vehicle position information 90D is given to the images 90P to 90R and the information 90A to 90C for generating these. In another embodiment, attribute information representing time (an image captured at a certain point in time, or an image generated based on information obtained at a certain point in time) is associated with the images 90P to 90R and the information 90A to 90C You can also.

Also, as mentioned above, a three-dimensional coordinate system can typically be represented by coordinates (x, y, z), and a spherical coordinate system can typically be represented by coordinates (r, θ, ϕ). Therefore, in another embodiment, the driver image 90Q and the vehicle image 90R may be processed using a spherical coordinate system similarly to the vehicle surrounding image 90P by using a well-known coordinate conversion. Alternatively, the vehicle surrounding image 90P may be processed by the three-dimensional coordinate system in the same manner as the driver image 90Q and the vehicle image 90R.

In this embodiment, a single-eye camera is used as the imaging device 11A, but a compound-eye camera may be used instead. In this way, the photographed object can be photographed together with the distance information, so it can relatively easily use the three-dimensional coordinate system to process the vehicle surrounding image 90P.

According to the image display system SY, a user (for example, a driver) can use the composite image 90X to visually recognize the driving vehicle 1 and its driver from a desired virtual viewpoint, for example, or to make the first The three see the situation. Therefore, according to the present embodiment, the composite image 90X can be used for various purposes, and can also be used for practice of driving operations as an example.

(Example of management aspect of composite image) FIG. 7 is a schematic diagram showing a management aspect in which the composite image 90X can be obtained in the above-described manner. The composite image 90X can be individually managed by, for example, each identification code for distinguishing each vehicle 1 and its driver, etc., and each identification code is individually stored in the database DB. For example, a moving image showing the driving state of a certain vehicle 1a and a certain driver Ua is stored in the database DBaa as a plurality of composite images 90X. Moreover, moving images showing the driving state of a certain vehicle 1a and another driver Ub are stored in the database DBab as a plurality of composite images 90X. A moving image showing the driving state of another vehicle 1b and a certain driver Ua is stored in the database DBba as a plurality of composite images 90X. In addition, moving images showing the driving state of the other vehicle 1b and the other driver Ub are stored in the database DBbb as a plurality of composite images 90X. The same applies to other databases DBac, DBbc, etc.

Although the above-mentioned database DBaa and the like can be individually managed according to each identification code, the following is only represented by the database DB. Furthermore, the number of composite images 90X stored in each database DB typically depends on the number corresponding to the time and frame rate of moving images.

(Example of how the composite image can be used) As illustrated in FIG. 8 , the arithmetic unit 22 can synthesize any two corresponding to mutually different identification codes in the synthesized image 90X stored in the database DB for comparison. For example, the computing unit 22 may compare a composite image corresponding to a certain identification code (set as “synthetic image 91X” for distinction) and a composite image corresponding to another identification code (set as “synthetic image 92X” for distinction ) are superimposed to generate a composite image 93X for comparison.

In the case where there is a difference above the reference level between the composite image 91X and the composite image 92X for comparison, the composite image 93X may be displayed on the display unit 33 so that the portion is emphasized. The highlighted portion may be displayed in at least one of the composite image 91X and the composite image 92X in the composite image 93X for comparison. As a method of emphasis, well-known aspects such as coloring and highlighting of outlines can be adopted. According to such a display aspect, the user can easily visually recognize the difference between the composite images 91X and 92X.

The composite image 91X is a vehicle surrounding image (set as “vehicle surrounding image 91P” for distinction), driver image (set as “driver image 91Q” for distinction), and vehicle image (set as “driver image 91Q” for distinction "Vehicle image 91R") is superimposed. In addition, in order to distinguish the vehicle surrounding information, the driver posture information, and the vehicle status information used to generate the images 91P, 91Q, and 91R, respectively, they are referred to as "vehicle surrounding information 91A" and "driver posture information 91B," respectively. " and "Vehicle Status Information 91C". In addition, in order to distinguish the vehicle position information given to the composite image 91X, the images 91P to 91R constituting the composite image 91X, and the information 91A to 91C for generating the images 91P to 91R, it is set as “vehicle position”. Information 91D".

The composite image 92X is a vehicle surrounding image (set as "vehicle surrounding image 92P" for distinction), driver image (set as "driver image 92Q" for distinction), and vehicle image (set as "driver image 92Q" for distinction "Vehicle image 92R") is superimposed. Also, in order to distinguish the vehicle surrounding information, driver posture information, and vehicle state information used to generate the images 92P, 92Q, and 92R, respectively, they are referred to as "vehicle surrounding information 92A" and "driver posture information 92B", respectively. and "Vehicle Status Information 92C". In addition, in order to distinguish the vehicle position information given to the composite image 92X, the images 92P to 92R constituting the composite image 92X, and the information 92A to 92C for generating the images 92P to 92R, it is set as "vehicle position information" 92D".

When the vehicle position information 91D and 92D represent the same position, the images 91Q and 91R, and the images 92Q and 92R are images representing a case where the vehicle is traveling at substantially the same position. Therefore, the composite image 93X for comparison is generated by combining the composite images 91X and 92X when the vehicle position information 91D and 92D indicate the same position. According to the example of FIG. 7 , a certain vehicle 1a and its driver Ua, and a vehicle 1b and its driver Ub, which are traveling independently of the vehicle 1a, can be placed side by side in the composite image 93X for comparison (a state where they are driven side by side). displayed on a single screen.

Here, in the above-mentioned case, the background image of the composite image for comparison 93X is formed by the vehicle surrounding images 91P and 92P, but the formation of the background image does not require both the images 91P and 92P. Therefore, when the composite image 93X for comparison is generated, as represented as "91P/92P" in FIG. 8, one of the images 91P and 92P may be omitted, for example, the image 91P may be employed as the background image picture.

In order to form the above-mentioned background image, as another embodiment, in S1040 (see FIG. 3 ) of generating the composite image 90X, the vehicle surrounding image 90P may be omitted. In this case, the image 91P surrounding the vehicle is omitted when generating the composite image 91X, and the image 92P surrounding the vehicle is omitted when generating the composite image 92X. Furthermore, when the composite image 93X for comparison is generated, the following figure can be used. Like one of 91P and 92P as the background image.

In addition, when forming the background image of the composite image 93X for comparison, the computing unit 22 can switch from one of the vehicle surrounding images 91P and 92P to the other, and can arbitrarily select the background image.

As another embodiment, both of the vehicle surrounding images 91P and 92P may be omitted when generating the composite image 93X for comparison. In this case, other images different from the images 91P and 92P (eg, a monochrome image, an image representing the scenery of other regions, an image of a virtual reality, etc.) may also be used for the background image. That is, the background image used as the image around the vehicle may not be an image that actually shows the situation around the vehicle 1 . From this point of view, the computing unit 22 may generate two or more vehicle surrounding images including the vehicle surrounding images 91P and 92P, and when generating the composite image 93X for comparison, the background image may be selected from these two images. One of the above vehicle surrounding images is switched to the other. Thereby, any one of these two or more vehicle surrounding images can be used as a background image.

FIG. 9 is a schematic diagram showing a composite image 93X for comparison. According to the present embodiment, a certain driving state e1 and a driving state e2 independent of the driving state are displayed on a single screen side by side/overlaid on the display unit 33 of the terminal 3 . The aspect e1 represents, for example, the driving pattern (the driving pattern of the vehicle 1a and its driver Ua) according to the database DBaa. The aspect e2 represents, for example, the driving pattern according to the database DBbb (the driving pattern of the vehicle 1b that is traveling independently of the vehicle 1a and the driving pattern of the driver Ub). As the user of the terminal 3, the driver Ua can compare his own driving operation with the driving operation of the driver Ub by referring to the composite image 93X for comparison, for example, and can use it for practice of the driving operation.

To summarize the above, the image processing device 2 (mainly the computing unit 22 ) obtains the driver posture information 91B and 92B, the vehicle state information 91C and 92C, and the vehicle position information 91D and 92D. The image processing device 2 generates a composite image 91X by superimposing the driver image 91Q based on the driver posture information 91B and the vehicle image 91R based on the vehicle state information 91C for a certain vehicle and its driver. In addition, the image processing device 2 superimposes the driver image 92Q based on the driver posture information 92B and the vehicle image 92R based on the vehicle state information 92C with respect to the vehicle that is traveling independently of the above-mentioned vehicle and its driver to generate a composite Image 92X. Then, the image processing device 2 associates the composite image 91X with the composite image 92X based on the vehicle position information 91D on the above-mentioned certain vehicle and the vehicle position information 92D on the vehicle running independently of the above-mentioned certain vehicle.

According to such image processing, the two driving states can be compared on a single screen (typically, a moving image) based on the composite images 91X and 92X. According to the embodiment, the composite images 91X and 92X are transmitted to the terminal device 3, and the above-mentioned two driving states are displayed on the terminal device 3 as a composite image 93X for comparison on a single screen. In this way, the user can refer to, for example, the composite image 93X for comparison in which the composite image 91X representing the driving state of the user and the composite image 92X representing another driving state are displayed by being superimposed, and the image 93X can be used. in driving practice.

(1st application example) In the above-mentioned embodiment, the composite image 93X for comparison has illustrated the driving state of a certain vehicle 1a and its driver Ua, and the driving state of the vehicle 1b and its driver Ub, which travel independently of the vehicle 1a. Contents displayed side by side (see FIG. 7 ).

However, the two driving patterns displayed side by side in the composite image for comparison 93X only need to be the vehicles 1 that are driving independently of each other, and they do not need to be different vehicles and/or drivers. That is, the vehicles 1 to be compared may be different vehicles that travel substantially at the same time, or the same vehicle that travels during mutually different periods.

Therefore, the user can not only compare his own driving style with that of others, but also compare his current driving style with his own past driving style. In addition, the user can also compare the current or past driving behavior when using a certain vehicle (eg, the vehicle 1a) with the current or past driving behavior when using another vehicle (eg, the vehicle 1b). Therefore, it is possible to manage the composite image 90X described while referring to FIG. 7 , and it is only necessary to carry out the management for the drivers independent of each other, and the identification codes required for the management can also be used in addition to the vehicle 1 and its driver. It is determined based on various attribute information such as the driving time period and the driving place.

(2nd application example) According to the above-described embodiment, when the vehicle position information 91D and 92D represent the same position, the composite images 91X and 92X represent the state of driving at substantially the same position, and these are combined to generate the composite image 93X for comparison. Therefore, the vehicle position information 91D and 92D may indicate the same position at least in the front-rear direction of the vehicle body (the forward direction of the vehicle 1 ).

On the other hand, the two driving patterns e1 and e2 (refer to FIG. 9 ) displayed side by side in the composite image for comparison 93X are preferably compared so that the user can compare them appropriately by referring to them. display them close to each other. Therefore, in the composite image 93X for comparison, the distance between the driving patterns e1 and e2 (the distance in the left-right direction of the vehicle body) may be designated and fixed by the user. In this case, in the moving image, the driving states e1 and e2 are displayed so that the distance between them is fixed.

In addition, the distance between the running patterns e1 and e2 can be arbitrarily set by the user, for example, in the range of 0 to 1 [m (meter)] or the like. When the distance between the driving patterns e1 and e2 is set to 0 [m (meters)], these are displayed in a superimposed manner. Thereby, the user who refers to the driving patterns e1 and e2 can compare the differences in detail.

In addition, the difference between the above-mentioned driving states e1 and e2 may be displayed on the display unit 33 of the terminal 3 so as to be visibly recognizable by characters, symbols, icons, etc. for notifying the user of the difference. In addition/alternatively, a guide (such as an audio guide) that uses sound to express the above-mentioned difference may also be indicated.

(3rd application example) The display of the composite image 93X for comparison may be partially omitted according to the user's request, that is, the driver images 91Q and 92Q and the vehicle images in the composite images 91X and 92X of the composite image 93X for comparison are formed. A part of 91R and 92R may be omitted.

For example, when the driver images 91Q and 92Q are omitted and the vehicle images 91R and 92R are displayed, the user can compare the inclination of the vehicle body of the vehicle 1 when cornering. Alternatively, when the driver images 91Q and 92Q are displayed and the vehicle images 91R and 92R are omitted, the user can compare the driver's posture when the vehicle 1 turns a corner.

It is generally considered that the posture of the vehicle body and/or the driver changes according to the vehicle speed of the vehicle 1 and the curvature (radius of revolution) of the bend. Therefore, the composite image 92X associated with the composite image 91X can also be used as a restriction according to predetermined conditions, for example, according to the speed of the vehicle and the driving location.

(4th application example) In the above-described embodiment, the user has exemplified the aspect in which the composite image 93X for comparison is used for the practice of driving operation, but the application is not limited to the aspect illustrated here. For example, the content of the embodiment can also be applied to generate a CG (computer graphics) representing a state in which a plurality of vehicles 1 are traveling substantially simultaneously to create a virtual locomotive racing moving image.

In this case, the vehicle position information 91D and 92D can also allow a predetermined amount of positional shift in the front-rear direction of the vehicle body. The allowable range may be, for example, 0 to 5 [m (meter)], etc., and can be arbitrarily set by the user.

In such a use, the number of composite images 90X required to generate the composite image 93X for comparison may be three or more, and the composite image 93X for comparison may be changed to a composite image for audiovisual use and a composite image for advertising. Diverse representations of images, or just composite images.

In addition, in such a use, the driver images 91Q and 92Q may not reflect the driver's posture, and the driver's posture information 91B and 92B may be expressed only by the driver's information.

Several examples shown above may be combined with other application examples individually, and any combination of these may be incorporated into the content of the present embodiment.

(Variation of image display system) According to the aforementioned image display system SY (refer to FIG. 1 ), although the function of the image processing device 2 can be realized in a place (for example, a server) different from the vehicle 1, the display of the composite image 90X and the display of the virtual viewpoint The modification may be implemented in the terminal device 3, but the present invention is not limited to this aspect.

FIG. 6A shows an example of the configuration of the image display system SYa. In the present system SYa, the image processing apparatus 2 is mounted on the vehicle 1 . In this case, the transmission of the composite image 90X from the vehicle 1 to the terminal 3 may be performed via the network N, or may be performed by a well-known communication means (eg, blue tооth (Bluetooth) (registered trademark)). .

FIG. 6B shows a configuration example of the image display system SYb. In the present system SYb, the image processing apparatus 2 is provided in the terminal 3 . That is, the terminal 3 only needs to receive the information 90A to 90D from the vehicle 1 via the network N or the like, and based on the information 90A to 90D, the composite image 90X may be generated and displayed on the display unit 33 .

In addition, as another example, the terminal 3 may also be a vehicle screen display (eg, a car navigation system). In this case, the driver can visually recognize the surrounding situation from a desired virtual viewpoint while driving the vehicle 1 .

(other) In the above description, for ease of understanding, each element is represented by a name related to its function. However, each element is not limited to what has been described in the embodiment as a main function, and it is also These may be provided auxiliary.

(Summary of the implementation form) The first aspect relates to an image processing apparatus (eg, 2), and the image processing apparatus is characterized in that it includes information acquisition means (eg, S1010, S1020, S1050) for acquiring a driver representing a driver of a vehicle information (eg 90B, 91B, 92B), and vehicle status information (eg 90C, 91C, 92C) indicating the status of the vehicle; the first image generation means (eg The first driver image (eg 91Q) based on the above driver information and the first vehicle image (eg 91R) based on the vehicle status information are superimposed to generate a first composite image (eg 91X); the second image Generating means (eg S1040 ), which, for a vehicle and its driver that runs independently of the above-mentioned certain vehicle, combines the second driver image (eg, 92Q) according to the above-mentioned driver information with the second vehicle according to the above-mentioned vehicle state information The images (eg, 92R) are superimposed to generate a second composite image (eg, 92X); and an associating means (eg, S1050 ) associates the first composite image with the second composite image according to predetermined information. Thereby, it can be set as a state in which the driving state of a certain first vehicle can be viewed on a single screen (typically a moving image) based on the first to second composite images independently of the first The driving state of the second vehicle that is running while the vehicle is being compared is compared. In addition, the first to second vehicles only need to travel independently of each other, and they may be the same vehicle that travels during mutually different periods, or may be mutually different vehicles that travel substantially simultaneously.

In a second aspect, the vehicle state information includes information indicating vehicle speed, steering angle, posture of the vehicle body, and/or state of the lamp body. Thereby, the first to second vehicle images can represent the vehicle speed, the steering angle, the posture of the vehicle body, and/or the state of the lamp body.

In the third aspect, the information obtaining means further obtains vehicle position information (eg, 90D, 91D, 92D) indicating a position on the travel path of the vehicle, and the predetermined information is the above-mentioned information about the vehicle. Vehicle location information and the above vehicle location information about vehicles that travel independently of one of the above vehicles. Thereby, the first to second composite images can be appropriately associated with each other.

In a fourth aspect, the information acquisition means acquires the vehicle position information based on GPS (Global Positioning System). Thereby, a vehicle and its driver can be displayed on a single screen side by side with a vehicle and its driver running independently of the vehicle (so that it is in a state of side-by-side driving).

In a fifth aspect, the information obtaining means further obtains vehicle surrounding information (for example, 90A, 91A, 92A) indicating the surrounding situation of the vehicle, and the image processing device is further provided with a method for obtaining information based on the vehicle surrounding information. A third image generation means (eg, S1000 ) that generates images of the surrounding of the vehicle (eg, 90P, 91P, 92P), and the predetermined information is the image of the surrounding of the vehicle. In this way, the surrounding image of the vehicle can be used as the background image to display a vehicle and its driver, and a vehicle and its driver that drive independently of the vehicle.

In a sixth aspect, the third image generating means can generate two or more images of the surrounding of the vehicle, and the two or more images of the surrounding of the vehicle include the image of the surrounding of the vehicle based on the image of the surrounding of the vehicle. The above-mentioned vehicle surrounding image of the information, and the above-mentioned vehicle surrounding image based on the above-mentioned vehicle surrounding information about the vehicle traveling independently of the above-mentioned vehicle, the above-mentioned association means may associate the above-mentioned vehicle surrounding image from the above. One of the two or more vehicle surrounding images is switched to the other. Thereby, any one of the two or more vehicle surrounding images can be used as the background image.

In a seventh aspect, the third image generating means processes the image around the vehicle using a spherical coordinate system (see FIGS. 4A and 4D ). Thereby, the processing of the image of the surrounding of the vehicle can be realized relatively simply.

In an eighth aspect, the first image generating means processes the first vehicle image using a three-dimensional coordinate system, and the second image generating means processes the second vehicle image using a three-dimensional coordinate system. (Refer to FIGS. 4B, 4C, and 4D). Thereby, the processing of the first to second vehicle images can be realized relatively simply.

In a ninth aspect, the driver information is driver posture information (eg, 90B, 91B, 92B) indicating the driver posture of the vehicle. Thereby, the user can compare the posture of the driver when cornering, for example, based on the first to second composite images.

A tenth aspect relates to an image display device (eg, 3), and the image display device is characterized by comprising: the above-mentioned image processing device (eg, 2); and a display (eg, 33), which uses the above-mentioned first 1. The composite image and the above-described second composite image are displayed on a single screen in a side-by-side or overlapping manner with the vehicle and its driver, and the vehicle and driver running independently of the vehicle. Thereby, the driving state of a certain first vehicle and the driving state of the second vehicle that is traveling independently of the first vehicle can be referred to while comparing.

In the eleventh aspect, the image processing device is characterized in that, with respect to at least one of the first composite image and the second composite image, a portion having a difference greater than or equal to a reference value between them is added. The emphasis is placed on the display above. Thereby, the user can easily visually recognize the difference between the first composite image and the second composite image.

A twelfth aspect relates to an image display device (eg, 3) that can communicate with the image processing device (eg, 2); characterized in that the image display device comprises a and the above-mentioned second composite image are displayed side by side or superimposed on a display (eg, 33 ) of a single screen. This makes it possible to refer to the running state of a certain first vehicle and the running state of the second vehicle that runs independently of the first vehicle while comparing.

In a thirteenth aspect, it is characterized in that it further includes an operation unit (eg, 32 ) that can accept an operation input for changing the display content of the display to a viewpoint from an arbitrary position. Thereby, the user can visually recognize the driving states of the above-mentioned first to second vehicles from an arbitrary viewpoint. In addition, in the embodiment, although the display and the operation part can be realized by the touch panel type display which is formed integrally or the like, in another embodiment, these can also be formed separately.

The present invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the present invention. Accordingly, in order to disclose the scope of the present invention, the appended claims are appended.

1: Vehicle 2: Image processing device 3: Terminal 8a, 8b, 8c, 8d: Icons 11A, 11B: Cameras 12: Vehicle state detection device 13: Vehicle location specific device 14: Communication device 21,31: Department of Communications 22: Operation Department 32: Operation Department 33: Display part 90A, 91A, 92A: Vehicle surrounding information 90B, 91B, 92B: Driver posture information 90C, 91C, 92C: Vehicle status information 90D, 91D, 92D: Vehicle location information 90P, 91P, 92P: Images around the vehicle 90Q, 91Q, 92Q: Driver images 90R, 91R, 92R: Vehicle images 90X, 91X, 92X: Composite image 93X: Composite image for comparison DB, DBaa, DBab, DBac, DBba, DBbb, DBbc: database e1, e2: driving state N: network SY, SYa, SYb: Image Display System

The accompanying drawings are included in the specification, constitute a part thereof, show embodiments of the present invention, and are used to explain the principle of the present invention together with the description thereof.

FIG. 1 is a schematic diagram showing a configuration example of an image display system. FIG. 2 is a schematic diagram showing a configuration example of a vehicle. FIG. 3 is a flowchart showing an example of an image processing method. FIG. 4A is a schematic diagram showing an image of the surroundings of the vehicle. FIG. 4B is a schematic diagram showing an image of a driver. FIG. 4C is a schematic diagram showing an image of a vehicle. FIG. 4D is a schematic diagram showing the composite image. FIG. 5A is a diagram showing an example of a composite image from a certain virtual viewpoint. FIG. 5B is a diagram showing an example of a composite image from another virtual viewpoint. FIG. 6A is a schematic diagram showing another configuration example of the image display system. FIG. 6B is a schematic diagram showing another configuration example of the image display system. FIG. 7 is a diagram showing an example of a management aspect of a composite image. FIG. 8 is a diagram showing an example of a method of generating a composite image for comparison. FIG. 9 is a schematic diagram showing an example of a composite image for comparison.

8a, 8b, 8c, 8d: Icons

93X: Composite image for comparison

e1, e2: driving state

Claims (13)

An image processing device, characterized in that it has: Information acquisition means, which acquires driver information representing the driver of the vehicle, and vehicle status information representing the state of the vehicle; A first image generating means for generating a first composite image by superimposing a first driver image based on the above-mentioned driver information and a first vehicle image based on the above-mentioned vehicle state information for a certain vehicle and its driver ; A second image generating means for superimposing a second driver image based on the driver information and a second vehicle image based on the vehicle state information on a vehicle that is traveling independently of the vehicle and its driver. generating a second composite image; and An associating means for associating the first composite image and the second composite image based on predetermined information. The image processing apparatus of claim 1, wherein, The above-mentioned vehicle status information includes information representing vehicle speed, steering angle, vehicle body posture, and/or lamp body status. The image processing apparatus of claim 1, wherein, The above-mentioned information obtaining means further obtains vehicle position information indicating a position on the travel path of the above-mentioned vehicle, and the above-mentioned predetermined information is the above-mentioned vehicle position information on the above-mentioned certain vehicle and the above-mentioned vehicle position information on a vehicle that travels independently of the above-mentioned certain vehicle. . The image processing apparatus of claim 3, wherein, The above-mentioned information acquisition means acquires the above-mentioned vehicle position information according to GPS (Global Positioning System). The image processing apparatus of claim 1, wherein, The above-mentioned information obtaining means further obtains vehicle surrounding information indicating the surrounding situation of the above-mentioned vehicle, The image processing device further includes a third image generating means for generating a vehicle surrounding image based on the vehicle surrounding information, and The above-mentioned predetermined information is the above-mentioned vehicle surrounding image. The image processing apparatus of claim 5, wherein, The third image generating means may generate two or more vehicle surrounding images, wherein the two or more vehicle surrounding images include the vehicle surrounding image based on the vehicle surrounding information about the vehicle, and the vehicle surrounding image based on the vehicle surrounding information about the vehicle. the above-mentioned vehicle surrounding image of the above-mentioned vehicle surrounding information of the vehicle driving on the above-mentioned certain vehicle, The associating means switches the associated vehicle surrounding image from one of the two or more vehicle surrounding images to the other. The image processing apparatus of claim 5, wherein, The above-mentioned third image generating means processes the above-mentioned vehicle surrounding image using a spherical coordinate system. The image processing apparatus of claim 7, wherein, The first image generating means processes the first vehicle image using a three-dimensional coordinate system, The second image generating means processes the second vehicle image using a three-dimensional coordinate system. The image processing apparatus of claim 1, wherein, The above-mentioned driver information is driver posture information indicating the posture of the driver of the vehicle. An image display device, characterized in that it has: The image processing device described in claim 1; and A display that uses the above-mentioned first composite image and the above-mentioned second composite image to display the above-mentioned certain vehicle and its driver, and a vehicle that runs independently of the above-mentioned certain vehicle and its driver on the side-by-side or superimposed A single picture. The image display device of claim 10, wherein, The said image processing apparatus displays on the said display at least one of the said 1st synthetic|combination image and the said 2nd synthetic|combination image so that the part which exists the difference of the reference value or more between them may be emphasized. An image display device capable of communicating with the image processing device described in claim 1, wherein the first composite image and the second composite image are arranged side by side or overlapped displayed on a single-screen display. The image display device according to any one of claims 10 to 12, further comprising an operation unit that accepts an operation input for changing the display content of the display to a viewpoint from an arbitrary position.

Images