JP6906576B2 - Image processing equipment, image processing methods and in-vehicle equipment - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and an in-vehicle device.

Conventionally, there is a technique known for generating a virtual viewpoint image which is an image viewed from a virtual viewpoint based on a peripheral image of the vehicle captured by an imaging means installed at a predetermined position of the vehicle and displaying the virtual viewpoint image on the display means. (See, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2011-244239

However, in the prior art, only the virtual viewpoint image is displayed, and it is considered desirable to increase the convenience if the display using the virtual viewpoint image can be performed more appropriately.

The present invention has been made in view of the above, and an object of the present invention is to provide an image processing device, an image processing method, and an in-vehicle device capable of more appropriately displaying a virtual viewpoint image.

In order to solve the above problems and achieve the object, the present invention synthesizes an image generating unit that generates an image of a virtual viewpoint based on an image captured by the imaging unit and an image of the virtual viewpoint in an image processing apparatus. The image processing unit includes an image processing unit that generates an image displaying the image for use, and when the image of the virtual viewpoint is an image of the viewpoint position during movement of the virtual viewpoint, the image processing unit determines the transparency of the composite image. It is characterized in that the enhanced image of the virtual viewpoint is output to the display unit.

According to the present invention, it is possible to provide an image processing device, an image processing method, and an in-vehicle device capable of more appropriately displaying a virtual viewpoint image.

FIG. 1 is an explanatory diagram showing an image processing method according to an embodiment of the present invention. FIG. 2 is a diagram showing a configuration example of an in-vehicle device according to an embodiment of the present invention. FIG. 3 is a diagram showing an arrangement example of the imaging device. FIG. 4 is an explanatory diagram of the compositing process performed by the compositing unit when the compositing image is used as a guide line image. FIG. 5 is a diagram showing an example of each viewpoint position when the viewpoint position continuously moves from the viewpoint position which is the start point to the viewpoint position which is the end point. FIG. 6 is an explanatory diagram showing the processing contents by the image processing unit when the virtual viewpoint moves. FIG. 7 is a flowchart showing an image processing procedure executed by the image processing apparatus.

Hereinafter, embodiments of an image processing device, an image processing method, and an in-vehicle device according to the present invention will be described in detail with reference to the drawings.

[1. Image processing method]
FIG. 1 is an explanatory diagram showing an image processing method according to an embodiment of the present invention, and the image processing method is, for example, by a vehicle image processing device (hereinafter, referred to as an image processing device) mounted on a vehicle. Will be executed.

The image processing device according to the present embodiment is based on an image (hereinafter referred to as an captured image) captured by an imaging device (an example of an imaging unit) (not shown) that is installed at a predetermined position of the vehicle and images the periphery of the vehicle. , As shown in FIG. 1A, an image viewed from a virtual viewpoint (hereinafter referred to as a virtual viewpoint image) is generated.

Further, the image processing device generates a composite image in which the composite image is combined with the virtual viewpoint image, and outputs the composite image to the display device (an example of the display unit) to display the composite image on the virtual viewpoint image. The displayed image is displayed on the display device.

The composite image is an image that serves as an index for supporting driving, and is, for example, a guide line image, a warning image, an image showing a parking space, and the like. Such a composite image is generated by, for example, an image generation device (not shown) and output to an image processing device.

The guide line is, for example, an image of a guide line showing a traveling locus of the vehicle predicted based on a steering angle detected by a steering angle sensor (not shown) provided in the vehicle. The warning image is, for example, an image for notifying the position of an obstacle when an obstacle is present behind the vehicle, and is, for example, an image of an enclosure line surrounding the obstacle. Since the positions of the guide line, the obstacle, and the parking space change according to the virtual viewpoint, the image generation device generates a composite image according to the virtual viewpoint.

When the image processing device synthesizes a composite image with a virtual viewpoint image in which the viewpoint position of the virtual viewpoint is moving, both the virtual viewpoint image and the composite image are displayed while the viewpoint position continuously changes. Become. Therefore, the occupant of the vehicle must visually recognize the changes in the virtual viewpoint image and the composite image, which may make it difficult to grasp the images, and may also make it difficult to grasp the sense of distance.

Therefore, when the virtual viewpoint image is an image of the viewpoint position during movement of the virtual viewpoint, the image processing device performs a process of reducing the visibility of the composite image. As a result, the occupant of the vehicle can easily grasp the image, the sense of distance, and the like.

For example, in the image processing device, as shown in FIG. 1B, when the virtual viewpoint image is an image of the viewpoint positions A and B before and after the movement of the virtual viewpoint, the composite image is displayed on the virtual viewpoint image. While the image is output to the display device, when the virtual viewpoint image is an image of the viewpoint position during movement of the virtual viewpoint, an image in which the composite image is not displayed on the virtual viewpoint image is output to the display device.

The process of reducing the visibility of the composite image is not limited to the process of outputting the virtual viewpoint image to the display device without synthesizing the composite image to make the visibility of the composite image zero. For example, as a process of reducing the visibility of the composite image, there is a process of increasing the transparency of the composite image and outputting the composite image to the virtual viewpoint image to the display device. In the following, an example of a process of outputting a virtual viewpoint image to a display device without synthesizing a composite image will be described in detail.

[2. In-vehicle device configuration]
FIG. 2 is a diagram showing a configuration example of an in-vehicle device according to an embodiment of the present invention. As shown in FIG. 2, the in-vehicle device 100 includes an image pickup device 1 (an example of an image pickup unit), an image control device 2, an image processing device 3, and a display device 4 (an example of a display unit). The image processing device 3 may include, for example, at least one of the image control device 2 and the display device 4.

The image pickup device 1 includes, for example, an image pickup element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and outputs a captured image which is a moving image captured by the image pickup device to the image processing device 3. As shown in FIG. 3, for example, the image pickup device 1 is arranged at the rear portion of the vehicle 5 and images the rear portion of the vehicle 5. FIG. 3 is a diagram showing an arrangement example of the image pickup apparatus 1. The arrangement of the image pickup device 1 and the number of the image pickup devices 1 are not limited to the example shown in FIG.

The image control device 2 includes a composition image generation unit 6, an image control unit 7, and an operation unit 8. The compositing image generation unit 6 generates and outputs a compositing image for performing chroma key compositing on the virtual viewpoint image. The composite image is an image that serves as an index for supporting driving, and is, for example, a guide line image or a warning image.

For example, when generating an image of a guide line, the composite image generation unit 6 acquires information on the steering angle detected by a steering angle sensor (not shown) provided in the vehicle, and obtains the steering angle information or the like. Based on this, the traveling locus of the vehicle is predicted. The compositing image generation unit 6 generates an image of a guide line showing the predicted traveling locus.

Further, for example, when generating a warning image, the composite image generation unit 6 analyzes the captured image of the imaging device 1 and detects an obstacle existing behind the vehicle. The composite image generation unit 6 generates a warning image for notifying the occupants of the vehicle of the presence of an obstacle. Such a warning image is, for example, an image for notifying the position of an obstacle, and is an image of a surrounding line surrounding the obstacle.

The image control unit 7 controls the image processing device 3 and the image generation unit 6 for compositing based on, for example, an input operation to the operation unit 8. When there is a request to change the virtual viewpoint due to an input operation to the operation unit 8, the image control unit 7 performs a change process of the virtual viewpoint.

For example, when the image control unit 7 requests a change from the viewpoint position of the current virtual viewpoint (hereinafter referred to as the viewpoint position A) to the change destination viewpoint position (hereinafter referred to as the viewpoint position B), the viewpoint The information of the position B is output to the image generation unit 6 for compositing and the image processing device 3.

The compositing image generation unit 6 generates a compositing image according to the viewpoint position B based on the information of the viewpoint position B output from the image control unit 7, and outputs the compositing image to the image processing device 3. Further, the image processing device 3 generates a virtual viewpoint image which is an image viewed from a continuous viewpoint position up to the viewpoint position B based on the information of the viewpoint position B output from the image control device 2.

The image processing device 3 acquires an captured image from the image pickup device 1 and also acquires a composite image from the image control device 2. The image processing device 3 generates a virtual viewpoint image (hereinafter, referred to as a virtual viewpoint image) based on the acquired captured image, chromakey-synthesizes the virtual viewpoint image and the composite image, and produces the generated composite image. Output to display device 4.

The display device 4 includes an LCD (Liquid Crystal Display) and displays a composite image or a virtual viewpoint image based on the data of the composite image or the virtual viewpoint image acquired from the image processing device 3.

[3. Configuration of image processing device 3]
As shown in FIG. 2, the image processing device 3 includes an image generation unit 11, an image processing unit 12, and an operation management unit 13. The operation management unit 13 may be provided in the image generation unit 11 and the image processing unit 12, respectively.

The image generation unit 11 performs coordinate conversion processing on the captured image output from the image pickup apparatus 1 to generate a virtual viewpoint image which is an image viewed from the virtual viewpoint, and outputs the virtual viewpoint image to the image processing unit 12. The image processing unit 12 acquires the compositing image output from the image control device 2, combines the compositing image with the virtual viewpoint image, and outputs the composite image to the display device 4. The operation management unit 13 manages the operations of the image generation unit 11 and the image processing unit 12.

The image processing device 3 is a device realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), for example. Further, the image processing device 3 includes a processor such as a CPU or MPU and a storage unit such as a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, and the processor executes a program stored in the storage unit. It may be configured to realize the function of each part described above.

The image generation unit 11 includes a captured image acquisition unit 21 and a virtual viewpoint image generation unit 22. The captured image acquisition unit 21 acquires the captured image output from the imaging device 1. The image pickup device 1 outputs captured image data (hereinafter referred to as captured image data) to the image processing device 3 by, for example, a video signal such as an RGB signal or a component signal.

When the captured image acquisition unit 21 acquires the captured image data from the imaging device 1, the captured image acquisition unit 21 outputs the acquired captured image data to the virtual viewpoint image generation unit 22. The captured image data is data corresponding to the image of each frame of the moving image captured by the imaging device 1, and is sequentially output from the captured image acquisition unit 21 to the virtual viewpoint image generation unit 22 in frame units of the captured moving image. ..

The virtual viewpoint image generation unit 22 performs coordinate conversion processing on the captured image to generate a virtual viewpoint image which is an image viewed from the virtual viewpoint, and the data of the virtual viewpoint image (hereinafter, referred to as virtual viewpoint image data). ) Is output to the synthesis unit 35.

In such a coordinate conversion process, the virtual viewpoint image generation unit 22 projects (maps) the captured image onto a predetermined projection surface, and among the captured images projected on the predetermined projection surface, a predetermined field of view when viewed from the virtual viewpoint. The image of the area included in the corner is used as the virtual viewpoint image.

The predetermined projection surface may be, for example, a three-dimensional curved surface in a virtual three-dimensional space, or the road surface shown in FIG. 1A. Further, the projection of the captured image on the predetermined projection surface can be performed, for example, by converting the coordinates of the captured image with the imaging device 1 as the origin coordinate to the coordinate system with the projection plane as the reference.

The image processing unit 12 includes a composition image acquisition unit 31, a compression unit 32, a storage unit 33, an expansion unit 34, a composition unit 35, an image quality adjustment processing unit 36, and an image output unit 37.

The compositing image acquisition unit 31 acquires a compositing image output from the image control device 2. The image control device 2 outputs data of a composite image (hereinafter, referred to as composite image data) to an image processing device 3 by, for example, a video signal such as an RGB signal or a component signal. When the compositing image acquisition unit 31 acquires the compositing image data from the image control device 2, the compositing image acquisition unit 31 outputs the acquired compositing image data to the compression unit 32.

The compression unit 32 compresses the composite image data by a predetermined compression process, and stores the compressed composite image data (hereinafter, referred to as compressed image data) in the storage unit 33. For example, when the image data for compositing uses a color specified in advance, the compression unit 32 can perform data compression by using a CLUT (Color Look-Up Table), and also compresses in this way. The resulting data can be further compressed by the run-length method. The CLUT and run-length methods are examples, and other compression treatments may be performed.

The stretching unit 34 performs a process of stretching the compressed image data stored in the storage unit 33 into the composite image data. For example, the decompression unit 34 performs a process of inversely converting the compressed image data stored in the storage unit 33 by the run-length method, and further performing an inverse conversion by the CLUT to decompress the compressed image data into the composite image data. conduct.

The compositing unit 35 is based on the virtual viewpoint image data acquired from the image generation unit 11 and the compositing image data acquired from the stretching unit 34, and the compositing image data obtained by chromakey-synthesizing the compositing image on the virtual viewpoint image (hereinafter referred to as (Described as composite image data) is generated. For example, the compositing unit 35 sets the color of a pixel having a predetermined color (for example, blue) among a plurality of pixels constituting the compositing image as a transparent color, and then synthesizes the compositing image on the virtual viewpoint image. do.

The compositing unit 35 turns on the compositing function for compositing the virtual viewpoint image and the compositing image, or turns off the compositing function, for example, in response to a request from the operation management unit 13. When the compositing function is ON, the compositing unit 35 synthesizes the compositing image with the virtual viewpoint image, and when the compositing function is OFF, the compositing unit 35 does not synthesize the virtual viewpoint image and the compositing image, and is an image generation unit. The virtual viewpoint image data acquired from 11 is output to the image quality adjustment processing unit 36.

FIG. 4 is an explanatory diagram of the compositing process performed by the compositing unit 35 when the compositing image is an image of a guide line. In the example shown in FIG. 4, the compositing unit 35 generates a composite image in which the image of the guide line is combined with the virtual viewpoint image. With such a composite image, the occupant of the vehicle can see the virtual viewpoint image on which the guide line is superimposed, and can drive more appropriately.

The image quality adjustment processing unit 36 performs processing for adjusting the image quality of the image output from the composition unit 35, and outputs the image data after the image quality adjustment processing to the image output unit 37. The image output unit 37 converts the image data from the image quality adjustment processing unit 36 into a video signal format that can be received by the display device 4 and outputs the image data to the display device 4.

_{The operation management unit 13 describes continuous viewpoint positions (hereinafter, C 1 to} C _n) up to the viewpoint position B after movement based on the information of the viewpoint position B of the movement destination output from the image control device 2. n is a natural number) is output to the image generation unit 11. _{As a result, the image generation unit 11 virtualizes the continuous viewpoint positions C 1 to} C _n (hereinafter, may be collectively referred to as the viewpoint position C) from the viewpoint position A before the movement to the viewpoint position B after the movement. Viewpoint images are sequentially generated and output.

FIG. 5 is an explanatory diagram of the _{viewpoint positions A, C 1 to} C _n , and B when the viewpoint position continuously moves from the viewpoint position A to the viewpoint position B, and C of _{the viewpoint positions C 1 to} C _n . _{Only n / 3} and C _{2n / 3} are shown for convenience. Further, in the example shown in FIG. 5, the height of the viewpoint position increases from the viewpoint position A toward the viewpoint position B.

The number of viewpoint positions C between the viewpoint positions A and B is such that the occupant of the vehicle can seamlessly visually recognize the virtual viewpoint image of the viewpoint position that continuously moves between the viewpoint positions A and B, for example. , The number may be such that it can be visually recognized frame by frame.

Returning to FIG. 2, the description of the operation management unit 13 will be continued. The operation management unit 13 outputs the image in which the composite image is displayed on the virtual viewpoint image from the image processing unit 12 to the display device 4, or displays the virtual viewpoint image in which the composite image is not displayed from the image processing unit 12. It is possible to control whether to output to 4.

When the virtual viewpoint images of the viewpoint positions A and B are output from the image generation unit 11, the operation management unit 13 causes the image processing unit 12 to generate and output an image obtained by synthesizing the composite image with the virtual viewpoint image. On the other hand, when the virtual viewpoint image at the viewpoint position C is output from the image generation unit 11, the operation management unit 13 outputs the virtual viewpoint image in which the composite image is not synthesized from the image processing unit 12 to the display device 4. Let me.

FIG. 6 is an explanatory diagram showing the processing contents by the image processing unit 12 when the virtual viewpoint moves from the viewpoint position A to the viewpoint position B, and is based on the vertical synchronization signal, the viewpoint position of the virtual viewpoint image, and the compositing unit 35. The correspondence of ON / OFF of the synthesis function is shown.

In FIG. 6, an example in which the information of the viewpoint position B of the movement destination is acquired from the image control device 2 before the time t2, and the operation management unit 13 changes the viewpoint position from the time t2 based on the information. Is shown. Further, the vertical synchronization signal shown in FIG. 6 is shown for convenience in order to indicate the frame unit division of the image.

As shown in FIG. 6, the operation management unit 13 causes the image generation unit 11 to generate a virtual viewpoint image of the viewpoint position A before movement until time t2, and the virtual viewpoint image of the viewpoint position A corresponds to the viewpoint position A. A composite image obtained by synthesizing the composite image is generated by the image processing unit 12 and output.

Then, the operation management unit 13, from the time t2 to t8, the virtual viewpoint image of viewpoint position _C 1 -C ₆ during the movement is generated to the image generating unit 11 does not synthesize the synthesized image viewpoint position _C 1 -C ₆ The virtual viewpoint image of is output from the image processing unit 12. Although the example shown in FIG. 6 shows an example in which the viewpoint changes for each frame, the viewpoint may change in units of a plurality of frames.

The image processing unit 12, under the control of the operation management unit 13, between the time t2 to t8, the expansion processing and synthesis processing by the synthesis section 35 by the decompression unit 34 is stopped, the virtual viewpoint position C ₁ -C ₆ The viewpoint image is output from the compositing unit 35 to the image quality adjusting processing unit 36 as it is. Thus, the virtual viewpoint image synthesis image and the non-synthetic viewpoint position C ₁ -C ₆ is outputted from the image processing unit 12.

After that, at time t8, the operation management unit 13 causes the image generation unit 11 to generate a virtual viewpoint image of the viewpoint position B after the movement is stopped, outputs the image to the image processing unit 12, and displays the viewpoint on the virtual viewpoint image of the viewpoint position B. The image processing unit 12 generates a composite image obtained by synthesizing the composite image according to the position B, and outputs the composite image to the display device 4.

The composite image corresponding to the viewpoint position B is generated by the image control device 2 and output to the image processing device 3 before the time t8 (for example, before the time t2 or before the time t3). The operation management unit 13 controls the image processing unit 12 and stores it in the storage unit 33 after compressing the data of the composite image according to the viewpoint position B output from the image control device 2.

Then, the operation management unit 13 expands the compressed image data stored in the storage unit 33 by the expansion unit 34 and outputs the compressed image data from the expansion unit 34 to the synthesis unit 35. At time t8, the viewpoint position is displayed on the virtual viewpoint image at the viewpoint position B. The compositing unit 35 synthesizes the compositing image corresponding to B. As a result, the image processing unit 12 can output a composite image obtained by synthesizing the composite image corresponding to the viewpoint position B with the virtual viewpoint image at the viewpoint position B.

The image processing unit 12 can also acquire a composite image corresponding to each viewpoint position in advance from the image control device 2 and store it in the storage unit 33. In this case, the operation management unit 13 can acquire the compositing image according to the viewpoint position of the virtual viewpoint image from the storage unit 33, stretch it to the stretching unit 34, and output it to the compositing unit 35.

As described above, when the virtual viewpoint image is an image of the viewpoint position before and after the movement of the virtual viewpoint, the image processing unit 12 displays the composite image on the display device 4, and the virtual viewpoint image is the viewpoint position during the movement of the virtual viewpoint. In the case of the image of, a virtual viewpoint image is displayed on the display device 4 instead of the composite image. As a result, the occupants of the vehicle do not have to visually recognize the virtual viewpoint image and the composite image when the virtual viewpoint is moving, so that the sense of distance and the like can be obtained based on the virtual viewpoint image displayed on the display device 4. It can be easily grasped.

Further, when the virtual viewpoint image is an image of the viewpoint position during movement of the virtual viewpoint, the image processing unit 12 stops the composition of the virtual viewpoint image and the compositing image, so that the processing load can be reduced. ..

Further, the image processing unit 12 synthesizes an image corresponding to the virtual viewpoint as a composite image with the image of the virtual viewpoint. Therefore, it is not necessary to visually recognize the virtual viewpoint image whose viewpoint position continuously changes and the composite image at the same time, so that it is easy to grasp the sense of distance from the virtual viewpoint image, for example.

In the above-mentioned example, the composite image is described as an image corresponding to the viewpoint position of the virtual viewpoint, but the composite image does not have to be continuously changed according to the viewpoint position of the virtual viewpoint. .. Even in such a case, the visibility is improved by not synthesizing the composite image with the virtual viewpoint image whose viewpoint position changes continuously, so that the display using the virtual viewpoint image is performed more appropriately. be able to.

[4. Image processing of image processing device 3]
Next, the image processing executed by the image processing apparatus 3 will be described with reference to FIG. 7. FIG. 7 is a flowchart showing an image processing procedure executed by the image processing device 3.

As shown in FIG. 7, the image generation unit 11 and the image processing unit 12 of the image processing device 3 perform image acquisition processing (step S1). In such processing, the image generation unit 11 acquires the captured image from the image pickup device 1, and the image processing unit 12 acquires and stores the composite image from the image control device 2.

Next, the image generation unit 11 converts the captured image acquired in step S1 into a virtual viewpoint image by coordinate conversion processing (step S2). Further, the operation management unit 13 determines whether or not the viewpoint position of the virtual viewpoint is moving (step S3).

When it is determined in step S3 that the viewpoint position of the virtual viewpoint is not moving (step S3; No), the image processing unit 12 acquires the compositing image stored in step S1 (step S4) and performs compositing processing. (Step S5). Such a compositing process is performed by synthesizing the compositing image acquired in step S4 with the virtual viewpoint image generated in step S2.

When it is determined in step S3 that the viewpoint position of the virtual viewpoint is moving (step S3; Yes), or when the processing of step S5 is completed, the image processing unit 12 outputs the image to the display device 4. The image quality is adjusted (step S6), the image after the image quality is adjusted is output (step S7), and the process is completed.

As described above, when the virtual viewpoint image is an image of the viewpoint position during movement of the virtual viewpoint, the image processing device 3 according to the embodiment outputs the virtual viewpoint image to the display device 4 instead of the composite image. The visibility of the composite image can be reduced, and the display using the virtual viewpoint image can be performed more appropriately.

In the above-described embodiment, the image processing device 3 hides the composite image to reduce the visibility of the composite image to zero, but increases the transparency of the composite image and reduces the color. By doing so, it is possible to reduce the visibility of the composite image.

For example, when the virtual viewpoint image is an image of the moving viewpoint position of the virtual viewpoint, the compositing unit 35 performs a process of increasing the transparency or lightening the color of the compositing image according to the moving viewpoint position. After that, the image for compositing can be combined with the virtual viewpoint image. Further, the image processing device 3 can also reduce the visibility of the composite image to zero by synthesizing the virtual viewpoint image on the composite image.

Further, in the above-described embodiment, the image pickup device 1 is arranged at the rear of the vehicle and the image pickup device 1 is used to image the rear of the vehicle. The imaging device 1 may be arranged and in any imaging direction. Further, the image processing device 1 can be arranged at a plurality of locations, and the image processing device 3 can generate a virtual viewpoint image based on the captured images output from the plurality of image pickup devices 1.

Further, in the above-described embodiment, the image processing device 3 constituting the in-vehicle device 100 mounted on the vehicle has been described as an example, but the image processing device 3 is, for example, an image processing device built in a smartphone or the like. May be good.

Further, in the above-described embodiment, the composite image has been described by taking an image as an index for supporting driving as an example, but for example, if it is a composite image for enhancing user convenience, etc. good.

Further, in the above-described embodiment, an example has been described in which a composite image is output at the viewpoint positions of the start point and the end point of the virtual viewpoint when the virtual viewpoint is moved, and the composite image is not output at the viewpoint position between the start point and the end point. However, it is not limited to such an example.

For example, the image processing unit 12 can output a composite image at one or more viewpoint positions between the start point and the end point of the virtual viewpoint. In this case, since the change in the composite image can be grasped in fragments, it is possible to easily grasp the position change of the composite image while easily grasping the sense of distance and the like based on the virtual viewpoint image.

Further, in the above-described embodiment, an example of synthesizing one composite image with a virtual viewpoint image has been described, but a plurality of different types of composite images can be combined with a virtual viewpoint image. In this case, when the virtual viewpoint image is an image of the viewpoint position during movement of the virtual viewpoint, the image processing unit 12 does not combine at least one of the plurality of composite images of different types with the virtual viewpoint image. The visibility is lowered by such means. Even in such a case, the display using the virtual viewpoint image can be performed more appropriately.

In this case, the image processing unit 12 can perform the compositing process with the virtual viewpoint image so that the compositing image is inconspicuous, for example, by lightening the color of the compositing image to be combined with the virtual viewpoint image.

Further, in the above embodiment, the image processing apparatus 3 has been described as generating information of the viewpoint position C ₁ -C _n when acquiring the information of the destination of the viewpoint position B, the viewpoint position C ₁ ~ C _n may also be output is generated by the image control device 2 to the image processing apparatus 3.

Further, in the above-described embodiment, the image processing device 3 has been described as storing the composite image after data compression, but the image processing device 3 stores the composite image without performing data compression. You can also.

Further effects and variations can be easily derived by those skilled in the art. For this reason, the broader aspects of the invention are not limited to the particular details and representative embodiments expressed and described as described above. Therefore, various modifications can be made without departing from the spirit or scope of the general concept of the invention as defined by the appended claims and their equivalents.

1 Imaging device (an example of imaging unit)
2 Image control device 3 Image processing device 4 Display device (example of display unit)
5 Vehicle 6 Image generation unit for composition 7 Image control unit 8 Operation unit 10 Image acquisition unit 11 Image generation unit 12 Image processing unit 13 Operation management unit 21 Captured image acquisition unit 22 Virtual viewpoint image generation unit 31 Image acquisition unit for composition 32 Compression unit 33 Storage unit 34 Decompression unit 35 Synthesis unit 36 Image quality adjustment processing unit 37 Image output unit

Claims (7)

An image generation unit that generates a virtual viewpoint image based on the image captured by the image pickup unit,
It is provided with an image processing unit that generates an image in which a composite image is displayed on the virtual viewpoint image.
The image processing unit
When the image of the virtual viewpoint is an image of the viewpoint position while the virtual viewpoint is moving, the composite image displayed when the image of the virtual viewpoint is an image of the viewpoint position where the virtual viewpoint is not moving. An image processing device characterized in that an image of the virtual viewpoint whose transparency is increased so that the visibility of the composite image is lowered as compared with the transparency is output to a display unit. An image generation unit that generates a virtual viewpoint image based on the image captured by the image pickup unit,
It is provided with an image processing unit that generates an image in which a composite image is displayed on the virtual viewpoint image.
The image processing unit
When the image of the virtual viewpoint is an image of the viewpoint position while the virtual viewpoint is moving, the composite image displayed when the image of the virtual viewpoint is an image of the viewpoint position where the virtual viewpoint is not moving. compared to the color image processing apparatus characterized by visibility of the synthesis image is output to the display unit an image of the virtual viewpoint with a reduced color to decrease. The imaging unit is arranged in the vehicle and
The image processing apparatus according to claim 1 or 2, wherein the image processing unit synthesizes an image of a guide line showing a predicted traveling locus of the vehicle as an image for compositing with an image of the virtual viewpoint. .. A generation process for generating a virtual viewpoint image, which is an image viewed from a virtual viewpoint, based on the captured image of the imaging unit, and
A processing step of generating an image in which a composite image is displayed on the virtual viewpoint image is provided.
The processing step is
When the image of the virtual viewpoint is an image of the viewpoint position while the virtual viewpoint is moving, the composite image displayed when the image of the virtual viewpoint is an image of the viewpoint position where the virtual viewpoint is not moving. An image processing method characterized in that an image of the virtual viewpoint whose transparency is increased so that the visibility of the composite image is lowered as compared with the transparency is output to a display unit. A generation process for generating a virtual viewpoint image, which is an image viewed from a virtual viewpoint, based on the captured image of the imaging unit, and
A processing step of generating an image in which a composite image is displayed on the virtual viewpoint image is provided.
The processing step is
When the image of the virtual viewpoint is an image of the viewpoint position while the virtual viewpoint is moving, the composite image displayed when the image of the virtual viewpoint is an image of the viewpoint position where the virtual viewpoint is not moving. compared to the color image processing method characterized in that visibility of the synthesis image is output to the display unit an image of the virtual viewpoint with a reduced color to decrease. A display unit that displays images and
An imaging unit that captures an image of the surroundings of the vehicle and outputs the captured image,
A virtual viewpoint image generation unit that generates a virtual viewpoint image based on the captured image of the imaging unit, and a virtual viewpoint image generation unit.
An image generator for compositing that generates an image for compositing,
An image processing unit that generates an image displaying a composite image on the virtual viewpoint image and outputs the image to the display unit is provided.
The image processing unit
When the image of the virtual viewpoint is an image of the viewpoint position while the virtual viewpoint is moving, the composite image displayed when the image of the virtual viewpoint is an image of the viewpoint position where the virtual viewpoint is not moving. An in-vehicle device characterized in that an image of the virtual viewpoint whose transparency is increased so that the visibility of the composite image is lowered as compared with the transparency is output to the display unit. A display unit that displays images and
An imaging unit that captures an image of the surroundings of the vehicle and outputs the captured image,
A virtual viewpoint image generation unit that generates a virtual viewpoint image based on the captured image of the imaging unit, and a virtual viewpoint image generation unit.
An image generator for compositing that generates an image for compositing,
An image processing unit that generates an image displaying a composite image on the virtual viewpoint image and outputs the image to the display unit is provided.
The image processing unit
When the image of the virtual viewpoint is an image of the viewpoint position while the virtual viewpoint is moving, the composite image displayed when the image of the virtual viewpoint is an image of the viewpoint position where the virtual viewpoint is not moving. compared to the color, vehicle and wherein the visibility of the composite image to output an image of the virtual viewpoint with a reduced color to decrease on the display unit.

Images