WO2022190166A1

WO2022190166A1 - Image processing device, image processing method, and image processing program

Info

Publication number: WO2022190166A1
Application number: PCT/JP2021/008973
Authority: WO
Inventors: 竜也山田
Original assignee: 三菱電機株式会社
Priority date: 2021-03-08
Filing date: 2021-03-08
Publication date: 2022-09-15
Also published as: JPWO2022190166A1; JP7031799B1

Abstract

A road surface lighting ECU 110 projects a road surface lighting image 201, the luminance of which changes with time in a prescribed luminance change pattern, instead of projecting a road surface lighting image 201 with a constant luminance as in the past. A camera ECU 120 then identifies an area in a captured image, the luminance of which changes with time in the luminance change pattern, and adjusts image capture data so that the luminance value of the identified area decreases. A main SoC 130 then displays, on a display unit 135, a captured image corresponding to the image capture data adjusted for the luminance value. As a result, even if the road surface lighting image 201 is reflected in a captured image, the luminance value of the captured image as displayed on the display unit 135 is adjusted so as to make it difficult to visually recognize the road surface lighting image 201.

Description

Image processing device, image processing method, and image processing program

The present invention relates to an image processing device, an image processing method, and an image processing program.

Conventionally, there is a technology that supports the driver's driving by displaying an image of the outside of the vehicle taken by a camera on the display screen inside the vehicle. Further, for example, there is a technique of transmitting information to pedestrians present around the vehicle by projecting an image showing the traveling direction of the vehicle and a warning on the road surface around the vehicle (Patent Document 1). Hereinafter, an image projected on the road surface around the vehicle will be referred to as a "road surface image", and an image outside the vehicle captured by a camera will be referred to as a "captured image".

JP 2018-149876 A

By the way, when the road surface image is projected while the camera is shooting, the captured image including the road surface image will be displayed on the display screen. For example, when the road surface image shows a warning, the image showing the warning is displayed on the display screen inside the vehicle. In this case, it is difficult for the occupant to distinguish whether the information indicating the alert displayed on the display screen is information for the occupant or information for pedestrians outside the vehicle. Therefore, the occupant may feel annoyed by the fact that the captured image including the road surface image is displayed on the display screen.

SUMMARY OF THE INVENTION The present invention is intended to solve the above-described problems, and to reduce annoyance even when displaying on a display screen an image obtained by capturing a road surface image projected onto the road surface around the vehicle. aim.

A display processing device according to the present invention includes a projection control unit that projects an image whose brightness changes with time in a predetermined pattern onto a road surface around a vehicle, and an acquisition unit that acquires a photographed image of the surroundings of the vehicle. a storage unit for storing the photographed image acquired by the acquisition unit; specifying an area in which luminance changes with time according to the pattern from the photographed image stored in the storage unit; It is characterized by comprising a luminance adjusting section for adjusting a luminance value, and a display output section for outputting the photographed image adjusted by the luminance adjusting section to a display section.

The present invention has the effect of reducing the annoyance even when displaying on the display screen an image obtained by capturing a road surface image projected on the road surface around the vehicle.

1 is a diagram showing the configuration of an image processing apparatus 100 according to this embodiment; FIG. 2 is a conceptual diagram showing software operating on the image processing apparatus 100. FIG. 4 is a conceptual diagram of a memory 122 of the camera ECU 120; FIG. FIG. 2 is a conceptual diagram showing aspects of road surface lighting; This is an example of photographed data stored in the frame buffer 171 when the conventional road surface lighting image 201 is photographed by the camera 125 . FIG. 4 is a conceptual diagram showing an example of a luminance change pattern; 4 is a flowchart regarding processing of the image processing apparatus 100. FIG. 4 is a conceptual diagram showing a captured image stored in a frame buffer 171; FIG. 4 is a conceptual diagram showing a photographed image with adjusted luminance values stored in a frame buffer 171. FIG. 2 is a functional block diagram of the image processing apparatus 100; FIG. FIG. 5 is a diagram showing a luminance change pattern of road surface lighting associated with identification information; 4 is a flowchart regarding processing of the image processing apparatus 100. FIG. 2 is a conceptual diagram of memory 132 of main SoC 130. FIG. 4 is a flowchart regarding processing of the image processing apparatus 100. FIG. FIG. 4 is a diagram showing a situation in which a plurality of road surface lighting images are captured; It is an example of photographed data stored in a frame buffer when a plurality of road surface lighting images are photographed by the camera 125 . 4 is a flowchart regarding processing of the image processing apparatus 100. FIG. 4 is a conceptual diagram showing a photographed image with adjusted luminance values stored in a frame buffer 440. FIG.

It is assumed that the image processing device in the present disclosure is installed in an automobile. In the following description, an automobile equipped with an image processing device will be referred to as a "vehicle".

Embodiment 1.
FIG. 1 is a diagram showing the configuration of an image processing apparatus 100 according to this embodiment. The image processing apparatus 100 includes a road surface lighting ECU (Electric Control Unit) 110 , a camera ECU 120 , a main SoC (System-on-a-Chip) 130 and an in-vehicle communication bus 101 . The road surface lighting ECU 110, the camera ECU 120, and the main SoC 130 are connected by an in-vehicle communication bus 101, respectively. It should be noted that the present invention can also be implemented with other connection modes for the in-vehicle communication bus 101 .

The road surface lighting ECU 110 includes a CPU (Central Processing Unit) 111 (however, multiple CPU cores may be provided), a memory 112 , an interface 113 and a storage 114 . The interface 113 is connected to the projection section 115 . The projection unit 115 is, for example, a projector. The projection unit 115 projects an image showing information to be notified to people (pedestrians, occupants of other vehicles) around the vehicle, such as the traveling direction of the vehicle and a warning, onto the road surface around the vehicle. . Hereinafter, the image projected on the road surface around the vehicle will be referred to as a "road surface lighting image". Projecting the road surface lighting image onto the road surface is called "road surface lighting". Note that the road surface lighting image includes a still image and/or a moving image. The CPU 111 is an arithmetic unit that controls the road surface lighting ECU 110 . The memory 112 is, for example, a volatile storage device, and stores programs executed by the CPU 111, stacks, variables, and the like. The storage 114 is, for example, a non-volatile storage device such as a flash memory, and stores data such as programs and image data representing a road surface lighting image. In the present embodiment, image processing apparatus 100 includes projection unit 115 as a part of its configuration, but projection unit 115 may be configured as an external component.

The camera ECU 120 includes a CPU 121 , a memory 122 , an interface 123 and a storage 124 . Interface 123 is connected to camera 125 . The camera 125 captures an image of the surroundings of the vehicle to generate captured data. Therefore, when the road surface lighting image is projected during photographing, the camera 125 generates a photographed image including the road surface lighting image. In the present embodiment, image processing apparatus 100 includes camera 125 as part of its configuration, but camera 125 may be configured as an external component. Also, the interface 123 may be connected to multiple cameras 125 .

The main SoC 130 comprises a CPU 131, a memory 132, an interface 133, and a storage 134. The interface 133 is connected to the display section 135 . The display unit 135 is, for example, a liquid crystal screen provided in a car navigation system, a room mirror, or the like. The display unit 135 displays a captured image corresponding to the captured data. In the present embodiment, image processing apparatus 100 includes display unit 135 as part of its configuration, but display unit 135 may be configured as an external component.

FIG. 2 is a conceptual diagram showing software that operates on the image processing apparatus 100. FIG. The CPU 111 of the road surface lighting ECU 110 activates the operating system 141 . Also, the CPU 111 of the road surface lighting ECU 110 operates a communication library 142 , a road surface light driver 143 , and a road surface lighting application 140 on the operating system 141 . The road surface lighting application 140 drives the road surface light driver 143 to perform road surface lighting.

The CPU 121 of the camera ECU 120 activates the operating system 151 . Also, the CPU 121 of the camera ECU 120 operates a communication library 152 , a camera driver 153 and a camera application 150 on this operating system 151 . The camera application 150 drives the camera driver 153 to photograph the surroundings of the vehicle, and sequentially transmits photographed data generated by photographing to the main SoC 130 .

The CPU 131 of the main SoC 130 starts up the operating system 161. Also, the CPU 131 of the main SoC 130 operates a communication library 162 , a display driver 163 and a driving support application 160 on this operating system 161 . The driving assistance application 160 receives the photographed data transmitted from the camera ECU 120 . Further, the driving support application 160 drives the display driver 163 to display the captured image corresponding to the captured data on the display unit 135 .

The present invention can be implemented regardless of whether the respective operating systems and communication libraries are of the same type or different types.

FIG. 3 is a conceptual diagram of the memory 122 of the camera ECU 120. As shown in FIG.
Camera driver 153 uses part of memory 122 as frame buffer 171 . The frame buffer 171 is used for temporary storage until a photographed image (still image) photographed by the camera 125 is encoded as a video (moving image). The camera application 150 sequentially stores the photographed data in units of one frame in the frame buffer 171 . Further, when the camera application 150 stores the photographed data up to the end of the frame buffer 171, it returns to the beginning of the frame buffer 171 and resumes storing the photographed data. Furthermore, the camera application 150 encodes the shooting data stored in the frame buffer 171 as video in units of one frame or multiple frames. Common techniques are used for this encoding method. In the present disclosure, the data before encoding and the data after encoding are referred to as "capture data" without distinguishing between them.

FIG. 4 is a conceptual diagram showing the road surface lighting image 201 projected from the projection unit 115 onto the road surface around the own vehicle 200. FIG. 4 shows the front of the own vehicle 200, and the upper part of FIG. 4 shows the rear of the own vehicle 200. As shown in FIG. A projection unit 115 and a camera 125 are provided behind the own vehicle 200 . A shooting range 202 indicates a range shot by the camera 125 . A non-illuminated area 203 indicates an area that is included in the imaging range 202 but is not illuminated with the road surface lighting image 201 . The brightness value obtained by measuring the brightness of the area illuminated with the road surface lighting image 201 is higher than the brightness value obtained by measuring the brightness of the non-illuminated area 203 . Therefore, the occupant may feel annoyed by the fact that the captured image including the road surface image is displayed on the display screen. Although the road surface lighting image 201 shows the shape when the own vehicle 200 makes a left turn while backing up, the projected image and the timing are not limited to this.

FIG. 5 is an example of photographed data stored in the frame buffer 171 when the conventional road surface lighting image 201 is photographed by the camera 125 . Frames 403A to 403E store captured data generated over time in units of one frame. A conventional road lighting ECU projects a road lighting image with a constant luminance. Therefore, frames 403A to 403E store captured images including road surface lighting images with a constant luminance value regardless of the display cycle. In the following description, if there is no need to distinguish between the frames 403A to 403E, the letters at the end will be omitted.

The road surface lighting ECU 110 according to the present embodiment does not project the road surface lighting image 201 with a constant luminance as in the conventional art, but changes the luminance over time in a predetermined pattern (hereinafter referred to as a luminance change pattern). A road surface lighting image 201 is projected. Then, the camera ECU 120 identifies from the photographed image an area in which the luminance changes over time according to the luminance change pattern, and adjusts the photographed data so that the luminance value of the identified area becomes low. Then, the main SoC 130 displays on the display unit 135 the captured image corresponding to the captured data whose luminance value has been adjusted. As a result, even if the road surface lighting image 201 is reflected in the photographed image, the display unit 135 displays the photographed image in which the brightness value is adjusted so that the road surface lighting image 201 becomes difficult to see.

In FIG. 6, the conventional luminance change pattern 410 is set to a constant luminance (high luminance) regardless of the period. On the other hand, in the brightness change pattern 411 of the present disclosure, the brightness value for projecting the road surface lighting image 201 is set for each section 412 . A section 412 indicates the minimum section that constitutes the luminance change pattern. The brightness change pattern 411 is indicated, for example, as high brightness, high brightness, low brightness, high brightness, and low brightness. The road surface lighting ECU 110 of the present disclosure repeats this luminance change pattern 411 as one period only while the road surface lighting image 201 is being projected. By the way, in general, the camera 125 has a predetermined number of frames (frame rate) that can be captured in one second. For example, when using the camera 125 of 30 fps (frames/second), it is desirable that the section 412 is set to approximately 33 milliseconds or longer.

FIG. 7 is a flow chart regarding processing of the image processing apparatus 100 .
At the start, the road lighting EUC 110 is in the state before road lighting begins. Also, the camera ECU 120 is in a state before starting photographing. Further, the main SoC 130 is in a state in which it is possible to display an image corresponding to the photographing data transmitted from the camera ECU 120 on the display section 135 .

When the road surface lighting ECU 110 receives a trigger signal that triggers the start of road surface lighting, the road surface lighting ECU 110 notifies the camera ECU 120 of the luminance change pattern 411 of the road surface lighting (step 301). This communication is performed using the in-vehicle communication bus 101 . Communication is assumed to be in-vehicle Ethernet, but other methods may be used. The road surface lighting ECU 110 starts road surface lighting (step 302), and projects the road surface lighting image 201 whose luminance is changed according to the content of the luminance change pattern 411 notified in step 301 (step 303).

The camera ECU 120 receives the brightness change pattern 411 from the road surface lighting ECU 110 (step 304). The camera ECU 120 secures in the memory 122 a frame buffer 171 area capable of storing photographed data for one cycle or more of the brightness change pattern 411 in order to specify a region where the brightness changes over time in the brightness change pattern 411 . The camera ECU 120 starts photographing with the camera 125 (step 305), and stores photographed data generated by photographing in the frame buffer 171 (step 306).

FIG. 8 is a conceptual diagram showing captured images stored in the frame buffer 171. As shown in FIG.
The frame buffer 171 stores photographed data obtained by continuously photographing road surface lighting images whose luminance changes with time according to the luminance change pattern 411 . In FIG. 8, frames 403A, 403B, and 403D store captured images of high-brightness road

surface lighting images

201A, 201B, and 201D.

Frames

403C and 403E store captured images obtained by capturing low-luminance road

surface lighting images

201C and 201E.

Now, return to the description of FIG. The camera ECU 120 measures the change in the brightness value of the captured image stored in each frame 403, and determines the area where the brightness changes over time according to the brightness change pattern 411 notified in step 301, that is, the road surface lighting images 201A to 201E. (step 307). If the corresponding area could not be specified in the process of step 307 , the camera ECU 120 proceeds to the process of step 309 . On the other hand, when the relevant area is specified in the process of step 307, the camera ECU 120 adjusts the brightness value of the captured image in the relevant area. For example, the camera ECU 120 calculates the difference between the luminance values indicated by the regions identified in step 307 from the frame 403A corresponding to the high luminance luminance change pattern 411 and the frame 403C corresponding to the low luminance luminance change pattern 411. Then, the camera ECU 120 adjusts the brightness values of the captured images stored in the

frames

403A, 403B, and 403D corresponding to the high-brightness brightness change pattern 411 so as to be lower by the corresponding difference (step 308).

Note that the manner of adjusting the luminance value is not limited to this, as long as the camera ECU 120 adjusts such that the higher the luminance of the image projected by the road surface lighting ECU 110, the greater the extent to which the luminance value of the area is reduced. , any technique may be used. For example, the camera ECU 120 calculates the difference between the brightness value of the area where the high-brightness road surface lighting image 201A of the frame 403A corresponding to the high-brightness brightness change pattern 411 is captured and the brightness value of the other area. Then, the camera ECU 120 may adjust the luminance value of the area where the high-luminance road surface lighting image 201A is captured so as to be lower by the corresponding difference. Further, for example, the camera ECU 120 may adjust the luminance value of the area where the high-luminance road surface lighting image 201A of the frame 403A corresponding to the high-luminance luminance change pattern 411 is captured to be lower by a predetermined value. .

FIG. 9 is a conceptual diagram showing a photographed image with adjusted luminance values stored in the frame buffer 171. FIG. In the frame buffer 171, the brightness values of the road

surface lighting images

201A, 201B, and 201D illuminated with high brightness are adjusted to be approximately the same as the brightness values of the road

surface lighting images

201C and 201E illuminated with low brightness. Captured data is stored.

Now, return to the description of FIG. The camera ECU 120 checks for free space in the frame buffer 171 (step 309). Step 309 is implemented to temporarily retain the number of frames required to detect the luminance change pattern in step 307 in the frame buffer 171 . If there is space in the frame buffer 171 (step 309 returns No), the camera ECU 120 proceeds to step 306 to store the shooting data generated by shooting in the frame buffer 171 . On the other hand, if the frame buffer 171 has no free space (step 309; Yes), the camera ECU 120 transmits the oldest captured data among the captured data stored in the frame buffer 171 to the main SoC 130, and deletes the photographic data stored in , and makes it reusable (step 310). The main SoC 130 displays the captured image corresponding to the received captured data on the display unit 135 (step 311).

FIG. 10 is a functional block diagram of the image processing apparatus 100. As shown in FIG.
The image processing apparatus 100 includes a projection control section 11 , an acquisition section 12 , a storage section 13 , a luminance adjustment section 14 and a display output section 15 . When the projection control unit 11 receives a trigger signal that triggers the start of road surface lighting from an in-vehicle device (not shown), the projection control unit 11 projects an image (road surface lighting image) in which the luminance changes with time in a predetermined pattern on the road surface around the vehicle. Project. The projection control unit 11 is realized by the CPU 111 of the road surface lighting ECU 110 in FIG. The acquisition unit 12 acquires a photographed image of the surroundings of the vehicle. The acquisition unit 12 is implemented by the CPU 121 of the camera ECU 120 in FIG. The storage unit 13 stores the captured image acquired by the acquisition unit 12 . In this embodiment, the storage unit 13 is implemented by the memory 122 of the camera ECU 120 . The brightness adjustment unit 14 identifies an area in which the brightness changes with time according to a pattern from the captured image stored in the storage unit 13, and adjusts the brightness value of the captured image in that area. Further, the brightness adjustment unit 14 adjusts the width of decreasing the brightness value of the region to increase as the brightness of the image projected by the projection control unit 11 increases. In this embodiment, the brightness adjustment unit 14 is implemented by the CPU 121 of the camera ECU 120 . The display output unit 15 outputs the captured image adjusted by the brightness adjustment unit 14 to the display unit 135 . The display output unit 15 is implemented by the CPU 131 of the main Soc 130 .

With the above processing, even when the road surface lighting image 201 is photographed by the camera 125, the image processing apparatus 100 displays the photographed image on the display unit so that the luminance value of the area corresponding to the road surface lighting image 201 is adjusted to be low. 135 can be displayed. Therefore, even if a photographed image obtained by photographing the road surface lighting image 201 is displayed on the display unit 135, it is difficult for the occupant of the own vehicle 200 to distinguish between the area corresponding to the road surface lighting image 201 and other areas. Become. That is, the image processing device 100 can reduce the annoyance of the passenger even when the image obtained by capturing the road surface lighting image 201 is displayed on the display unit 135 . In addition, the image processing apparatus 100 can prevent the occupant from recognizing the surrounding situation. In addition, the image processing device 100 does not change the position or size at which the road surface lighting image 201 is projected. As a result, the image processing apparatus 100 can reduce the effect of calling attention obtained by projecting the road surface lighting image 201, compared to the case where the position and size of the projection of the road surface lighting image 201 are changed. None. Also, in general, in order to remove a specific object from a photographed image taken by a camera, it is necessary to recognize and extract the object from the photographed image. Recently, machine learning has been used for object recognition for arbitrary shapes. However, even when a trained model is used for object recognition, processing with a relatively high computational cost such as matrix multiplication and sum operations is performed. High difficulty. Furthermore, near-real-time object recognition processing is required in order to perform object recognition in a short period of time that allows the driver to recognize, judge, and operate, and to remove the object from the captured image displayed on the display unit. On the other hand, the image processing apparatus 100 projects the road surface lighting image 201 with a luminance change pattern in which the luminance changes over time, and shoots images in which the luminance value of the area where the luminance changes over time is adjusted according to the luminance change pattern. It just displays the image. Therefore, the image processing apparatus 100 can realize real-time object recognition and removal from images to the extent that the passenger can recognize, judge, and operate the object without using a high-cost, high-performance SoC.

Embodiment 2.
In the first embodiment, the road surface lighting ECU 110 notifies the camera ECU 120 of the road surface lighting pattern, so that the camera ECU 120 identifies from the captured image an area in which luminance changes over time according to the luminance change pattern, and detects the identified area. The shooting data was adjusted so that the brightness value of However, depending on the complexity of the brightness change pattern, the camera ECU 120 may not be able to secure the capacity of the frame buffer. Therefore, in the second embodiment, a mode is described in which the camera ECU 120 notifies the road surface lighting ECU 110 of the identification information indicating the luminance change pattern of the road surface lighting. The hardware configuration (FIG. 1), software configuration (FIG. 2), and functional configuration (FIG. 10) are the same as those of the first embodiment.

FIG. 11 is a diagram showing a luminance change pattern of road lighting associated with identification information. The storage 114 of the road surface lighting ECU 110 and the storage 124 of the camera ECU 120 store luminance change pattern information 420 indicating the luminance change pattern of the road surface lighting associated with the identification information. In the second embodiment, unlike the first embodiment, the brightness change pattern is specified not by the road surface lighting ECU 110 but by the camera ECU 120 . Therefore, the brightness change pattern information 420 is information that has been previously confirmed to be practicable in terms of the performance of the road surface lighting ECU 110 . The brightness change pattern information 420 may be shared by the road surface lighting ECU 110 and the camera ECU 120, and may be stored in a storage other than the

storages

114 and 124.

FIG. 12 is a flowchart regarding processing of the image processing apparatus 100 .
The camera ECU 120 selects the previously shared identification information of the luminance change pattern of the road surface lighting, and transmits it to the road surface lighting ECU 110 (step 312).

The road surface lighting ECU 110 receives identification information indicating the luminance change pattern of the road surface lighting from the camera ECU 120 . The road surface lighting ECU 110 starts road surface lighting with the luminance change pattern associated with the identification information (step 302). That is, the road surface lighting ECU 110 illuminates a road surface lighting image whose luminance is changed according to the luminance change pattern associated with the identification information (step 303).

The camera ECU 120 secures a frame buffer area capable of storing one cycle of the photographed data of the luminance change pattern associated with the identification information (step 304). Then, the camera ECU 120 starts photographing with the camera 125 (step 305). Since the subsequent operation is the same as that of the first embodiment, the explanation is omitted.

As described above, the camera ECU 120 selects the previously shared identification information of the luminance change pattern of the road surface lighting and transmits it to the road surface lighting ECU 110, thereby suppressing the possibility that the capacity of the frame buffer cannot be secured. In other words, the image processing apparatus 100 can display on the display unit 135 the captured image adjusted so that the luminance value of the area corresponding to the road surface lighting image 201 is low, even without the high-performance camera ECU 120 . .

Embodiment 3.
In the first embodiment, camera ECU 120 defines a region of frame buffer 171 capable of storing image data for one cycle or more of brightness change pattern 411 in order to specify a region where brightness changes over time in brightness change pattern 411. Secured in the memory 122 . On the other hand, in the present embodiment, in the main SoC 130, an area in which luminance changes over time is specified in the luminance change pattern, and the brightness value of the specified area is adjusted. This embodiment can be applied when the performance of the camera ECU 120 is low and the memory capacity required for the frame buffer cannot be secured. The hardware configuration (FIG. 1), software configuration (FIG. 2), and functional configuration (FIG. 10) are the same as those of the first embodiment. However, in this embodiment, the storage unit 13 is realized by the memory 132 of the main SoC 130 . Further, in the present embodiment, the brightness adjustment unit 14 is implemented by the CPU 131 of the main SoC 130 .

FIG. 13 is a conceptual diagram of the memory 132 of the main SoC 130. As shown in FIG.
Communications library 162 uses a portion of memory 132 as frame buffer 172 . The frame buffer 172 is used for temporary storage until the captured image (still image) transmitted from the camera ECU 120 is encoded as a video (moving image). The main SoC 130 sequentially stores the received shot data in units of one frame in the frame buffer 172 . Further, when the captured data is stored up to the end of the frame buffer 172, the main SoC 130 returns to the beginning of the frame buffer 172 and resumes storing captured data. The driving assistance application 160 encodes the photographed data stored in the frame buffer 172 as video in units of one frame or multiple frames. Common techniques are used for this encoding technique. In the present disclosure, the data before encoding and the data after encoding are referred to as "capture data" without distinguishing between them.

FIG. 14 is a flowchart regarding processing of the image processing apparatus 100 .
At the start, the road lighting EUC 110 is in the state before road lighting begins. Also, the camera ECU 120 is in a state before starting photographing. Also, the main SoC 130 is in a state capable of receiving photographed data transmitted from the camera ECU 120 .

The road surface lighting ECU 110 notifies the main SoC 130 of the luminance change pattern of the road surface lighting (step 301). This communication is performed using the in-vehicle communication bus 101 . Communication is assumed to be in-vehicle Ethernet, but other methods may be used. The road surface lighting ECU 110 starts road surface lighting (step 302), and projects a road surface lighting image whose luminance is changed according to the content of the luminance change pattern notified in step 301 (step 303).

The main SoC 130 receives the luminance change pattern from the road surface lighting ECU 110 (step 320). The main SoC 130 secures in the memory 132 a frame buffer 172 area capable of storing image data for one cycle or more of the brightness change pattern in order to specify a region where the brightness changes over time in the brightness change pattern. Further, after the main SoC 130 secures the area of the frame buffer 172 in the memory 132, the main SoC 130 transmits to the camera ECU 120 a receivable notification indicating that preparations for receiving photographed data have been completed (step 321).

The camera ECU 120 starts shooting with the camera 125 (step 305). Further, after receiving the receivable notification from the main SoC 130, the camera ECU 120 sequentially transmits the photographed data generated by photographing to the main SoC 130 without buffering (step 323).

The main SoC 130 stores the received imaging data in the frame buffer 172 (step 324). The main SoC 130 measures changes in the luminance value of the captured image stored in the frame buffer 172, and identifies areas where the luminance changes over time according to the luminance change pattern notified in step 301 (step 325). If the relevant area could not be specified in step 325 , the main SoC 130 proceeds to process step 327 . On the other hand, when the relevant area is specified in step 325, the main SoC 130 adjusts the brightness value of the captured image in the relevant area (step 326). For example, the main SoC 130 replaces the brightness value of the high-brightness region of the oldest frame in the frame buffer with the brightness value of the low-brightness region that appeared immediately before. Note that the main SoC 130 may adjust the brightness value of the captured image using the same method as the camera ECU 120 .

Next, the main SoC 130 checks whether there is space in the frame buffer (step 327). Step 327 is implemented to temporarily store in frame buffer 172 the number of frames required to detect the luminance change pattern in step 325 . If there is space in the frame buffer 172 (step 327 ; No), the main SoC 130 transitions to step 324 to store the shooting data generated by shooting in the frame buffer 172 . On the other hand, if the frame buffer 172 has no free space (step 327: Yes), the main SoC 130 displays the captured image corresponding to the oldest captured data among the captured data stored in the frame buffer 172 on the display device. Delete the framebuffer area and make it available for reuse (328). Note that the main SoC 130 may cause the display unit 135 to display a captured image corresponding to the captured data according to the specifications of the display unit 135 .

As described above, the pixel processing device 100 replaces the buffering by the camera ECU 120 with the main SoC 130, so that the luminance value of the area corresponding to the road surface lighting image 201 is lowered even without the high-performance camera ECU 120. The captured image adjusted as above can be displayed on the display unit 135 .

Embodiment 4.
In Embodiment 1, only the case where the road surface lighting is performed by the own vehicle is assumed. On the other hand, due to the spread of road lighting, it is conceivable that other vehicles close to the own vehicle 200 also have similar luminance change patterns.

FIG. 15 is a diagram showing a situation in which a plurality of road surface lighting images are captured.
When own vehicle 200 and other vehicle 210 project road surface lighting images, camera 125 of own vehicle 200 not only projects road surface lighting image 201 projected from own vehicle 200 but also road surface lighting image 211 projected from other vehicle 210. I will be taking pictures.

FIG. 16 is an example of photographed data stored in the frame buffer when a plurality of road surface lighting images are photographed by the camera 125. FIG. The frame buffer 440 stores photographed data obtained by continuously photographing the road surface lighting image 211 of the other vehicle 210 and the road surface lighting image 201 of the own vehicle 200 . Here, when each road surface lighting image is projected with the same brightness change pattern, the image processing apparatus 100 cannot distinguish between each road surface lighting image. For this reason, the image processing apparatus 100 has a problem that, for example, like the photographed images stored in the buffer 442, the luminance value of the area corresponding to each road surface lighting image is adjusted to be low.

As a countermeasure for this problem, even if not only the road surface lighting image 201 projected from the own vehicle 200 but also the road surface lighting image 211 projected from the other vehicle 210 is captured, only the road surface lighting image 201 of the own vehicle 200 is captured. will be replaced with low luminance.

The hardware configuration (Fig. 1), software configuration (Fig. 2), and functional configuration (Fig. 10) are the same as in the first embodiment.

FIG. 17 is a flowchart regarding processing of the image processing apparatus 100 .
The image processing device 100 determines that the road surface lighting of the other vehicle 210 has been photographed when the ratio of the area in which the luminance changes with time in the luminance change pattern 411 increases, and changes the luminance of the road surface lighting of the own vehicle 200. The difference is that the change pattern 411 is changed.

At the start, the road surface lighting EUC 110 is in the state before starting the road surface lighting. Also, the camera ECU 120 is in a state before starting photographing. Further, the main SoC 130 is in a state capable of displaying on the display unit 135 an image corresponding to the photographed data transmitted from the camera ECU 120 .

The road surface lighting ECU 110 notifies the camera ECU 120 of the brightness change pattern 411 (step 301). The road surface lighting ECU 110 starts road surface lighting (step 302), and projects the road surface lighting image 201 whose luminance is changed according to the content of the luminance change pattern 411 notified in step 301 (step 303).

The camera ECU 120 receives the brightness change pattern 411 from the road surface lighting ECU 110 . The camera ECU 120 secures in the memory 122 a frame buffer 440 capable of storing photographed data for one cycle or more of the brightness change pattern 411 in order to specify an area in which the brightness changes over time in the brightness change pattern 411 (step 304). The camera ECU 120 starts shooting with the camera 125 (step 305), and stores shooting data generated by shooting in the frame buffer 440 (step 306).

The camera ECU 120 measures changes in the brightness value of the captured image stored in the frame buffer 440, and identifies whether there is an area in which the brightness changes over time in the brightness change pattern 411 notified in step 301 (step 307).

Furthermore, the camera ECU 120 determines whether the ratio of the region corresponding to the luminance change pattern 411 to the captured image stored in the frame buffer 440 is greater than a preset threshold (step 331). This threshold value is set according to product specifications related to automobile safety standards, using parameters such as the size of the road surface lighting image, the height at which the projection unit 115 is provided, the projection angle, the projection range, and the permissible range of the degree of proximity between the vehicle and other vehicles. be done. Also, this threshold value may be set, for example, according to the past average ratio of the area corresponding to the luminance change pattern 411 in the captured image stored in the frame buffer 440 .

If it is determined to be greater than the threshold value (step 331: Yes), the camera ECU 120 captures the road surface lighting of the other vehicle 210 projected with the same brightness change pattern 411 as that of the own vehicle 200, in addition to the road surface lighting of the own vehicle 200. It is determined that The camera ECU 120 then notifies the road surface lighting ECU 110 of the change in the luminance change pattern (step 332). After transmitting all the photographed data stored in the frame buffer 440 to the main SoC 130, the camera ECU 120 deletes all the photographed data stored in the frame buffer 440 (step 333).

The road surface lighting ECU 110 changes the brightness change pattern of the own vehicle 200 in accordance with the request from the camera ECU 120 to change the brightness change pattern (step 330). Then, the road surface lighting ECU 110 notifies the camera ECU 120 of the luminance change pattern after the change (step 301), and restarts the road surface lighting (step 302).

If no luminance change pattern is detected in step 307 (step 307: No), or if it is determined that the value is below the threshold in step 331 (step 331: No), the camera ECU 120 performs steps 308 to 308 as in the first embodiment. 310 is executed. Then, the main SoC 130 displays the captured image corresponding to the received captured data on the display unit 135 (step 311).

FIG. 19 is a conceptual diagram showing a photographed image stored in the frame buffer 440 and having its luminance value adjusted.
As shown in the figure, the image processing device 100 changes the luminance change pattern of the own vehicle 200 so that even when the road surface lighting image projected from the own vehicle 200 and the other vehicle 210 is captured, the image of the own vehicle 200 is changed. Only the luminance value of the area corresponding to the road surface lighting image 201 can be adjusted. That is, the image processing device 100 can reduce the discomfort felt by the occupant by displaying on the display screen an image corresponding to the road surface lighting image projected from the own vehicle 200 .

Since Embodiment 4 is independent of Embodiments 2 and 3, it is possible to combine them.

In Embodiments 1, 2, 3, and 4, it was assumed that the camera ECU 120 and the main SoC 130 are different devices. is feasible. In this case, communication using the in-vehicle communication bus is replaced with communication using the bus within the SoC.

11 projection control unit, 12 acquisition unit, 13 storage unit, 14 brightness adjustment unit, 15 display output unit, 100 image processing device, 101 in-vehicle communication bus, 112 memory, 113 interface, 114 storage, 115 projection unit, 122 memory, 123 interface, 124 storage, 115 projection unit, 122 memory, 123 interface, 124 storage, 125 camera, 132 memory, 133 interface, 134 storage, 135 display unit, 140 road surface lighting application, 141 operating system, 142 communication library, 143 road surface light Driver, 150 camera application, 151 operating system, 152 communication library, 153 camera driver, 160 driving support application, 161 operating system, 162 communication library, 163 display driver, 171 frame buffer, 172 frame buffer, 200 own vehicle, 201 road lighting Image, 202 Projection range, 203 Non-illuminated area, 210 Other vehicle, 211 Road lighting image, 301 Brightness change pattern, 410 Brightness change pattern, 411 Brightness change pattern, 412 Section, 420 Brightness change pattern information, 440 Frame buffer, 442 Buffer .

Claims

a projection control unit that projects an image whose luminance changes over time in a predetermined pattern onto the road surface around the vehicle;
an acquisition unit that acquires a photographed image of the surroundings of the vehicle;
a storage unit that stores the captured image acquired by the acquisition unit;
a luminance adjustment unit that identifies an area in which luminance changes over time according to the pattern from the captured image stored in the storage unit, and adjusts the luminance value of the captured image in the area;
and a display output unit that outputs the captured image adjusted by the brightness adjustment unit to a display unit.
The brightness adjustment unit
2. The image processing apparatus according to claim 1, wherein the higher the brightness of the image projected by the projection control unit, the greater the amount of reduction in the brightness value of the area.
The brightness adjustment unit
3. The projection control unit is notified of a change in the pattern when a ratio of the region to the captured image stored in the storage unit is larger than a threshold value. The image processing device according to .
The projection control unit
notifying the luminance adjustment unit of the pattern for changing the luminance, and
4. The image processing apparatus according to any one of claims 1 to 3, wherein the area is specified based on the pattern notified from the projection control unit.
The brightness adjustment unit
notifying the projection control unit of the pattern for changing the luminance;
The projection control unit
4. The image processing apparatus according to any one of claims 1 to 3, wherein the brightness is changed based on the pattern notified from the brightness adjustment section.
a projection control step of projecting an image whose luminance changes over time in a predetermined pattern onto the road surface around the vehicle;
an acquisition step of acquiring a photographed image of the surroundings of the vehicle;
a storage step of storing the captured image acquired in the acquisition step;
a luminance adjustment step of identifying an area in which luminance changes with time according to the pattern from the photographed image stored in the storing step, and adjusting the luminance value of the photographed image in the area;
and a display output step of outputting the captured image adjusted in the brightness adjustment step to a display unit.
a projection control step of projecting an image whose luminance changes over time in a predetermined pattern onto the road surface around the vehicle;
an acquisition step of acquiring a photographed image of the surroundings of the vehicle;
a storage step of storing the captured image acquired in the acquisition step;
a luminance adjustment step of identifying an area in which luminance changes with time according to the pattern from the photographed image stored in the storing step, and adjusting the luminance value of the photographed image in the area;
and a display output step of outputting the photographed image adjusted in the brightness adjustment step to a display unit.