JP2022138682A - Image processing device, image processing system, image processing method and program - Google Patents

Abstract

To provide an image processing device that can provide an excellent rear field of view, even when a photographing part is arranged to be offset from a center in a width direction of a vehicle, an image processing method, an image processing method and a program.SOLUTION: An image forming device includes: an image obtaining part, arranged at a rear part of a vehicle to be offset with respect to a center in a width direction of the vehicle, which obtains images photographed by a photographing part which photographs a view behind; and a control part that corrects a position of the behind-view image which is displayed on a display part provided in the vehicle, of the images obtained by the image obtaining part, so that positional shift in the width direction of a following vehicle in the behind-view image is suppressed on the basis of an offset amount in the width direction of the photographing part and a distance between the vehicle and the following vehicle.SELECTED DRAWING: Figure 8

Description

The present invention relates to an image processing device, an image processing system, an image processing method, and a program.

Conventionally, it is installed at the rear of the vehicle, offset from the longitudinal center line, and a camera that captures the rear of the vehicle and the image captured by the camera are input. an image processing unit for generating a rearward image to be projected and displaying it on a display device, wherein the image processing unit performs image processing so that the vertical center of the rearward image substantially coincides with the longitudinal center line of the vehicle. There is a monitoring system that performs (for example, see Patent Document 1).

JP-A-2002-374523

By the way, when the imaging unit is arranged offset from the center in the width direction of the vehicle and an image including the following vehicle is displayed on the display unit inside the vehicle, the width of the image in the width direction varies depending on the distance between the vehicle and the following vehicle. The displacement amount of the following vehicle with respect to the center changes. Specifically, the closer the following vehicle is, the larger the amount of positional deviation becomes. For this reason, the rear view obtained through the display unit is not good, and the user may feel uncomfortable.

Therefore, it is an object of the present invention to provide an image processing device, an image processing system, an image processing method, and a program capable of providing a good rearward field of view even if the imaging unit is arranged offset from the center in the width direction of the vehicle. do.

An image processing apparatus according to an embodiment of the present invention includes an image acquisition unit that acquires an image captured by an imaging unit that is offset with respect to the center in the width direction of the vehicle and that is provided at the rear of the vehicle and captures an image of the rear; Among the images acquired by the image acquiring unit, the position of the rear image displayed on the display unit provided in the vehicle is determined by the offset amount in the width direction of the imaging unit and the distance between the vehicle and the following vehicle. and a control unit configured to suppress positional deviation of the following vehicle in the width direction in the rearward image based on the above.

It is possible to provide an image processing device, an image processing system, an image processing method, and a program that can provide a good rear field of view even if the imaging unit is arranged offset from the center in the width direction of the vehicle.

It is a figure showing vehicles 10 in which an image processing device and an image processing system of an embodiment are carried. 1 is a diagram showing an image processing system 100; FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing device 130; FIG. 2 is a diagram showing the configuration of an image processing device 130; FIG. FIG. 3 is a diagram showing a flowchart representing processing executed by an image processing device 130; FIG. 10 is a diagram showing an example of a rear image cut out from the camera 120 when the control unit 134 does not perform correction; 4 is a diagram showing a clipping position PC corrected with the correction amount CA and a clipping position P0 not corrected with the correction amount CA in the image 50 captured by the camera 120. FIG. 6 is a diagram showing a rear image 60; FIG. It is a figure explaining the acquisition method of the back image 60 by the modification of embodiment.

Embodiments to which the image processing apparatus, image processing system, image processing method, and program of the present invention are applied will be described below.

<Embodiment>
FIG. 1 is a diagram showing a vehicle 10 equipped with an image processing device and an image processing system according to an embodiment. An XYZ coordinate system will be defined and explained below. A direction parallel to the X axis (X direction), a direction parallel to the Y axis (Y direction), and a direction parallel to the Z axis (Z direction) are orthogonal to each other. The +X direction is the forward direction of the vehicle 10, the -X direction is the rearward direction of the vehicle 10, the +Y direction is the left direction, the -Y direction is the right direction, the +Z direction is the upward direction, and the -Z direction is the downward direction. In addition, the length, thickness, thickness, etc. of each part may be exaggerated below to make the configuration easier to understand.

A vehicle 10 has a rear spoiler 12 provided above a rear glass 11 . FIG. 1 shows the center C in the width direction (Y direction) of the vehicle 10 . A rear spoiler 12 is provided with a high mount stop lamp 13 . The center of the high mount stop lamp 13 in the Y direction coincides with the center C of the vehicle 10 in the width direction. Therefore, the camera 120 is provided on the left side of the high mount stop lamp 13 as an example. The camera 120 is an example of an imaging unit that acquires an image behind the vehicle 10 . Camera 120 is provided offset from center C in the width direction of vehicle 10 . The offset amount of the camera 120 from the center C in the width direction of the vehicle 10 is given by the design value of the vehicle 10 .

FIG. 2 is a diagram showing the image processing system 100. As shown in FIG. The image processing system 100 is an electronic mirror system mounted on a vehicle 10 (see FIG. 1). Image processing system 100 includes display 110 , camera 120 , and image processing device 130 . Although the display 110 and the image processing device 130 are shown separately here, the image processing device 130 may be provided inside the housing of the display 110 .

Display 110 is an example of a display unit. The display 110 is attached in place of the interior rearview mirror of the vehicle 10 as an example. The display 110 may be of a type in which the rear-view mirror is removed and attached instead of the rear-view mirror, or of a type that is attached over the rear-view mirror. The display 110 is, for example, a display device using liquid crystal or organic EL (Electroluminescence), and displays a rearward image generated by the image processing device 130 based on the image captured by the camera 120 . The display 110 may include a touch panel, operation buttons, a power button, etc. for receiving user's input operations.

The camera 120 is an example of an imaging unit, captures an image (video) behind the vehicle 10 , and outputs the captured image to the image processing device 130 . As described above, the camera 120 is offset leftward from the center C in the width direction of the vehicle 10 and provided at the rear of the vehicle 10 to capture an image of the rear of the vehicle 10 . As an example, the horizontal angle of view of the camera 120 is 43 degrees, and the vertical angle of view is 9 degrees.

The image processing device 130 determines the position of the rearward image displayed on the display 110 from among the images acquired by the camera 120 based on the offset amount of the camera 120 in the width direction of the vehicle 10 and the distance between the vehicle 10 and the following vehicle. Based on and, image processing is performed to correct the displacement of the following vehicle in the width direction of the vehicle 10 in the rearward image.

FIG. 3 is a diagram showing an example of the hardware configuration of the image processing device 130. As shown in FIG. The image processing apparatus 130 has a general computer configuration, for example, a CPU (Central Processing Unit) 301, a memory 302, a storage device 303, a communication I/F (Interface) 304, an external connection I/F 305, a GPU (Graphics Processing Unit) 306, a system bus 307, and the like.

The CPU 301 is, for example, an arithmetic device that executes each function of the image processing device 130 by executing a program stored in the storage device 303, memory 302, or the like. The memory 302 includes, for example, a random access memory (RAM), which is a volatile memory used as a work area for the CPU 301, and a ROM (ROM), which is a nonvolatile memory for storing a program for booting the image processing apparatus 130 and the like. Read Only Memory), etc.

The storage device 303 is, for example, a large-capacity storage device that stores an OS (Operating System), application programs, and various data, and is realized by, for example, a HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like. be.

A communication I/F 304 is an interface for communicating with an external device. For example, the communication I/F 304 connects the image processing device 130 to an in-vehicle network and enables communication with one or more ECUs mounted on the vehicle 10, a camera, and the like. Also, the communication I/F 304 may enable communication with an external device through a wireless LAN (Local Area Network), short-range wireless communication, or the like.

An external connection I/F 305 is an interface for connecting an external device to the image processing apparatus 130 . The external connection I/F 305 includes, for example, an interface for connecting the display 110 that displays the rear image to the image processing device 130, an interface for connecting the camera 120, an interface for connecting an external storage device, and the like. can be included.

The GPU 306 is an arithmetic device that mainly performs calculations related to image processing among the processes executed by the image processing device 130 . Note that the image processing apparatus 130 may execute image processing using the CPU 301 . In this case, the image processing device 130 may not have the GPU 306 .

A system bus 307 is connected to each component described above, and transmits, for example, address signals, data signals, and various control signals. Note that the hardware configuration of the image processing apparatus 130 shown in FIG. 3 is an example, and the image processing apparatus 130 may have various hardware configurations including a computer configuration.

FIG. 4 is a diagram showing the configuration of the image processing device 130. As shown in FIG. The image processing device 130 has a main control section 131 , an image acquisition section 132 , a distance detection section 133 , a control section 134 and a memory 135 .

The main control unit 131, the image acquisition unit 132, the distance detection unit 133, and the control unit 134 represent functions of programs executed by the image processing device 130 as functional blocks. A memory 135 functionally represents the memory of the image processing device 130 .

The main control unit 131 is a processing unit that supervises the processing of the image processing device 130, and performs processing other than the processing executed by the image acquisition unit 132, the distance detection unit 133, and the control unit .

The image acquisition unit 132 performs processing for acquiring an image of the rear of the vehicle 10 (own vehicle) captured by the camera 120 .

The distance detection unit 133 performs image processing on an image of the rear of the vehicle 10 captured by the camera 120 to detect the distance to the following vehicle one behind the vehicle 10 . As an example, the distance detection unit 133 detects the distance from the vehicle 10 to the following vehicle one behind by triangulation based on the positional relationship between the structure on the road and the following vehicle.

When the control unit 134 cuts out (extracts) the rearward image to be displayed on the display 110 from the rearward image of the vehicle 10 captured by the camera 120, the displacement in the width direction of the following vehicle included in the rearward image is suppressed. Then, the image at the corrected position is clipped as the rear image.

More specifically, the control unit 134 converts an image behind the vehicle 10 captured by the camera 120 into a rear image based on the distance to the vehicle following the vehicle 10 and the offset amount of the camera 120 . Calculate the amount of correction when cutting out . A distance detection unit 133 detects the distance to the following vehicle behind the vehicle 10 . The offset amount of camera 120 is the offset amount of camera 120 in the width direction of vehicle 10 and is stored in memory 135 . The rear image is an image cut out by the control unit 134 from among the images of the rear of the vehicle 10 captured by the camera 120 so as to be displayed on the display 110 .

Since the offset amount of the camera 120 is a fixed value included in the design value of the vehicle 10, the correction amount is obtained in advance based on a plurality of distances and the offset amount, and the table data that associates the distance and the correction amount is stored. 135. The control unit 134 may read the correction amount corresponding to the distance from the table data.

When cutting out the rear image from the image of the rear of the vehicle 10 captured by the camera 120, the control unit 134 cuts out the image included in the correction area shifted in the width direction from the predetermined area in front of the camera 120 by the correction amount. Cut out as an image. Then, the control unit 134 displays the clipped rear image on the display 110 . Details of the processing of the control unit 134 will be described later with reference to FIG.

The memory 135 stores programs and data necessary for processing executed by the main control unit 131, the image acquisition unit 132, the distance detection unit 133, and the control unit 134, data generated along with the processing, and the like. The data stored in the memory 135 includes table data that associates the distance to the following vehicle with the amount of correction.

FIG. 5 is a diagram showing a flowchart representing processing executed by the image processing device 130. As shown in FIG. The method realized by the flowchart shown in FIG. 5 is the image processing method of the embodiment, and is realized by the image processing apparatus 130 executing the image processing program of the embodiment.

When the power of the image processing system 100 is turned on and the processing shown in FIG. 5 becomes executable, the image processing apparatus 130 repeatedly executes the processing shown in FIG. 5 at a predetermined control cycle. When the process starts, the main control unit 131 controls and starts a series of processes.

The image acquisition unit 132 performs image acquisition processing for acquiring an image of the rear of the vehicle 10 captured by the camera 120 (step S1).

The distance detection unit 133 performs image processing on the image behind the vehicle 10 captured by the camera 120, and performs detection processing for detecting the distance to the following vehicle one behind the vehicle 10 (step S2). As described above, triangulation is used as an example to detect the distance from the vehicle 10 to the following vehicle one vehicle behind.

Based on the distance detected by the distance detection unit 133, the control unit 134 determines the correction amount when extracting the rear image from the image behind the vehicle 10 captured by the camera 120 from the table data stored in the memory 135. is obtained, and a control process of cutting out the rear image at a position shifted by the correction amount is performed (step S3).

The control unit 134 displays the clipped rear image on the display 110 (step S4). As described above, the display 110 displays the rear image corrected for the positional deviation based on the distance to the following vehicle. After finishing the process of step S4, a series of processes ends (end).

FIG. 6 is a diagram showing an example of a rearward image cut out from the camera 120 when the control unit 134 does not perform correction. FIG. 6 shows, from the bottom to the top, rearward images when the distances from the vehicle 10 to the following vehicle 20 one behind are 2 m, 3 m, 5 m, and 20 m. In FIG. 6, C0 represents the center of the image captured by camera 120 in the lateral direction (the width direction of vehicle 10). In FIG. 6, the center C0 is indicated by a dashed line. C1 represents the center of the following vehicle 20 in the width direction. In FIG. 6, the center C1 is indicated by a dashed line.

As shown in FIG. 6, the positional deviation D from the center C0 in the lateral direction (the width direction of the vehicle 10) of the center C1 of the following vehicle 20 in the rearward image is 20m, 5m, 3m, 2m, and it gets bigger as it gets shorter. The angle of view of the camera 120 is narrow, for example, the angle of view in the horizontal direction is 43 degrees and the angle of view in the vertical direction is 9 degrees. When using the camera 120 with such a narrow angle of view, if the camera 120 is offset from the center C in the width direction of the vehicle 10, the image of the following vehicle 20 is greatly deviated, and the user feels uncomfortable. likely to be caught. Therefore, the image processing device 130 performs correction so as to suppress such positional deviation.

FIG. 7 is a diagram showing a clipping position PC corrected with the correction amount CA and a clipping position P0 not corrected with the correction amount CA in the image 50 captured by the camera 120. As shown in FIG. A solid line indicates the cutout position P0, and a dashed line indicates the cutout position PC. The cutout position PC is a position obtained by shifting the cutout position P0 to the right side (left side in FIG. 7) of the vehicle 10 by the correction amount CA. It is shown slightly shifted downward with respect to the position P0.

A cutout position P0 that is not corrected with the correction amount CA is an area that is positioned at the center of the image 50 captured by the camera 120 in the vertical and horizontal directions and has a size that corresponds to the display area of the display 110 . In other words, the cut-out position P0 that is not corrected by the correction amount CA is displayed on the display 110 in front of the camera 120 that is offset to the left from the center C (see FIG. 1) in the width direction of the vehicle 10. A region having a size corresponding to the region.

FIG. 7A shows the cutout positions P0 and PC in the image 50 captured by the camera 120 when the distance between the vehicle 10 (self-vehicle) and the following vehicle 20 one behind is 20 m, and the cutout position P0 and PC. A rear image 60 cut out from the position PC is shown.

FIG. 7B shows the cutout positions P0 and PC in the image 50 captured by the camera 120 when the distance between the vehicle 10 (own vehicle) and the following vehicle 20 one behind is 2 m, and the cutout position P0 and PC. A rear image 60 cut out from the position PC is shown.

When the distance from the vehicle 10 to the following vehicle 20 one behind is 2 m, the positional deviation of the following vehicle 20 in the lateral direction in the image 50 captured by the camera 120 is greater when the distance is 2 m. Therefore, the correction amount CA of the cutout position PC with respect to the cutout position P0 is increased.

Therefore, the control unit 134 sets the correction amount CA larger as the detected value of the distance from the vehicle 10 to the following vehicle 20 one behind is smaller (shorter). (longer), the smaller the correction amount CA is set. By setting in this way, the cutout position PC in the image 50 captured by the camera 120 is corrected according to the distance (detection value) detected by the distance detection unit 133 .

As a result, even if the camera 120 is arranged offset from the center C (see FIG. 1) in the width direction of the vehicle 10, the displacement of the following vehicle 20 in the rearward image displayed on the display 110 can be suppressed. can be done.

FIG. 8 is a diagram showing a rearward image 60. As shown in FIG. FIG. 8(A) shows a rearward image 60 obtained at the clipping position P0 (see FIGS. 7(A) and 7(B)) that is not corrected with the correction amount CA for comparison. FIG. 8(B) shows the rearward image 60 obtained at the cutout position PC (see FIGS. 7(A) and 7(B)) corrected with the correction amount CA. FIG. 8(C) shows, for comparison, an image captured by a camera placed on the center C in the width direction of the vehicle 10. In the image, the rearward image obtained at the cutout position corresponding to the cutout position P0 that is not corrected with the correction amount CA. An image 60 is shown.

8(A), 8(B), and 8(C), the left side shows the rear image 60 when the distance to the following vehicle 20 is 20 m, and the right side shows the rear image 60 when the distance to the following vehicle 20 is 2 m. 6 shows a rearward image 60 in the case of . In addition, in FIGS. 8A, 8B, and 8C, the center C2 represents the center of the rear image 60 in the horizontal direction.

Comparing FIG. 8(A) and FIG. 8(B), when the distance is 20 m, the displacement of the following vehicle 20 in FIG. 8(A) is small. It can be clearly seen that the position of the following vehicle 20 is greatly displaced, and the position of the following vehicle 20 is corrected toward the center C2 in the rearward image 60 shown in FIG. 8(B).

Comparing FIG. 8B and FIG. 8C, it can be seen that the position and appearance of the following vehicle 20 in the rearward image 60 shown in FIG. , there is almost no difference in the position and appearance of the following vehicle 20 in the rearward image 60 shown in FIG. 8(C). Strictly speaking, when the distance is 2 m, the following vehicle 20 in the rearward image 60 shown in FIG. As seen from the front, the direction is off, but it's at a level that doesn't matter much.

As described above, even if the camera 120 is arranged offset from the center C (see FIG. 1) in the width direction of the vehicle 10, the displacement of the following vehicle 20 in the rearward image displayed on the display 110 is suppressed. can do.

Therefore, an image processing device 130, an image processing system 100, an image processing method, and a program are provided that can provide a good rear view even when the camera 120 is arranged offset from the center C in the width direction of the vehicle 10. be able to. Since the displacement of the following vehicle 20 in the rearward image is suppressed in this way, it is possible to suppress the discomfort felt by the user who checks the rear on the display 110 . In particular, even if a camera 120 with a narrow angle of view, for example, a horizontal angle of view of 43 degrees and a vertical angle of view of 9 degrees, is used, it is possible to ensure a good rearward field of view with little discomfort. .

In addition, since image processing device 130 includes distance detection unit 133 that detects the distance to following vehicle 20 based on image 50 captured by camera 120, the distance from image 50 captured by camera 120 to following vehicle 20 is determined. The distance can be detected, and the distance to the following vehicle 20 can be detected with a simple configuration.

The control unit 134 sets the correction amount CA larger as the detected value of the distance to the following vehicle 20 is smaller, and sets the correction amount CA smaller as the detected value of the distance to the following vehicle 20 is larger. , the cutting position PC for obtaining the rear image 60 in the image 50 can be corrected, and according to the distance to the following vehicle 20, the following vehicle 20 can be positioned substantially in the center of the image behind the image. An image 60 can be obtained.

In addition, since the following vehicle 20 is a following vehicle that exists one vehicle behind the vehicle 10, even if there are a plurality of vehicles behind the vehicle 10, the following vehicle that looks the largest in the rear image 60 is the following vehicle. A rearward image 60 that matches the vehicle 20 can be acquired.

In the above description, a form using the corrected clipping position PC for obtaining the rearward image 60 in the image 50 according to the distance from the following vehicle 20 has been described, but the following may be adopted.

FIG. 9 is a diagram illustrating a method of obtaining a rearward image 60 according to a modified example of the embodiment. In the acquisition method according to the modified example, in the image 50 captured by the camera 120, after clipping the rear image at the clipping position P0 (see FIGS. 7A and 7B) that is not corrected with the correction amount CA, Move the clipped image by the correction amount.

In FIGS. 9(A) and 9(B), C1 represents the center of the following vehicle 20 in the width direction and is indicated by a dashed line. A center C2 represents the center of the rear image 60 in the horizontal direction and is indicated by a solid line. FIG. 9A shows the rear image 60 cut out at the cutout position P0 (see FIGS. 7A and 7B). The center C1 in the width direction of the following vehicle 20 is shifted to the right side (left side in FIG. 9A) of the vehicle 10 with respect to the center C2 in the lateral direction of the rearward image 60 . The displacement is D1.

In such a case, in the rear image 60 shown in FIG. 9A, the rear image 60 is positioned so that the center C1 in the width direction of the following vehicle 20 and the center C2 in the lateral direction of the rear image 60 are aligned with each other. 10 to the right in the width direction (left side in FIG. 9A), the center C1 in the width direction of the following vehicle 20 and the rear image 60 are aligned as shown in FIG. 9B. It can be aligned with the center C2 in the lateral direction. The correction amount is D1, which is equal to the correction amount CA described above.

Even when such a correction method is used, when the camera 120 is offset from the center C (see FIG. 1) in the width direction of the vehicle 10, the rearward image displayed on the display 110 Positional deviation of the following vehicle 20 can be suppressed.

Therefore, an image processing device 130, an image processing system 100, an image processing method, and a program are provided that can provide a good rear view even when the camera 120 is arranged offset from the center C in the width direction of the vehicle 10. be able to.

Although the image processing apparatus, image processing system, image processing method, and program according to exemplary embodiments of the present invention have been described above, the present invention is not limited to the specifically disclosed embodiments. Various modifications and changes are possible without departing from the scope of the claims.

10 vehicle 20 following vehicle 50 image 60 rear image 100 image processing system 110 display 120 camera 130 image processing device 131 main control unit 132 image acquisition unit 133 distance detection unit 134 control unit 135 memory

Claims (9)

an image acquisition unit that acquires an image captured by an imaging unit that is provided at the rear part of the vehicle and captures the rearward direction, offset from the center in the width direction of the vehicle;
Among the images acquired by the image acquiring unit, the position of the rearward image displayed on the display unit provided in the vehicle is determined by the offset amount in the width direction of the imaging unit and the distance between the vehicle and the following vehicle. and a control unit that performs correction so as to suppress positional deviation of the following vehicle in the width direction in the rearward image based on the above. The image processing device according to claim 1, wherein the control section corrects the position of the rear image by correcting the position of the image cut out from the image captured by the imaging section according to the positional deviation. The image according to claim 1, wherein the control unit corrects the position of the rear image by correcting a region displayed on the display unit in the image captured by the imaging unit according to the positional deviation. processing equipment. The image processing apparatus according to any one of claims 1 to 3, further comprising a distance detection section that detects a distance to said following vehicle based on the image captured by said imaging section. The control unit sets the correction amount in the correction to be larger as the detected value of the distance to the following vehicle is smaller, and sets the correction amount in the correction to be smaller as the detected value of the distance to the following vehicle is larger. Item 5. The image processing apparatus according to any one of Items 1 to 4. The image processing device according to any one of claims 1 to 5, wherein the following vehicle is a following vehicle that exists one behind the vehicle. an imaging unit that is offset from the center in the width direction of the vehicle and is provided at the rear part of the vehicle and captures an image of the rear;
a display unit provided on the vehicle;
an image acquisition unit that acquires an image captured by the imaging unit;
Among the images acquired by the image acquisition unit, the position of the rearward image displayed on the display unit is determined based on the offset amount in the width direction of the imaging unit and the distance between the vehicle and the following vehicle. and a control unit that corrects the positional deviation of the following vehicle in the width direction in the rearward image so as to be suppressed. an image acquisition process for acquiring an image captured by an imaging unit that is offset from the center in the width direction of the vehicle and is provided at the rear of the vehicle and captures the rear;
Among the images acquired by the image acquisition process, the position of the rearward image displayed on the display unit provided in the vehicle is determined by the offset amount in the width direction of the imaging unit and the distance between the vehicle and the following vehicle. and a control process for performing correction so as to suppress positional deviation of the following vehicle in the width direction in the rearward image based on the above. A program that causes an image processing apparatus to execute the image processing method according to claim 8 .

Images