WO2023120603A1

WO2023120603A1 - Imaging device and image processing method

Info

Publication number: WO2023120603A1
Application number: PCT/JP2022/047211
Authority: WO
Inventors: 昭典佐藤; 薫草深
Original assignee: 京セラ株式会社
Priority date: 2021-12-21
Filing date: 2022-12-21
Publication date: 2023-06-29

Abstract

The present invention is provided with a windshield, a camera configured so as to capture an image of at least the eyes of a driver of a moving body through the windshield, and a controller which controls the camera. The controller carries out conversion to transform the image captured by the camera into a first processing image in which overlapping images are positioned side by side in a prescribed direction on the basis of a first point spread function representing the effect of the windshield, corrects the first processing image into a second processing image in which overlapping images have been removed on the basis of a second point spread function representing features of the overlapping images in the first processing image, carries out a reverse conversion relative to the conversion to transform the second processing image into the first processing image, and generates a third processing image in which the effects of the first point spread function have been eliminated.

Description

Imaging device and image processing method

The present disclosure relates to an imaging device and an image processing method.

A conventional imaging device is described in Patent Document 1, for example.

JP 2021-138319 A

An imaging device according to the present disclosure includes a windshield, a camera configured to capture an image of at least the eyes of a driver of a mobile object through the windshield, and a controller that controls the camera, wherein the controller transforming the image captured by the camera into a first processed image in which double images are positioned side by side in a predetermined direction, based on a first point spread function representing the effect of the windshield on the image captured by the camera; and correcting the first processed image to a second processed image from which the double image is removed based on a second point spread function representing the feature of the double image in the first processed image; An inverse transform of the transform is performed on the second processed image to produce a third processed image in which the effects of the first point spread function are removed.

In the image processing method of the present disclosure, a windshield, a camera configured to capture an image of at least a driver's eye through the windshield, and a controller for controlling the camera are prepared, and the controller controls the transforming an image captured by the camera into a first processed image in which double images are positioned side by side in a predetermined direction based on a first point spread function representing the influence of the windshield; is corrected to a second processed image from which the double image is removed based on a second point spread function representing the feature of the double image in the first processed image, and the second processed image is converted to the transformed to produce a third processed image in which the effects of the first point spread function have been removed.

Objects, features and advantages of the present disclosure will become more apparent from the following detailed description and drawings.
1 is a diagram schematically showing an overall configuration of a stereoscopic image display device including an imaging device according to an example of an embodiment of the present disclosure; FIG. It is a figure which shows the structure of a three-dimensional projection apparatus typically. It is a figure which shows the double image elimination procedure by distortion processing. It is a figure which shows the double image elimination procedure by distortion processing. It is a figure which shows the double image elimination procedure by distortion processing. It is a figure which shows the double image elimination procedure by distortion processing. It is a figure which shows the structure of a three-dimensional projection apparatus. 1 is a diagram showing a mobile object according to an embodiment of the present disclosure; FIG. FIG. 4 illustrates the path of light reflected by the windshield of the imaging device; FIG. 10 is a diagram showing the relationship between the thickness of the windshield and the amount of deviation of double images; 4 is a flowchart for explaining the operation of the imaging device; FIG. 10 is a diagram showing the difference in the amount of deviation of double images depending on the reflection position of the windshield; FIG. 5 is a diagram showing changes in the width of a double image due to bending of the windshield; FIG. 10 is a diagram showing a photographing panel on which a measurement pattern is drawn; 12 is a diagram showing an image to be processed of the photographing panel shown in FIG. 11; FIG.

For example, as described in the above-mentioned Patent Document 1, conventionally, there is known an imaging device mounted on a vehicle, which includes a camera for imaging the driver's eyes through the windshield of the vehicle.

The image captured through the conventional windshield produces a double image and becomes an unclear image, but it is required to detect the driver's eyes with high accuracy.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the diagrams used in the following description are schematic. The dimensions, ratios, etc. on the drawings do not necessarily match the actual ones.

FIG. 1 is a diagram schematically showing the overall configuration of a stereoscopic image display device 2 including an imaging device 1 according to an example of the embodiment of the present disclosure. The stereoscopic image display device 2 includes an imaging device 1 and a three-dimensional projection device 12 . The three-dimensional projection device 12 includes a display controller 107 , an acquisition unit 103 , a memory 108 , an illuminator 4 , a display panel 5 and a parallax barrier 6 . The stereoscopic image display device 2 may be mounted on the mobile object 10 . The imaging device 1 includes a windshield 25, a camera 11 configured to capture an image of at least the eyes 31 of a driver 13 of a mobile body 10 through the windshield 25, and a controller 7 for controlling the camera 11. .

The controller 7 is configured as a processor, for example. Controller 7 may include one or more processors. The processor may include a general-purpose processor configured to load a specific program to perform a specific function, and a dedicated processor specialized for specific processing. A dedicated processor may include an Application Specific Integrated Circuit (ASIC). The processor may include a programmable logic device (PLD). A PLD may include an FPGA (Field-Programmable Gate Array). The controller 7 may be either a SoC (System-on-a-Chip) with which one or more processors cooperate, or a SiP (System In a Package).

A "moving object" in the present disclosure may include, for example, a vehicle, a ship, an aircraft, and the like. Vehicles may include, for example, automobiles, industrial vehicles, railroad vehicles, utility vehicles, fixed-wing aircraft that travel on runways, and the like. Motor vehicles may include, for example, cars, trucks, buses, motorcycles, trolleybuses, and the like. Industrial vehicles may include, for example, industrial vehicles for agriculture and construction, and the like. Industrial vehicles may include, for example, forklifts, golf carts, and the like. Industrial vehicles for agriculture may include, for example, tractors, tillers, transplanters, binders, combines, lawn mowers, and the like. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, mobile cranes, tippers, road rollers, and the like. Vehicles may include those that are powered by humans. Vehicle classification is not limited to the above example. For example, automobiles may include road-drivable industrial vehicles. Multiple classifications may contain the same vehicle. Vessels may include, for example, marine jets, boats, tankers, and the like. Aircraft may include, for example, fixed-wing aircraft, rotary-wing aircraft, and the like.

A case where the mobile object 10 is a passenger car will be described below as an example. The mobile body 10 is not limited to a passenger car, and may be any of the above examples. The camera 11 may be attached to the mobile object 10 . Camera 11 is configured to capture an image including the face (human face) of driver 13 of mobile object 10 . The mounting position of the camera 11 is arbitrary inside and outside the moving body 10 . For example, camera 11 may be located within the dashboard of mobile 10 .

The camera 11 may be a visible light camera or an infrared camera. The camera 11 may have the functions of both a visible light camera and an infrared camera. The camera 11 may include, for example, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The position of the eye 31 of the driver 13 may be the pupil position. In the imaging device 1 of the present disclosure, the target image is a captured image captured by the camera 11 . The captured image includes the eye 31 of the driver 13 or the part of the face for which the relative positional relationship with the eye 31 of the driver 13 is specified. The eyes 31 of the driver 13 included in the captured image may be both eyes, or may be only the right eye 31R or the left eye 31L. The part of the face for which the relative positional relationship with the eyes 31 of the driver 13 is specified may be, for example, the eyebrows or the nose.

The imaging device 1 is configured to capture an image of at least the eyes 31 of the driver 13 with light reflected through the windshield 25 . The imaging device 1 is configured to acquire an image of a subject and generate an image of the subject. A camera 11 of the imaging device 1 includes an imaging element. The imaging device may include, for example, a CCD (Charge Coupled Device) imaging device or a CMOS (Complementary Metal Oxide Semiconductor) imaging device. The imaging device 1 is arranged so that the face of the driver 13 is located on the subject side. The imaging device 1 is configured to detect the position of at least one of the driver's 13 left eye 31L and right eye 31R. For example, the imaging device 1 may be configured to use a predetermined position as an origin and detect the direction and amount of displacement of the position of the left eye 31 from the origin. The imaging device 1 may be configured to detect the position of at least one of the left eye 31L and the right eye 31R from the captured image of the imaging device 1 captured by the camera. The imaging device 1 may be configured to use two or more cameras 11 to detect the position of at least one of the left eye 31L and the right eye 31R as coordinates in a three-dimensional space.

The imaging device 1 does not have to be equipped with the camera 11. The imaging device 1 may comprise an input terminal configured to receive signals from a camera 11 external to the device. A camera 11 outside the device may be directly connected to the input terminal. An external camera 11 may be indirectly connected to the input terminal via a shared network. An imaging device 1 that does not include a camera 11 may include an input terminal configured for the camera 11 to input a video signal. The imaging device 1 without the camera 11 may be configured to detect the position of at least one of the left eye 31L and the right eye 31R from the video signal input to the input terminal.

The imaging device 1 may include a sensor. The sensor may be an ultrasonic sensor, an optical sensor, or the like. The imaging device 1 may be configured to detect the position of the head of the driver 13 with a sensor and detect the position of at least one of the left eye 31L and the right eye 31R based on the position of the head. The imaging device 1 may be configured to detect the position of at least one of the left eye 31L and the right eye 31R as coordinates in a three-dimensional space using one or more sensors.

The imaging device 1 may be configured to detect the movement distance of the left eye 31L and the right eye 31R along the eyeball arrangement direction based on the detection result of the position of at least one of the left eye 31L and the right eye 31R. .

The imaging device 1 is configured to output position information indicating the positions of the eyes 31 of the driver 13 to the acquisition unit 103 . The imaging device 1 may be configured to output the position information to the acquisition unit 103 via a communication network such as wired, wireless, and CAN (Controller Area Network).

The camera 11 is configured to be capable of capturing a first captured image including an image of an area where the face of the driver 13 is assumed. In this embodiment, the driver 13 may be, for example, the driver of the mobile object 10, which is a passenger car. The area where the driver's face is assumed to be, for example, near the top of the driver's seat. The camera 11 may be attached to the mobile object 10 . The mounting position of the camera 11 is arbitrary inside and outside the moving body 10 .

A captured image captured by the camera 11 is input to the display controller 107 via the acquisition unit 3 . The controller 7 is configured to detect the position of at least one eye 31L, 31R of the driver 13 based on the captured image. The detection result by the imaging device 1 may be coordinate information indicating the pupil position of the driver's 13 left eye 31L or right eye 31R.

The imaging device 1 is configured to output coordinate information regarding the detected pupil position of the left eye 31L or right eye 31R to the three-dimensional projection device 12 . Based on this coordinate information, the 3D projection device 12 may be configured to control the projected image.

In the imaging device 1, the controller 7 may be an external device. The camera 11 may be configured to output the captured image to the external controller 7 . The external controller 7 may be configured to detect the pupil position of the driver's 13 left eye 31L or right eye 31R from the image output from the camera 11 . The external controller 7 may be configured to output coordinate information regarding the detected pupil position of the left eye 31L or right eye 31R to the stereoscopic image display device 2 . Based on this coordinate information, the stereoscopic image display device 2 may be configured to control the image to be projected. The camera 11 may be configured to output a captured image captured via wired communication or wireless communication to the external controller 7 . The external controller 7 may be configured to output the coordinate information to the stereoscopic image display device 2 via wired communication or wireless communication. Wired communication may include CAN, for example.

FIG. 4 is a diagram showing the configuration of the three-dimensional projection device 12, and FIG. 5 is a diagram showing a moving body according to one embodiment of the present disclosure. The position of the three-dimensional projection device 12 is arbitrary inside and outside the moving body 10 . For example, the 3D projection device 12 may be located within the dashboard of the vehicle 10 . The three-dimensional projection device 12 is configured to emit image light toward the windshield 25 .

The windshield 25 is a reflector configured to reflect image light emitted from the three-dimensional projection device 12 . Image light reflected by the windshield 25 reaches the eyebox 16 . The eye box 16 is a real space area where the eyes 31L and 31R of the driver 13 are assumed to exist, for example, considering the physique, posture, and changes in the posture of the driver 13 . The shape of the eyebox 16 is arbitrary. Eyebox 16 may include two-dimensional or three-dimensional regions. A dashed line shown in FIG. 2 indicates a path along which at least part of image light emitted from the three-dimensional projection device 12 reaches the eyebox 16 . The path traveled by the image light is also referred to as the optical path. The image light emitted from the three-dimensional projection device 12 represents parallax images including a right-eye image and a left-eye image. When the eyes 31</b>L and 31</b>R of the driver 13 are positioned within the eyebox 16 , the driver 13 can visually recognize the virtual image 14 by the parallax image light reaching the eyebox 16 . The virtual image 14 is positioned on a path (indicated by a dashed line in FIG. 1) obtained by extending the path from the windshield 25 to the eyes 31L and 31R to the front of the mobile object 10 . The three-dimensional projection device 12 can function as a head-up display by making the driver 13 visually recognize the virtual image 14 . In FIG. 1, the direction in which the eyes 31L and 31R of the driver 13 are aligned corresponds to the x-axis direction. The vertical direction corresponds to the y-axis direction. The imaging range of camera 11 includes eyebox 16 .

At least part of the image light emitted from the three-dimensional projection device 12 reaches the windshield 25 via the optical member 110 (see FIG. 4). The image light is reflected by the windshield 25 and reaches the eyes 31 of the driver 13 . The eyes 31 of the driver 13 can visually recognize the first virtual image 14a located on the side of the windshield 25 in the negative direction of the z-axis. The first virtual image 14a corresponds to the image displayed by the three-dimensional projection device 12 . The opening region 6b of the parallax barrier 6 and the light shielding surface 6a form a second virtual image 14b in front of the windshield 25 and on the windshield 25 side of the first virtual image 14a. The driver 13 can visually recognize the image as if the display panel were present at the position of the first virtual image 14a and the parallax barrier 6 was present at the position of the second virtual image 14b. In the description below,

The three-dimensional projection device 12 causes the image light reflected by the windshield 25 to reach the driver's 13 left eye 31L and right eye 31R. That is, the three-dimensional projection device 12 causes the image light to travel from the stereoscopic image display device 2 to the left eye 31L and right eye 31R of the driver 13 along the optical path 140 indicated by the dashed line in FIG. The driver 13 can visually recognize the image light arriving along the optical path 140 as the virtual image 14 . The stereoscopic image display device 2 can provide stereoscopic vision according to the movement of the driver by controlling the display according to the positions of the left eye 31L and right eye 31R of the driver 13 .

A part of the configuration of the three-dimensional projection device 12 may be shared with other devices or parts included in the mobile body 10 . For example, the moving body 10 also uses the windshield 25 as part of the configuration of the imaging device 1 .

The display panel 5 is not limited to a transmissive display panel, and other display panels such as a self-luminous display panel can also be used. Transmissive display panels include MEMS (Micro Electro Mechanical Systems) shutter type display panels in addition to liquid crystal panels. Self-luminous display panels include organic EL (electro-luminescence) and inorganic EL display panels. If a self-luminous display panel is used as the display panel 5, the irradiator 4 becomes unnecessary. When a self-luminous display panel is used as the display panel 5, the parallax barrier 6 is positioned on the side of the display panel 5 from which the image light is emitted.

A stereoscopic image display device 2 according to an embodiment of the present disclosure includes an imaging device 1 and a three-dimensional projection device 12, as shown in FIG.

The imaging device 1 may be configured to acquire captured images from a camera 11 that is configured to capture an image of a space where the driver's eyes are expected to exist at regular imaging time intervals (eg, 20 fps). . The imaging device 1 is configured to sequentially detect images of a left eye (first eye) 31L and a right eye (second eye) 31R from captured images acquired from the camera 11 . The imaging device 1 is configured to detect respective positions of the left eye 31L and the right eye 31R in the real space based on the images of the left eye 31L and the right eye 31R in the image space. The imaging device 1 may be configured to detect the positions of the left eye 31L and the right eye 31R from the image captured by one camera 11 as coordinates in a three-dimensional space. The imaging device 1 may be configured to detect the positions of the left eye 31L and the right eye 31R as coordinates in a three-dimensional space from images captured by two or more cameras 11 . The imaging device 1 may include a camera 11 .

The stereoscopic image display device 2 includes an acquisition unit 103, an illuminator 4, a display panel 5, a parallax barrier 6 as an optical element, a memory 108, and a display controller 107. The image captured by the imaging device 1 is captured as, for example, a double image due to the influence of multiple reflections by the windshield 25 due to the light being reflected by the outer and inner surfaces of the windshield 25, and is accurately captured by the left eye 31L or Since the position of the right eye 31R cannot be detected, the controller 7 of the imaging device 1 is configured to remove double images.

FIG. 6 is a diagram showing paths of light reflected by the windshield 25 of the imaging device 1. FIG. When the eye 31 of the driver 13 is imaged by the camera 11 through the windshield 25, light is reflected on the outer and inner surfaces of the windshield 25 as shown in FIG. Let i(x, y) be an image captured by the camera 11, s(x, y) be an original image without double images, g(x, y) be a point spread function that defines a double image, and g(x, y) be a double image. Assuming that a captured image with an image is i(x, y) and the convolution operation code is "*", the following equation (1) holds.

i (x, y) = s (x, y) * g (x, y) (1)

An xy coordinate system was set with the coordinate origin (0, 0) as the upper left corner of the image, the x axis extending in the horizontal direction, and the y axis extending in the vertical direction. The point spread function may be determined using, for example, a pattern image for obtaining the point spread function, such as a captured image of a display panel on which a point image is displayed, or may be calculated based on the shape of the windshield 25. good.

Denote the Fourier transform of the function by F, the inverse Fourier transform by F ⁻¹ , and the Fourier transform of i(x, y), g(x, y) by I(u, v), G(u, v) , the original image s(x, y) can be obtained by Equation 2 below.

s(x, y)=F ⁻¹ (I(u, v)/G(u, v)) (2)

Alternatively, the original image s(x, y) can be obtained by Equation 3 below.

s (x, y) = K (x, y) * i (x, y) (3)

Here, the kernel K in Equation 3 above can be expressed by Equation 4 below.

K(x, y)=F ⁻¹ (1/G(u, v)) (4)

When obtaining the original image s(x, y) using the above formulas 2 and 3, it is necessary to execute two-dimensional calculations, which imposes a heavy load on the controller 7. To reduce this load, the following processing is performed. The controller 7 reads out the point spread function stored in the memory 108 and executes double image elimination processing. Next, the controller 7 deforms the captured image by distortion processing so that the double images are aligned in the y-axis direction, which is a predetermined direction, and converts the captured image into a first processed image using the first point spread function. Based on the point spread function of the double image of the processed image, the first processed image from which the double image has been removed is corrected by the reverse distortion process, and a restored image is produced.

In order to reduce the load on the controller 7, in this embodiment, as shown in FIGS. This is used to determine the point spread function. As a result, even if the sizes of multiple point images are different, the entire distribution can be obtained by applying a one-dimensional kernel to the entirety and interpolating the multiple point images. can be reduced.

3A to 3D are diagrams for explaining the procedure for removing double images by distortion processing. In this embodiment, in order to perform double image removal processing by distortion processing, as an example, a panel on which a plurality of point images shown in FIG. 3D are drawn is captured by the camera 11 through the windshield 25 . The captured image becomes a double image as shown in FIG. 3A due to the influence of the windshield 25, and each point image becomes two point images. The distortion in the x-axis direction of each of these two point images is corrected. Also, as shown in FIG. 3B, the interval in the y-axis direction between the two point images is made uniform. This processing is performed by enlarging in the y-axis direction. This distortion processing may be performed based on a point spread function representing the influence of the windshield 25 on the captured image. Due to the distortion processing, the two point images are positioned at the same interval in the y-axis direction. The same one-dimensional kernel K, which depends only on the y-coordinate, can thereby be applied to the entire captured image. Processing by the controller 7 is simplified because the same one-dimensional kernel K can be used for all pixels. Removing the double images from the image of FIG. 3B yields the point image shown in FIG. 3C. 3A to 3B, the image can be restored to the original image shown in FIG. 3D from which the double image is removed by performing the process of returning the image by the amount of movement for aligning it with the y-axis. I understand.

Since the controller 7 is a one-dimensional integral, the amount of calculation can be reduced and the calculation speed can be increased. If the alignment of the two point images caused by the influence of the windshield 25 deviates from the y-axis direction, integration will be performed in a long and narrow two-dimensional area. By using such an image, the area to be integrated with respect to the x-coordinate can be reduced, and the arithmetic processing load of the controller 7 can be reduced.

In the controller 7, the first processed image is transformed so that the distance in the y-axis direction between the double images is the same. This allows double images to be removed with a single kernel, reducing computational effort.

If the point spread function g(x, y), which is the first point spread function representing the double image that is the effect of the windshield 25, is known as described above, the captured image i(x , y) can restore the original image s(x, y) from which double images have been removed. If the point spread function g(x, y) is not known, the second point spread function g(x, y) is estimated from the photographed image of FIG. 3D, and the original image s(x, y) is you can ask.

The aforementioned second point spread function is determined based on the first point spread function. Also, the second point spread function may be determined based on the first point spread function and the deformation due to the distortion process.

The imaging device 1 may include a light emitting element (Light Emitting Diode; LED) that emits infrared rays. The captured image described above is a difference image between the first image captured without illumination of the driver and the second image captured with illumination of the driver. Thereby, the influence of light transmitted through the windshield 25 can be eliminated.

Next, the three-dimensional projection device 12 will be explained.

The acquisition unit 3 is configured to acquire the position data indicating the positions of the eyes sequentially transmitted by the imaging device 1 .

The illuminator 4 can be configured to illuminate the display panel 5 in a planar manner. The illuminator 4 may include a light source, a light guide plate, a diffusion plate, a diffusion sheet, and the like. The irradiator 4 emits irradiation light from a light source, and is configured to make the irradiation light uniform in the surface direction of the display panel 5 by means of a light guide plate, a diffusion plate, a diffusion sheet, and the like. Illuminator 4 may be configured to emit homogenized light towards display panel 5 .

The display panel 5 may employ a display panel such as a transmissive liquid crystal display panel. The display panel 5 has a plurality of partitioned areas on a planar active area. The active area is configured to display parallax images. The parallax images include a left-eye image (first image) and a right-eye image (second image) having parallax with respect to the left-eye image. The plurality of partitioned regions are regions partitioned in a first direction x and a direction y perpendicular to the first direction within the plane of the active area. The first direction x may be, for example, the horizontal direction. The direction y orthogonal to the first direction x may be, for example, the vertical direction. A direction orthogonal to the horizontal and vertical directions may be referred to as the depth direction. In the drawings, the horizontal direction is represented as the x-axis direction, the vertical direction is represented as the y-axis direction, and the depth direction is represented as the z-axis direction.

　One sub-pixel corresponds to each of the plurality of partitioned areas. Accordingly, the active area comprises a plurality of sub-pixels arranged in a grid along the horizontal and vertical directions.

Each of the plurality of sub-pixels can correspond to any one of colors R (Red), G (Green), and B (Blue). A set of three sub-pixels of R, G, and B can constitute one pixel. One pixel may be referred to as one pixel. A plurality of sub-pixels forming one pixel may be arranged horizontally. Multiple sub-pixels of the same color may be aligned vertically. The horizontal length Hpx of each of the plurality of sub-pixels may be the same as each other. The vertical length Hpy of each of the plurality of sub-pixels may be the same.

The display panel 5 is not limited to a transmissive liquid crystal panel, and other display panels such as organic EL can be used. Transmissive display panels include MEMS (Micro Electro Mechanical Systems) shutter type display panels in addition to liquid crystal panels. Self-luminous display panels include organic EL (electro-luminescence) and inorganic EL display panels. If the display panel 5 is a self-luminous display panel, the stereoscopic image display device 2 does not need to include the illuminator 4 .

The parallax barrier 6 is configured to define the light beam direction of the image light of the parallax image emitted from the display panel 5 . The parallax barrier 6 has a plane along the active area, as shown in FIG. The parallax barrier 6 is separated from the active area by a predetermined distance (gap) g. A parallax barrier 6 may be located on the opposite side of the illuminator 4 with respect to the display panel 5 . The parallax barrier 6 is positioned on the side of the display panel 5 facing the illuminator 4 . The parallax barrier 6 is an optical panel configured to define the traveling direction of incident light. As shown in the example of FIG. 1, when the parallax barrier 6 is positioned closer to the illuminator 4 than the display panel 5 is, the light emitted from the illuminator 4 enters the display panel 5 and is further parallaxed. Incident on the barrier 6 . In this case, the parallax barrier 6 is configured to block or attenuate part of the light emitted from the illuminator 4 and transmitted through the display panel 5, and to transmit the light toward the left eye 31L or the right eye 31R. The display panel 5 emits the incident light traveling in the direction defined by the parallax barrier 6 as image light traveling in the same direction. When the display panel 5 is positioned closer to the illuminator 4 than the parallax barrier 6 , the light emitted from the illuminator 4 enters the display panel 5 and then the parallax barrier 6 . In this case, the parallax barrier 6 is configured to block or attenuate part of the image light emitted from the display panel 5 and transmit the other part toward the eyes 31 of the driver 13. .

The display controller 107 is configured to store in the memory 108 the position data indicating the position (actually measured position) of the eye 31 acquired by the acquisition unit 103 and the order in which the position data was acquired. The memory 108 sequentially stores measured positions of the eye 31 based on each of a plurality of captured images captured at predetermined imaging time intervals. The memory 108 may also store the order in which the eye 31 is positioned at the actual measurement position. The predetermined imaging time interval is a time interval between one captured image and another captured image, which can be appropriately set according to the performance and design of the camera 11 .

The display controller 107 is configured as a processor, for example. Controller 7 may include one or more processors. The processor may include a general-purpose processor configured to load a specific program to perform a specific function, and a dedicated processor specialized for specific processing. A dedicated processor may include an Application Specific Integrated Circuit (ASIC). The processor may include a programmable logic device (PLD). A PLD may include an FPGA (Field-Programmable Gate Array). The controller 7 may be either a SoC (System-on-a-Chip) with which one or more processors cooperate, or a SiP (System In a Package).

The memory 108 is composed of arbitrary storage devices such as RAM (Random Access Memory) and ROM (Read Only Memory). The memory 108 is configured to store information received by the input unit, information converted by the controller 7, and the like. For example, the memory 108 is configured to store position information of the eye 31 of the driver 13 obtained by the input unit.

FIG. 6 is a diagram showing paths of light reflected by the windshield 25 of the imaging device 1. FIG. The windshield 25 has a translucent glass layer 25a and an infrared reflective film 25b interposed as an intermediate film in the glass layer 25a. The light is reflected at the interface between the glass layer 25a and the infrared reflecting film 25b and enters the imaging device 1. FIG. Therefore, the image of the driver incident on the imaging device 1 is received as a double image. The controller 7 is configured to be able to remove double images.

Since the spread function of the double image is not local, the larger the image size, the larger the kernel. However, if there is a difference in the intensity of the double images, the kernel will be small, so in practice it can be processed by a convolution integral of a small size.

FIG. 7 is a diagram showing the relationship between the thickness T of the windshield 25 and the deviation amount D of the double image. The images captured by the camera 11 are shifted by D and moved to the same position, creating a double image.

(5)

As a demonstration work, if T = 6 mm and S = 13°, the position shifted by D = 10 mm will be reflected. Since this is about the size of the pupil, it affects pupil detection. Assuming that the angle of view of the camera 11 is in the range of ±10°, there is a difference in the x direction of 2 mm between 11.2 mm and 9.2 mm. Because of this difference, we cannot use a single kernel. The deviation of 10 mm regardless of the distance to the object means that the distance between the double images to be photographed becomes narrow when the distance to the object is long. It is necessary to measure the distance to the subject and create an optimal kernel.

FIG. 8 is a flowchart for explaining the operation of the imaging device 1. FIG. The number of pixels between the double images is proportional to the reciprocal of the distance from the camera 11 to the object. Also, it is necessary to prepare a kernel for each distance to the subject. An image by transmitted light that does not form a double image may deteriorate the image of the driver that is to be taken as a double image by this processing.

When the processing operation for avoiding such a problem is started, in step S1, the lighting device is turned on, the imaging device 1 captures an image of the driver, and in step S2, the lighting device is turned off, and the vehicle is driven again. A person is imaged by the imaging device 1 . In step S3, the controller 7 takes in two captured images from the imaging device 1, calculates the difference between these captured images, and removes the transparent image. As a reference, some double images are removed.

Next, in step S5, the pupil position of the driver is detected from the captured image after processing, and in step S6, the distance to the subject is calculated using the standard interocular distance E. The remaining double images are removed using the modified kernel.

As a countermeasure, the pupil position is detected and the distance from the pupil distance to the pupil position is obtained. Applies a kernel suitable for double images determined by the distance in front of the subject, removes double images, and uses the clear image to use accurate pupil position detection results in a 3D HUD or driver monitoring system. Used in DMS and augmented reality head-up display AR-HUD. The image by transmitted light is erased by subtracting the captured image with the light emitting element on/off, and then the double image is processed.

FIG. 9 is a diagram showing the difference in the amount of deviation of the double image depending on the reflection position of the windshield 25. FIG. The smaller the angle S to the windshield 25, the greater the amount of deviation of the double image, requiring different processing for the upper and lower regions of the image.

FIG. 10 is a diagram showing changes in the width of the double image due to the curvature of the windshield 25. FIG. The width of the double image also changes due to deformation called undulation in which the windshield 25 has different curvatures in the x-, y-, and z-axis directions.

FIG. 11 is a diagram showing a photographing panel on which a measurement pattern is drawn. Therefore, as shown in FIG. 11, a panel on which a plurality of parallel lines are drawn as a pattern for photographing is prepared. is obtained. By calculating the distance and the brightness ratio between the double images from this picked-up image and performing image processing on the shift in the direction of one axis, it is possible to obtain a processed image in which the double images are aligned only along one axis. If the distance and the brightness ratio between the double images are basically known in this way, a kernel for that location can be created.

If 5 to 6 kernels are obtained by such processing, the distribution of kernels for the entire processed image can be obtained by interpolation processing. By applying these kernels to the corresponding locations, it is possible to reduce the amount of computation and obtain a restored image from which double images have been removed.

According to the imaging device and image processing method according to the present disclosure, the driver's eyes can be detected with high accuracy.

The imaging device according to the present disclosure can be implemented in the following configuration (1).
(1) a windshield;
a camera configured to image at least the eyes of a driver of a mobile object through the windshield;
a controller that controls the camera;
The controller is
Transformation for transforming the captured image of the camera into a first processed image in which double images are positioned side by side in a predetermined direction based on a first point spread function representing the effect of the windshield on the captured image of the camera. and run
correcting the first processed image to a second processed image from which the double image is removed based on a second point spread function representing a feature of the double image in the first processed image;
An imaging device that performs an inverse transform of the transform on the second processed image to generate a third processed image from which the effects of the first point spread function are removed.

The image processing method according to the present disclosure can be implemented in the following configurations (2) to (6).
(2) providing a windshield, a camera configured to image at least the driver's eyes through the windshield, and a controller for controlling the camera;
by the controller,
Transformation for transforming the captured image of the camera into a first processed image in which double images are positioned side by side in a predetermined direction based on a first point spread function representing the effect of the windshield on the captured image of the camera. and run
correcting the first processed image to a second processed image from which the double image is removed based on a second point spread function representing a feature of the double image in the first processed image;
An image processing method comprising performing an inverse transform of the transform on the second processed image to produce a third processed image from which the effects of the first point spread function have been removed.

(3) In the configuration (2) above, wherein the first point spread function is determined based on the image captured by the camera after capturing an image of a measurement plate on which a plurality of point images are drawn. The described image processing method.

(4) The image processing method according to configuration (3), wherein the second point spread function is determined based on the first point spread function.

(5) Configuration (2) above, wherein the captured image is a difference image between a first image captured without illumination of the driver and a second image captured with illumination of the driver. The image processing method according to any one of (4).

(6) The image processing according to any one of the above configurations (2) to (5), wherein the first processed image is converted so that the distance in the predetermined direction between the double images is the same. Method.

Although the embodiments of the present disclosure have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications, improvements, etc. can be made without departing from the scope of the present invention. It is possible. It goes without saying that all or part of each of the above-described embodiments can be appropriately combined within a non-contradictory range.

REFERENCE SIGNS LIST 1 imaging device 2 stereoscopic image display device 3 acquisition unit 7 controller 10 moving body 12 three-dimensional projection device 25 windshield 13 driver 31 eye 31L left eye 31R right eye

Claims

a windshield; and
a camera configured to image at least the eyes of a driver of a mobile object through the windshield;
a controller that controls the camera;
The controller is
Transformation for transforming the captured image of the camera into a first processed image in which double images are positioned side by side in a predetermined direction based on a first point spread function representing the effect of the windshield on the captured image of the camera. and run
correcting the first processed image to a second processed image from which the double image is removed based on a second point spread function representing a feature of the double image in the first processed image;
An imaging device that performs an inverse transform of the transform on the second processed image to generate a third processed image from which the effects of the first point spread function are removed.
providing a windshield, a camera configured to image at least the driver's eyes through the windshield, and a controller for controlling the camera;
by the controller,
Transformation for transforming the captured image of the camera into a first processed image in which double images are positioned side by side in a predetermined direction based on a first point spread function representing the effect of the windshield on the captured image of the camera. and run
correcting the first processed image to a second processed image from which the double image is removed based on a second point spread function representing a feature of the double image in the first processed image;
An image processing method comprising performing an inverse transform of the transform on the second processed image to produce a third processed image from which the effects of the first point spread function have been removed.
3. The image processing method according to claim 2, wherein said first point spread function is determined based on an image captured by said camera after capturing an image of a measurement plate on which a plurality of point images are drawn by said camera. .
The image processing method according to claim 3, wherein the second point spread function is determined based on the first point spread function.
5. The captured image according to any one of claims 2 to 4, wherein the captured image is a difference image between a first image captured without illumination of the driver and a second image captured with illumination of the driver. 2. The image processing method according to item 1.
The image processing method according to any one of claims 2 to 5, wherein the first processed image is transformed so that the distance in the predetermined direction between the double images is the same.