DE102023101189A1 - Image sensor matrix unit, image sensor device, method for creating an overall image of an environment, image sensor device, motor vehicle, computer program and computer-readable storage medium - Google Patents

Abstract

The invention relates to an image sensor matrix unit (16) for recording an image, wherein the image sensor matrix unit (16) comprises a matrix structure (18) rotatably mounted on a rotation axis (22) and having a plurality of image sensors (20) for detecting light (38).

Description

The invention relates to an image sensor matrix unit. The invention further relates to an image sensor device, a method for creating an overall image of an environment, an image sensor apparatus, a motor vehicle, a computer program and a computer-readable storage medium.

Cameras are increasingly being installed in motor vehicles these days. These cameras in motor vehicles can be used, for example, to monitor passengers in autonomous buses or the vehicle environment. This may require high-resolution images or multiple images at different wavelengths (e.g. infrared, black and white, etc.). One option may be to integrate several different cameras in a motor vehicle in order to collect all the necessary information. However, such a solution can be expensive depending on the number of cameras. In addition, conventional cameras often have a chip with a limited, two-dimensional sensor area that can only detect one direction of the environment and predefined wavelengths of the incident light.

The CN 105 611 128 A , the US 2004 / 0 008 992 A1 and the WO 2010 / 094 929 A1 are state of the art.

The invention has for its object to provide an improved and cheaper image sensor device, in particular for motor vehicles.

To achieve this object, the invention provides an image sensor matrix unit according to claim 1. An image sensor device, a method for creating an overall image of an environment, an image sensor apparatus, a motor vehicle, a computer program and a computer-readable storage medium are the subject of the independent claims.

Advantageous embodiments are the subject of the subclaims.

According to one aspect, the invention provides an image sensor matrix unit for recording an image, wherein the image sensor matrix unit comprises a matrix structure rotatably mounted on a rotation axis and having a plurality of image sensors for detecting light.

One advantage of the invention can be that a plurality of images can be recorded at different rotational positions of the matrix structure using the image sensor matrix unit. For example, a 360° sector of an area surrounding the image sensor matrix unit can be recorded. This allows significantly more information to be recorded than with conventional camera systems. The majority of images or the recorded information can be combined to form a high-resolution overall image or a plurality of overall images of different types.

It is preferred that the image sensors are each designed as an image pixel sensor for recording a pixel of the image.

It is preferred that the axis of rotation runs along a principal axis of inertia and/or an axis of symmetry of the matrix structure.

By selecting the axis of rotation along a main axis of inertia and/or an axis of symmetry of the matrix structure, no imbalances arise when rotating the matrix structure. This allows particularly precise images to be created.

It is preferred that the matrix structure is three-dimensional, rotationally symmetrical, spherical, cylindrical, ellipsoidal and/or cuboidal.

It is preferred that an image sensor has an image sensor type selected from a type group comprising a monochrome image sensor, a color image sensor, an infrared image sensor for detecting infrared light, a UV image sensor for detecting UV light, an X-ray image sensor for detecting X-rays and an image sensor for detecting light with a predefined wavelength or with a wavelength in a predefined wavelength range.

It is preferred that the image sensors are arranged in at least one row in the direction of the axis of rotation and/or in a direction of rotation.

With this arrangement, individual images from image sensors can be easily combined because the geometric arrangement of the image sensors can be described analytically.

It is preferred that the image sensors of the at least one row have the same image sensor type.

This arrangement makes it easier to combine individual images.

It is preferred that the plurality of image sensors are arranged in several rows in the direction of the axis of rotation and/or in the direction of rotation, wherein the image sensors of at least two rows have a different image sensor type.

Advantageously, the image sensors of at least two rows are of a different image sensor type. This allows overall images of different types to be recorded with a single rotation.

It is preferred that the image sensor matrix unit comprises a lens system that is decoupled from the matrix structure.

A decoupled lens system has the advantage that environmental sectors can be captured in different ways. For example, one environmental sector can be captured using a zoom function of the lens system and another environmental sector using a wide-angle function of the lens system.

It is preferred that the image sensor matrix unit comprises a lens system that can be coupled to the matrix structure and/or decoupled from the matrix structure.

The lens system can advantageously be decoupled and can itself be rotated in one direction and/or the corresponding opposite direction. In this way, the image sensors of different image sensor types can be combined with different lens settings.

A coupled lens system, on the other hand, has the advantage that an overall image can be created for each existing image sensor type using the same lens setting.

Advantageously, it is possible to switch between coupling and decoupling, so that a first coupled rotation can create an overall image for each existing image sensor type, for example, and a second decoupled rotation can capture an environmental sector, for example with a zoom function of the lens system.

According to a further aspect, the invention provides an image sensor device comprising an image sensor matrix unit and a rotating device for rotating the matrix structure on the axis of rotation.

The rotating device can advantageously rotate the matrix structure in both directions. This means that the matrix structure can be directed specifically to a relevant surrounding area for recording and can be pivoted several times within this area.

In the case of a decoupled or decoupleable lens system, the image sensor device can comprise a further rotating device for separately rotating the lens system (advantageously also in both directions of rotation).

1. a) Drehen der Matrixstruktur mittels der Drehvorrichtung;
2. b) Aufnehmen einer Mehrzahl von Bildern an unterschiedlichen Drehpositionen der Matrixstruktur mittels der Bildsensormatrixeinheit; und
3. c) Zusammenfügen der Mehrzahl von Bildern zu wenigstens einem Gesamtbild.

According to a further aspect, the invention provides a method for creating an overall image of an environment using an image sensor device according to claim 9, the method comprising:

1. a) rotating the matrix structure by means of the rotating device;
2. b) taking a plurality of images at different rotational positions of the matrix structure by means of the image sensor matrix unit; and
3. c) combining the plurality of images into at least one overall image.

By combining multiple images, much more information can be displayed in one overall image. For example, a high-resolution overall image can be created. Also, multiple overall images can be created at the same time.

It is preferred that the matrix structure is rotated several times and/or at a constant rotation speed, wherein the rotation is adapted to an exposure time of at least one of the plurality of image sensors.

It is preferred that the method comprises taking images of at least one environmental sector perpendicular to the axis of rotation at different rotational positions.

It is preferred that the method comprises simultaneously capturing an image of a plurality of sensor sectors perpendicular to the axis of rotation at a rotation position.

For use cases or application situations that may arise during the method and which are not explicitly described here, it may be provided that, in accordance with the method, an error message and/or a request to enter user feedback is issued and/or a default setting and/or a predetermined initial state is set.

The invention also includes further developments of the method according to the invention which have features as have already been described in connection with the further developments of the image sensor matrix unit according to the invention or the image sensor device according to the invention. For this reason, the corresponding further developments of the method according to the invention are not described again here.

According to a further aspect, the invention provides an image sensor device for creating an overall image, comprising an image sensor device according to one of the preceding embodiments and means adapted to carry out the steps of the method according to one of the preceding embodiments.

The image sensor device also belongs to the invention. The image sensor device can have a data processing device or a processor device that is set up to carry out an embodiment of the method according to the invention. For this purpose, the processor device can have at least one microprocessor and/or at least one microcontroller and/or at least one FPGA (Field Programmable Gate Array) and/or at least one DSP (Digital Signal Processor). Furthermore, the processor device can have program code that is set up to carry out the embodiment of the method according to the invention when executed by the processor device. The program code can be stored in a data memory of the processor device. The processor circuit of the processor device can, for example, have at least one circuit board and/or at least one SoC (System on Chip).

According to a further aspect, the invention provides a motor vehicle with an image sensor device or an image sensor apparatus.

The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

According to a further aspect, the invention provides a computer program comprising instructions that cause the image sensor device to carry out the steps of the method.

According to a further aspect, the invention provides a computer-readable storage medium on which the computer program is stored.

As a further solution, the invention also includes a computer-readable storage medium comprising program code that causes the image sensor device to carry out an embodiment of the method according to the invention. The storage medium can, for example, be provided at least partially as a non-volatile data memory (e.g. as a flash memory and/or as an SSD - solid state drive) and/or at least partially as a volatile data memory (e.g. as a RAM - random access memory). The storage medium can be arranged in the processor circuit in its data memory. However, the storage medium can also be operated, for example, as a so-called app store server on the Internet. A processor circuit with at least one microprocessor can be provided by the image sensor device. The program code can be provided as binary code or assembler and/or as source code of a programming language (e.g. C) and/or as a program script (e.g. Python).

The invention also includes combinations of the features of the described embodiments. The invention therefore also includes implementations that each have a combination of the features of several of the described embodiments, provided that the embodiments have not been described as mutually exclusive.

• 1 eine Ausführungsform einer Bildsensorvorrichtung;
• 2 eine Ausführungsform einer Bildsensormatrixeinheit;
• 3 Ausführungsform eines Verfahrens zum Erstellen eines Gesamtbilds einer Umgebung unter Verwendung einer Bildsensoreinrichtung; und
• 4 eine weitere Ausführungsform der Bildsensormatrixeinheit.

Embodiments of the invention are described below.

• 1 an embodiment of an image sensor device;
• 2 an embodiment of an image sensor matrix unit;
• 3 Embodiment of a method for creating an overall image of an environment using an image sensor device; and
• 4 another embodiment of the image sensor matrix unit.

The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the components of the embodiments described each represent individual features of the invention that are to be considered independently of one another and which also develop the invention independently of one another. Therefore, the disclosure should also include combinations of the features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by other features of the invention already described.

In the figures, identical reference symbols designate functionally identical elements.

1 shows an embodiment of an image sensor device 10 for creating an overall image of an environment.

The image sensor device 10 comprises an image sensor device 12 and a data processing device 14. The image sensor device 12 has an image sensor matrix unit 16 for recording an image.

2 shows an embodiment of the image sensor matrix unit 16.

The image sensor matrix unit 16 has a matrix structure 18 with a plurality of image sensors 20 for detecting light 38.

At the 2 In the embodiment shown, the matrix structure 18 is spherical. However, other shapes of the matrix structure 18 are possible in the present invention.

The matrix structure 18 is rotatably mounted on a rotation axis 22. In the present case, the rotation axis 22 runs along a main axis of inertia 24 and an axis of symmetry 26, in particular a rotational symmetry axis, of the matrix structure 18.

The image sensors 20 are arranged on the matrix structure 18 in several vertical rows 28 in the direction of the axis of rotation 22 and in several horizontal rows 30 in a direction of rotation 32. The horizontal rows 30 can completely encircle the matrix structure 18 in the direction of rotation 32.

In the present invention, the image sensors 20 are each designed as image pixel sensors 34 for recording one pixel of the image. However, the invention also includes embodiments in which the image sensors 20 are designed to record a plurality of pixels of the image.

The image sensors 20 have an image sensor type. The image sensor type is selected from a type group that includes a monochrome image sensor, a color image sensor, an infrared image sensor for detecting infrared light, a UV image sensor for detecting UV light, an X-ray image sensor for detecting X-rays and an image sensor 20 for detecting light 38 with a predefined wavelength or with a wavelength in a predefined wavelength range.

The image sensors 20 of one of the vertical rows 28 and/or one of the horizontal rows 30 can have the same image sensor type. The image sensors 20 of at least two vertical rows 28 and/or of at least two horizontal rows 30 can have a different image sensor type.

The matrix structure 18 also has a lens system 36 that focuses incident light 38 onto the image sensors 20.

The image sensor matrix unit 16 can record a sensor sector 42 perpendicular to the rotation axis 22 of the image sensor matrix unit 16 at a rotation position of the matrix structure 18. A sensor sector 42 can enclose an angle of 0° to 360° perpendicular to the rotation axis 22. Alternatively, the image sensor matrix unit 16 can simultaneously record an image of a plurality of sensor sectors 42 perpendicular to the rotation axis 22 at a rotation position of the matrix structure 18.

Reference is again made to 1 taken.

The image sensor device 12 further comprises a rotating device 40 for rotating the matrix structure 18 on the rotation axis 22.

With the described image sensor device 10, an overall image of an environment of the image sensor matrix unit 16 can be created. 3 shows an embodiment of a method for creating the overall image using the image sensor device 12.

• - Drehen der Matrixstruktur 18 mittels der Drehvorrichtung 40.

In a step S11, the method comprises:

• - Rotating the matrix structure 18 by means of the rotating device 40.

The matrix structure 18 can be rotated at a constant rotational speed. Preferably, therefore, in step S11, the rotational speed of the matrix structure 18 is adapted to an exposure time T _B of the image sensors 20. Advantageously, the matrix structure 18 can first be rotated in one direction of rotation 32 and then in an opposite direction of rotation 32.

• - Aufnehmen einer Mehrzahl von Bildern an unterschiedlichen Drehpositionen der Matrixstruktur 18.

In a step S12, the method comprises:

• - Taking a plurality of images at different rotational positions of the matrix structure 18.

In a step S13, the data processing device 14 can combine the plurality of images to form the overall image.

• Bei der Drehung der Matrixstruktur 18 kann die Bildsensormatrixeinheit 16 an einer ersten Drehposition ein Bild wenigstens eines ersten Umgebungssektors 44 aufnehmen und an einer zweiten Drehposition ein Bild wenigstens eines zweiten Umgebungssektors 46 aufnehmen. Ein Umgebungssektor 44, 46 kann einen Winkel von 0° bis 360° senkrecht zu der Drehachse 22 einschließen. Der zweite Umgebungssektor 46 kann sich mit dem ersten Umgebungssektor 44 überschneiden oder diesem entsprechen. Durch Aufnahme einer Mehrzahl von Bildern an unterschiedlichen Drehpositionen der Matrixstruktur 18 kann das Gesamtbild der Umgebung erstellt werden.

It is now 4 which shows another embodiment of the image sensor matrix unit 16:

• When the matrix structure 18 is rotated, the image sensor matrix unit 16 can record an image of at least a first environmental sector 44 at a first rotational position and record an image of at least a second environmental sector 46 at a second rotational position. An environmental sector 44, 46 can enclose an angle of 0° to 360° perpendicular to the axis of rotation 22. The second environmental sector 46 can overlap with the first environmental sector 44 or correspond to it. By recording a plurality of images at different rotational positions The overall picture of the environment can be created using the matrix structure 18.

The matrix structure 18 is rotated several times or at the constant rotation speed in the direction of rotation 32, preferably with a plurality of revolutions. As already mentioned above, the rotation is advantageously adapted to the exposure time T _B of the image sensors 20.

How 4 shows, it is also possible for the image sensor matrix unit 16 to enable a zoom function in the first surrounding sector 44, a wide-angle function in the second surrounding sector 46 and a macro function in a third surrounding sector 48. Other functions and sector arrangements are possible. In this case, the lens system 36 can be decoupled from the matrix structure 18 and adapted accordingly to the respective function in the surrounding sectors 44, 46, 48. By combining the plurality of images at different rotational positions of the matrix structure 18, a plurality of overall images can thus also be created, here for example a zoom overall image of the first surrounding sector 44, a wide-angle overall image of the second surrounding sector 46 and a macro overall image of the third surrounding sector 48.

The lens system 36 can alternatively also be connected to the matrix structure 18 and rotate with it. In this case, the lens system 36 can advantageously be adapted to the image sensor type(s) of the image sensors 20 in different areas of the matrix structure 18. In this case, by combining the plurality of images at different rotational positions of the matrix structure 18, an overall image with the same lens setting can be created, for example, for each existing image sensor type.

The invention also provides a motor vehicle (not shown) with the image sensor device 12 or the image sensor device 10. The invention further provides a computer program (not shown) comprising instructions that cause the image sensor device 10 to carry out the steps of the described method. The invention further provides a computer-readable storage medium (not shown) on which the computer program is stored.

• Die auf einer Kugel liegenden Bild-Pixel-Sensoren sind versetzt zueinander angebracht. Somit können unendlich viele Pixel zu einem Bild eingelesen werden.
• Die einzelnen Sektoren (der Kugel) können mit unterschiedlichen Sensoren, z.B. für verschiedene Lichtwellenlängen, ausgelegt werden. Somit können in einem Bild sehr viele Informationen zugleich abgelegt werden.

A solution principle of preferred embodiments of the invention can thus be summarized as follows:

• The image pixel sensors, which are located on a sphere, are offset from one another. This means that an infinite number of pixels can be read into an image.
• The individual sectors (of the sphere) can be designed with different sensors, e.g. for different wavelengths of light. This means that a large amount of information can be stored in one image at the same time.

A chip can detect several directions simultaneously. Furthermore, different sectors on the sphere can be occupied by image sensors with different functions.

One idea for preferred designs is to replace the flat (two-dimensional) chip with a rotating sphere. This can then contain different offset sensors. As the sphere rotates, new, different positions of incident light are constantly recorded.

This means that the number of pixels recorded is infinite. In addition, different light waves can be recorded in the image data at the same time. Multiple lens mechanisms can capture and record incident light from several different directions at the same time (for example, from the front and back).

The more image sensors are arranged offset on the sphere, the more information can be collected. The design means that sensors can be saved because they are mounted in a rotating manner.

It is also possible to use geometries other than a sphere. A cube, a cylinder, a barrel, etc. would be conceivable.

In a motor vehicle, it is therefore possible to look at the surroundings with a zoom on one side, into the vehicle with a wide angle on the other side and possibly with a macro on the dashboard of the vehicle, for example to read QR codes or ID cards or to scan documents.

In summary, in preferred embodiments of the invention, the image sensors can be mounted offset on a rotating surface. Different sensors for different wavelengths can look through the same lens system at the same time. Different lenses that look in different directions can use the same sensor at the same time. This means that cameras can be saved.

The invention can be integrated into all types of motor vehicles. The invention can also be installed in households and other surveillance media e.g. wherever high-resolution images are required, etc.

Overall, the examples show how a smart, high-resolution camera sensor can be provided.

List of reference symbols:

10

Image sensor device

12

Image sensor device

14

Data processing facility

16

Image sensor matrix unit

18

Matrix structure

20

Image sensor

22

Rotation axis

24

Principal axis of inertia

26

Axis of symmetry

28

Vertical row

30

Horizontal row

32

Direction of rotation

34

Image pixel sensor

36

Lens system

38

Light

40

Rotating device

42

Sensor sector

44

first surrounding sector

46

second environmental sector

48

third environmental sector

TB

Exposure time

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• CN105611128A [0003]
• US 20040008992 A1 [0003]
• WO 2010094929 A1 [0003]

Claims (15)

Image sensor matrix unit (16) for recording an image, wherein the image sensor matrix unit (16) comprises a matrix structure (18) rotatably mounted on a rotation axis (22) and having a plurality of image sensors (20) for detecting light (38). Image sensor matrix unit (16) according to Claim 1 , characterized in that the image sensors (20) are each designed as an image pixel sensor (36) for recording a pixel of the image. Image sensor matrix unit (16) according to one of the preceding claims, characterized in that the axis of rotation (22) runs along a main axis of inertia (24) and/or an axis of symmetry (26) of the matrix structure (18). Image sensor matrix unit (16) according to one of the preceding claims, characterized in that the matrix structure (18) is three-dimensional, rotationally symmetrical, spherical, cylindrical, ellipsoidal and/or cuboidal. Image sensor matrix unit (16) according to one of the preceding claims, characterized in that an image sensor (20) has an image sensor type which is selected from a type group comprising a monochrome image sensor, a color image sensor, an infrared image sensor for detecting infrared light, a UV image sensor for detecting UV light, an X-ray image sensor for detecting X-rays and an image sensor (20) for detecting light (38) with a predefined wavelength or with a wavelength in a predefined wavelength range. Image sensor matrix unit (16) according to one of the preceding claims, characterized in that the image sensors (20) are arranged in at least one row (28, 30) in the direction of the axis of rotation (22) and/or in a direction of rotation (32). Image sensor matrix unit (16) according to Claim 5 and 6 , characterized in that the image sensors (20) of the at least one row (28, 30) have the same image sensor type. Image sensor matrix unit (16) according to Claim 5 and 6 , characterized in that the plurality of image sensors (20) are arranged in several rows (28, 30) in the direction of the axis of rotation (22) and/or in the direction of rotation (32), wherein the image sensors (20) of at least two rows (20, 32) have a different image sensor type. Image sensor device (12) comprising an image sensor matrix unit (16) and a rotating device (40) for rotating the matrix structure (18) on the rotation axis (22). Method for creating an overall image of an environment using an image sensor device (12) according to Claim 9 , the method comprising: a) rotating the matrix structure (18) by means of the rotating device (40); b) recording a plurality of images at different rotational positions of the matrix structure (18) by means of the image sensor matrix unit (16); and c) combining the plurality of images to form at least one overall image. Procedure according to Claim 10 , characterized in that the matrix structure (18) is rotated several times and/or at a constant rotational speed, wherein the rotation is adapted to an exposure time (T _B ) of at least one of the plurality of image sensors (20). Image sensor device (10) for creating an overall image, comprising an image sensor device (12) according to Claim 9 and means adapted to carry out the steps of the procedure according to Claim 10 or 11 carry out. Motor vehicle with an image sensor device (12) according to Claim 9 or an image sensor device (10) according to Claim 12 . Computer program comprising instructions that cause the image sensor device (10) to Claim 12 the steps of the procedure according to Claim 10 or 11 executes. Computer-readable storage medium on which the computer program is Claim 14 is stored.

Images