DE102021123275B3 - Method and processor circuit for providing color image data, by means of which at least one object in a predetermined environment is represented in color, and motor vehicle with the processor circuit - Google Patents

Abstract

The invention relates to a method for providing color image data (24) by means of which at least one object (18) is displayed in color in a predetermined environment. The invention provides that a processor circuit (22) receives monochrome image data (16) from an image data source (13) from a predetermined ultraviolet and/or infrared frequency spectrum and uses a predetermined coloring algorithm (23) to receive the color image data (24) from the monochrome image data (16) are generated and the generated color image data (24) are provided.

Description

The invention relates to a method for providing a color image or a color video (image sequence) from color image data, by means of which at least one object in a predetermined environment is represented in color. The environment here is in particular a vehicle interior or passenger compartment of a motor vehicle. In particular, a person is depicted as an object. The invention also relates to a processor circuit for carrying out the method and a motor vehicle with such a processor circuit.

An RGB color camera (RGB—Red Green Blue) can be installed in a motor vehicle for use, for example, in social media or video telephony. These cameras are often attached to the rear-view mirror and capture the entire interior of the vehicle, although other camera positions are also conceivable.

RGB color cameras predominantly capture wavelengths of light that are also visible to the human eye. This essentially results in two disadvantages: the cameras deliver no or only a very weak signal in the dark and unpleasant lighting effects occur in strongly changing light conditions, for example when driving through an avenue or a curve.

From the U.S. 2016 065903 A1 It is known that in a motor vehicle, a traffic scene in a vehicle environment can also be captured by a camera in the non-visible frequency spectrum of the light in order to be able to recognize object properties.

From the DE 10 2019 112 449 A1 It is known to detect objects in a vehicle interior in the non-visible electromagnetic spectrum, in particular in the infrared range.

From the DE 10 046 309 C1 an image sensor is known which has both sensor elements for visible light and sensor elements for infrared light. A scene can be actively illuminated using an infrared light source.

From a scientific publication by Zhang et al. (Richard Zhang, Phillip Isola, Alexei A. Efros: "Colorful Image Colorization", 2016; http://arxiv.org/abs/1603.08511) a colorization algorithm for post-colorization of black and white images is known.

The use of the non-visible frequency range of light known from the prior art, for example the use of infrared light, does not lead to the quality of color images being able to be improved, as would be necessary for social media or video telephony, for example.

From the DE 11 2017 007 579 T5 It is known to subsequently color monochrome image data, for example image data from a night vision camera, in a motor vehicle and to use color information from color images that were recorded during the day or with appropriate lighting.

From a scientific publication by Limmer et al. (Matthias Limmer, Hendrik P.A. Lensch, "Infrared Colorization Using Deep Convolutional Neural Networks", 15th IEEE International Conference on Machine Learning and Applications, DOI 10.1109/ICMLA.2016.0019, 2016) it is known that monochrome image data from an area camera of a motor vehicle can be colored , to thereby provide a video image that shows the surroundings as in a day view even when driving at night.

The invention is based on the object of representing at least one object by means of color image data, in particular at least one object in a vehicle interior of a motor vehicle.

The object is solved by the subject matter of the independent patent claims. Advantageous developments of the invention result from the dependent patent claims, the following description and the figure.

As a solution to this problem, the invention includes a method for providing color image data, by means of which at least one object in a predetermined environment is represented in color. The environment can in particular be a vehicle interior or passenger compartment of a motor vehicle. Accordingly, the at least one object is a person or a vehicle occupant, for example. The color image data can be used, for example, to provide video telephony or a photograph, for example for a post in a social medium (social media).

In order to generate the color image data, the method comprises that a processor circuit from an image data source receives monochrome image data from a predetermined ultraviolet and/or the infrared frequency spectrum and uses a predetermined coloring algorithm or coloring algorithm to generate the color image data from the monochrome image data and to provide the generated color image data , ie the monochrome images are colored. In other words, the monochrome image data shows the at least one object in a predetermined ultraviolet and/or the infrared frequency spectrum. This can result in a representation in monochromatic grayscale values. The method is therefore based on a coloring algorithm, such as can be found in the scientific publication mentioned at the outset, for example. In other words, the monochrome image data from the non-visible frequency spectrum is interpreted as black-and-white image data or a black-and-white recording. It has been found that a coloring algorithm can also be successfully applied to such monochrome image data. The said ultraviolet frequency spectrum is taken in particular from a frequency range which (measured in air at 1000 hPa) is less than 400 nanometers in the electromagnetic wave range. The frequency spectrum of the infrared light lies in particular in a frequency range which (measured in air at 1000 hPa) corresponds to an electromagnetic wave range greater than 750 nanometers, in particular greater than 800 nanometers. The color image data provided can be displayed on a screen, for example, and/or the color image data can be sent, for example, as a video stream or color photo, ie in particular sent from the motor vehicle. The color image data contains, in particular, information on a plurality of color channels, for example RGB or YUV, per pixel (picture element). In particular, the color image data describes colors in a frequency range which (measured in air at 1000 hPa) lies in an electromagnetic wave range in a range between 400 and 750 nanometers. In particular, non-gray colors are represented by the color image data, ie in particular the colors red, green, blue, ie colors which are different from black, white and grey. Software or program code can be used as the coloring algorithm, the different predetermined color-independent image features, such as predetermined textures, shapes, areas, image areas detected in the monochrome image data (e.g. a strip at the upper edge of the image), a gray level value, an image context, one respectively assigns a predetermined color which is only signaled as a gray scale value in the monochrome image data, for example a brown tone, a red tone or generally a color with different intensity values for RGB.

The invention has the advantage that colored image information and/or object information for the visible frequency spectrum (corresponding to the waves in the wavelength range from 400 nanometers to 750 nanometers) can be determined and thus a colored image of the at least one object can be provided from monochrome image data .

The invention includes that at least some of the supporting image data is acquired at the same time as the monochrome image data, but an illumination measurement of an illumination of the object is below a predetermined threshold value in at least part of the visual frequency spectrum. The simultaneous generation of supporting image data and monochrome image data results in the advantage that a shape and/or texture can be recorded congruently in the supporting image data and monochrome image data. The monochrome image data is then accessed when the (colored) supporting image data does not have sufficient illumination, ie the illumination is below the threshold value. If the surroundings, in particular the vehicle interior, are insufficiently illuminated so that the illumination is less than the threshold value, the monochrome image data and their subsequent coloring can be accessed using the coloring algorithm. In this case, color information can be taken from the supporting image data, which may contain an underexposed image of the object, but can still contain color information. This prevents noisy color image data from being generated solely by the supporting image data and subsequent exposure amplification. Instead, the monochrome image data, which is less noisy than the supporting image data, can be used and the post-coloring can be configured or controlled using the coloring algorithm on the basis of the supporting image data.

The invention also includes developments that result in additional advantages.

An infrared camera and/or a UV-sensitive camera, for example, which can be provided in the motor vehicle, can be used as the image data source. The additional image data source can be a color camera, for example, whose detection range is oriented towards the environment, in particular towards the vehicle interior.

The coloring algorithm described at the beginning is based on the training of a machine learning model in order to provide sufficient prior knowledge (a-priori knowledge) about colors to be used for individual components or elements of an object, for example a face color, from training data, for example an image database. Developments of the invention offer the possibility of determining color information or a configuration for assigning a color even without such general prior knowledge and of using the at least one object itself for this purpose.

A further development includes that supporting image data is received from an additional image data source, which provides supporting image data of the respective object from a visual frequency spectrum, and the coloring algorithm is configured using the supporting image data in relation to a color selection of a color to be selected in each case of at least one component of the object. In this way, the visual spectrum preferably includes frequencies of visible light that correspond to a wavelength range (in air at 1000 hPa) from 400 nanometers to 700 nanometers and describe colors, in particular red, green, blue (i.e. non-white, not -grey, not black). Color information of the respective object is thus determined by the additional image data source and provided to the coloring algorithm for a color selection or a color to be selected for at least one element of the object. In other words, such situations or time intervals and/or lighting conditions are used in which supporting image data with color information from the visual spectrum about the at least one object can also be recorded.

A development here includes that at least some of the supporting image data include at least one historically stored image recording of the respective object at a time when sufficient lighting was available, namely as a lighting measurement of a lighting of the object, in particular in the surroundings, in at least a part of the visual Frequency spectrum was above a predetermined threshold. The illumination measurement can, for example, be a component of the additional image data source, ie, for example, an exposure meter of an automatic exposure system or an automatic shutter system. The support image data are thus not captured simultaneously with the associated monochrome image data or those to be colored, but rather in relation to a previous period of time or a previous point in time. The lighting measurement is used to determine a situation or a moment in which the lighting in the visual frequency spectrum or at least part of it meets a predetermined minimum brightness criterion, namely above the predetermined threshold value, so that the support image data is generated to produce a colored image of the respective Object can be generated by the additional image data source. For example, the fact that the interior lighting is active in a motor vehicle when a vehicle occupant gets in and a vehicle door is open can be used. This can be detected by the illumination measurement. For example, corresponding so-called intrinsic parameters of a camera can be evaluated, which indicate what level of illumination is currently present in the environment. Using the supporting image data, a machine learning model can be trained such that an assignment of, for example, a shape of an element of the object, for example a face of a person, and/or a texture, for example a hair texture, is assigned a color that is described or signaled in the supporting image data will. If the shape and/or texture is then detected in the monochrome image data, the color taken from the supporting image data can be assigned by the model in the coloring algorithm. Illumination can be measured in terms of luminous flux or luminance, for example. A lighting measurement from the prior art can be used.

A development includes that the coloring algorithm is implemented by a machine learning model and the model for the monochrome image data and the supporting image data has feature extractors or feature encoders that are operated independently of one another and by which feature data are generated. A linking unit for concatenating or combining the feature data is connected downstream of the feature encoders, and the color image data are generated by the model on the basis of the concatenated feature data. Feature encoders are known per se from the prior art in connection with machine learning models. The machine learning model may be provided based on an artificial neural network, for example. A feature encoder of the model then comprises a number of layers (layers) of the artificial neural network, these being inner layers, ie in particular not the output layer. The output of a feature encoder is a representation of the input data (i.e. the monochrome image data or the supporting image data), which signal features contained in the respective image data, for example contours and/or textures and/or basic shapes (e.g. circles or ovals) and/or edges or describe that were recognized or detected in the image data. In the supporting image data, the feature encoder also indicates in particular color information or color values that were recognized in the supporting image data. The feature data can be provided as a feature vector, for example. The feature data from both feature encoders (ie the feature data from the feature encoder for the supporting image data and the feature data from the feature encoder for the monochrome image data) can be combined or summarized by the linking unit. The machine learning model can then further process or link the concatenated feature data in such a way that corresponding image areas from the monochrome image (monochrome image data) and the Supporting image (supporting image data) are recognized based on corresponding features and then the color information from the supporting image data is assigned to the features from the monochrome image data, e.g. areas and/or textures and/or shapes, resulting in the coloring for the color image data. In other words, the coloring algorithm described at the outset for post-coloring black-and-white recordings can be expanded in such a way that instead of a single feature encoder for the monochrome image data, two feature encoders (one for the monochrome image data and one for the supporting image data) are operated or provided and these are operated via the linking unit be combined to concatenate the output data or feature data of the two feature encoders. This introduces a new use of a coloring algorithm based on a machine learning model.

A development includes that object data of a current pose and/or posture of the object are received from an object observation. A development includes the coloring algorithm being configured using the object data in relation to a color selection of a respective color to be selected of at least one component of the object by recognizing a respective object type of the component and selecting a respective color specified for the object type. An object observation can be provided in a motor vehicle, for example, by the occupant observation known per se, as described in the prior art described at the outset. A part of the object can be a component or an appendage of the object, such as an arm or a face. For example, the occupant observation can indicate where an arm or a face or hair of the object is located. A desired or specified color can then be assigned to the object type signaled in this way (hand, face, hair) using, for example, a look-up table or using an assignment function to create.

In the context of monochrome image data based on infrared light, it has been found that a contour or edge of a part of an object, such as a component or a limb, may be blurred or not imaged at all. For example, it can happen that a hand placed on a piece of clothing cannot be distinguished or recognized in the monochrome image data from the surrounding textile on which the hand is lying. A development includes that a contour missing in the monochrome image data and/or at least one edge of the respective component is determined on the basis of the object data and its course is signaled. Based on an occupant observation, object data can be provided which signal the course of a contour and/or an edge, so that the component can then be colored according to an object type, for example, i.e. the hand lying on the textile is reproduced in skin color in the color image data, while for example the textile on which the hand rests can be reproduced in the color of the textile in the color image data. The object observation can be based, for example, on a 3D object detection, for example a stereo camera and/or a LiDAR and/or a microradar.

A development includes the monochrome image data and the supporting image data being received from the same image sensor. For this purpose, the image sensor can be designed in the manner described at the outset. Combining image pixels or sensor elements on an image sensor that generate both monochrome image data and supporting image data results in the advantage that an interlaced pixel pattern for the monochrome image data and the supporting image data allows the perspective from which the at least one object is represented in the monochrome image data and the supporting image data is, is identical. A development includes the monochrome image data and the supporting image data being received from different image sensors. This results in the advantage that two suitable image sensors can be provided independently of one another with little technical effort.

A development includes that, in order to match the supporting image data to the monochrome image data, their image resolutions are matched by means of resolution scaling and/or their image perspectives are matched by means of perspective distortion. The adjustment of a resolution scaling, i.e. the adjustment of the image resolution (pixel in x and pixel in y) is preferably carried out towards the larger resolution number. A corresponding scaling algorithm cannot be found in the prior art. Adjusting the resolution has the advantage that the coloring algorithm can reliably assign image areas or pixel areas from the monochrome image data and the supporting image data to one another. Correction of a perspective distortion, that is, an adjustment, can be configured in advance, for example, such that the installation positions of the image data source and the supporting image data source can be recognized or determined in advance and the perspective distortion can be configured according to the positions. A perspective distortion can occur, for example based on an algorithm for shearing and/or so-called image warping.

A development includes that the object is illuminated by means of a self-illuminating light source of the ultraviolet and/or the infrared frequency spectrum. The active illumination can ensure that light is present in the frequency spectrum required for the monochrome image data or for generating the monochrome image data. Nevertheless, a person is not dazzled because the light source does not have to emit visible light.

A development includes the color image data being output as a video stream. In other words, a video camera is preferably used as the image data source. As a result, video telephony, for example, can be implemented in the environment, in particular in a vehicle interior. The color image data may additionally or alternatively include a still image.

As a further solution, the invention includes a processor circuit for providing color image data, the processor circuit being set up to carry out a further development of the method according to the invention. The processor circuit can have a data processing device or a processor device that is set up to carry out the development of the method according to the invention. For this purpose, the processor circuit can have at least one microprocessor and/or at least one microcontroller and/or at least one FPGA (Field Programmable Gate Array) and/or at least one DSP (Digital Signal Processor). Furthermore, the processor circuit can have program code which is set up to carry out the further development of the method according to the invention when executed by the processor circuit. The program code can be stored in a data memory of the processor circuit. The processor circuit can be provided, for example, by means of a control device or a combination of a number of control devices. The processor circuit is intended for operation in a motor vehicle.

As a further solution, the invention includes a motor vehicle with at least one camera as the image data source and with a development of the processor circuit according to the invention, which is coupled to the at least one camera. The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle. In the motor vehicle, for example, at least one image sensor can be provided in the manner described as an image data source and as a supporting image data source. The respective image sensor can be provided, for example, by means of a camera or multiple cameras.

The invention also includes combinations of the features of the developments described. The invention also includes implementations that each have a combination of the features of several of the developments described, provided that the developments were not described as mutually exclusive.

• Fig. eine schematische Darstellung einer Ausführungsform des erfindungsgemäßen Kraftfahrzeugs mit einer erfindungsgemäßen Prozessorschaltung die eine Ausführungsform des erfindungsgemäßen Verfahrens durchführen kann.

Exemplary embodiments of the invention are described below. The only figure shows this:

• 2 shows a schematic representation of an embodiment of the motor vehicle according to the invention with a processor circuit according to the invention which can carry out an embodiment of the method according to the invention.

The exemplary embodiments explained below are preferred developments of the invention. In the exemplary embodiments, the described components of the developments each represent individual features of the invention which are to be considered independently of one another and which also develop the invention independently of one another. The disclosure should therefore also include combinations of the features of the developments other than those shown. Furthermore, the developments described can also be supplemented by further features of the invention that have already been described.

In the figures, the same reference symbols designate elements with the same function.

The only figure shows a motor vehicle 10, which can be a motor vehicle, in particular a passenger car or truck. A person 11 can be in the motor vehicle 10 as a driver, for example. This is symbolized in the figure by a steering wheel 12 that is guided by person 11 . An image data source 13 and additionally a supporting image data source 14 can be provided in the motor vehicle 10 . The image data source 13 can be an infrared camera, for example, such as can be part of an occupant observation or driver attention observation in a manner known per se. The infrared camera 15, that is to say the image data source 13 in general, can generate monochrome image data 16 which, for example, can lie in a light wavelength range greater than 750 nanometers, in particular greater than 800 nanometers. Using the image data source 13, by aligning its detection area 17 as an object 18 For example, the person 11 can be detected. For illuminating or illuminating the object 18, a light source 19 can generate or radiate the light in the color spectrum that is provided for the image sensor of the image data source 13. In addition, the supporting image data source 14 can generate supporting image data 20 . The supporting image data source 14 can have a detection area 21 which can likewise be aligned with the object 18, ie the person 11, for example. the supporting image data source 14 can be a video camera, for example. It is in particular a color image camera.

The monochrome image data 16 and the supporting image data 20 can be received by a processor circuit 22, which can be implemented by a control unit of the motor vehicle 10, for example. A coloring algorithm 23 can be operated in the processor circuit 22, which uses the monochrome image data 16 to generate color image data 24. For this purpose, for example, a machine learning model 25 can be provided by the coloring algorithm 23 . The color image data 24 can be sent as a video stream 28, for example by means of a communication unit 26, for example a mobile radio module and/or a WiFi radio module, via a radio connection 27, so that the person 11 can conduct a video conference or a video conversation, for example. However, this is only an exemplary application for the coloring algorithm 23. The monochrome image data 16 can depict the object 18, for example the person 11, using different gray levels, as is known per se for infrared images. Components or parts of the object 18 can be post-colored by the coloring algorithm 23 . For this purpose, the machine learning model can have a feature extractor or feature encoder 30 for the monochrome image data 16 , ie for individual images which are described by the monochrome image data 16 . A feature extractor or feature encoder 31 for the supporting image data 20 may be provided. Features can therefore be extracted from individual images, which are represented by the supporting image data 20 . The extracted features can be linked by a linking unit 32, ie feature data 34 from feature extractor 30 and feature data 34 from feature extractor 31 can be chained or arranged contiguously or combined to form a feature vector. Color information 35 can be contained in supporting image data 20 and thus also in feature data 34 . For this purpose, the supporting image data 20 can be recorded, for example, when there is sufficient lighting, i.e. represent historical supporting image data, and/or color information can be determined simultaneously with the monochrome image data 16, for example by amplifying the image sensor signal of the supporting image data, in the event of insufficient lighting below a threshold value. The linked feature data 33, 34 can be linked by post-coloring 36 of the model 25, so that contours and/or shapes and/or textures, as represented by the monochrome image data 16 in the features according to the feature data 34, with the color information 35 from the Feature data 34 of the supporting image data 20 are colored or colored or supplemented with color information. The resulting color image data 24 can then be provided or sent in the manner described.

In addition to common approaches for coloring images, which only use prior general knowledge during the training of the coloring algorithm, knowledge about the current scene of the environment and/or at least the object is also added here during the inference by the coloring algorithm. Deviating from the approach of Zhang et al. (Richard Zhang, Phillip Isola, Alexei A. Efros: "Colorful Image Colorization", 2016; http://arxiv.org/abs/1603.08511) two encoders of the same scheme are used, one a monochrome image of the monochrome image data and one a multi-channel Image of the support image data processed. The extracted properties can be concatenated in a latent space. Depending on the training data, this gives the model the opportunity to use color information, either from the past or current color information, for example when the lighting is poor.

In addition to the waves visible to humans, other wavelengths can be recorded and conclusions can be drawn about the scenery and its appearance in the visible light range.

A good image or video signal is generated in the visible frequency range even in poor or changing light conditions.

• • Es kann entweder derselbe Bildsensor verwendet werden um alle Wellenbereiche zu erfassen oder es können mehrere unterschiedliche Sensoren verwendet werden.
• • Werden mehrere separate Bilder erfasst, können beide Bilder dieselbe oder unterschiedliche Auflösungen haben.
• • Optional kann eine aktive Beleuchtung für bestimmte oder alle Wellenlängen verwendet werden.

Either only signals from the non-visible frequency range or a combination of visible and non-visible frequency ranges can be used. There are the following differences in recording:

• • Either the same image sensor can be used to capture all wavebands or several different sensors can be used.
• • When capturing multiple separate images, both images may have the same or different resolutions.
• • Optionally, active illumination can be used for specific or all wavelengths.

A coloring algorithm uses the data from the non-visible frequency range or combines the data from all frequency ranges to generate an optimized image in the visible frequency range. This coloring algorithm can use knowledge from the past in addition to the current data.

Overall, the examples show how the video quality can be improved in a motor vehicle by using non-visible frequency ranges.

Claims (10)

Method for providing color image data (24), through which at least one object (18) is displayed in color in a predetermined environment, wherein monochrome image data (16) showing the at least one object in a predetermined ultraviolet and/or the infrared frequency spectrum are received by a processor circuit (22) from an image data source (13) and the color image data (24) are output by means of a predetermined coloring algorithm (23). the monochrome image data (16) are generated and the generated color image data (24) are provided, in which from an additional image data source (14), which provides supporting image data (20) of the respective object (18) from a visual frequency spectrum, supporting image data (20) are received and the coloring algorithm (23) uses the supporting image data (20) in relation to a color selection of a respective to selected color of at least one component of the object (18), at least some of the supporting image data (20) being recorded at the same time as the monochrome image data (16), but with an illumination measurement of an illumination of the object (18) in at least part of the visual frequency spectrum lies at a predetermined threshold value, which signals that the colored supporting image data does not have sufficient illumination, and color information is taken from the supporting image data, which although an underexposed image of the object still contains color information, the color information being determined by amplifying the image sensor signal of the supporting image data to be mediated. procedure after claim 1 , wherein at least some of the supporting image data (20) comprise at least one historical, stored image recording of the object (18) at a point in time when an illumination measurement of an illumination of the object (18), in particular in the environment, in at least part of the visual frequency spectrum above a predetermined threshold was. Method according to one of the preceding claims, wherein the coloring algorithm (23) is implemented by a machine learning model (25) and the model (25) for the monochrome image data (16) and the supporting image data (20) each have independently operated feature encoders (30) has, through which feature data (33, 34) of the at least one object is generated, and the feature encoders (30) are followed by a linking unit (32) for concatenating the feature data (33, 34) and using the concatenated feature data (33, 34) the Color image data (24) are generated by the model (25). Method according to one of the preceding claims, in which the monochrome image data (16) on the one hand and the supporting image data (20) on the other hand are received from the same image sensor or from different image sensors. Method according to one of the preceding claims, in which, in order to match the supporting image data (20) to the monochrome image data (16), their image resolutions are matched by means of resolution scaling and/or their image perspectives are matched by means of perspective distortion. Method according to one of the preceding claims, object data on a current pose and/or attitude of the object (18) being received from an object observation and a) the coloring algorithm (23) is configured using the object data in relation to a color selection of a color to be selected in each case of at least one component of the object (18), the respective component of the object being a component or a limb of the object, in particular an arm or a face, and the color is selected by recognizing a respective object type of the component and selecting a respective color predetermined for the object type by looking up the signaled object type (hand, face, hair) using a look-up table or using a Assignment function the object type is assigned a predetermined color, and thus color information for the monochrome image data is provided in order to generate the color image data, and/or b) a contour missing in the monochrome image data (16) and/or at least one edge of the respective component is determined on the basis of the object data and its course is signaled. Method according to one of the preceding claims, in which the object (18) is illuminated by means of a self-illuminating light source (19) of the ultraviolet and/or the infrared frequency spectrum. A method as claimed in any preceding claim, wherein the color image data (24) is output as a video stream (28). Processor circuit (22) for providing color image data (24), the processor circuit (22) being set up to carry out a method according to one of the preceding claims. Motor vehicle (10) with at least one camera as an image data source (13) and with a processor circuit (22) which is coupled to the at least one camera and which, according to claim 9 is trained.

Images