DE202013012088U1 - Device for image interlacing without memory buffer - Google Patents

Abstract

Apparatus for generating a target image, comprising:
a sensor unit (110) for providing a plurality of microimages, each of the microimages respectively imaging a field of view of a plurality of fields of view, and wherein each field of view of the plurality of fields of view overlaps at least one other field of view of the plurality of fields of view, and
a target image providing unit (120),
wherein the sensor unit (110) is adapted to provide a first portion of a first of the microimages and a first portion of a second of the microimages, and wherein the target image providing unit (120) is configured based on the first portion of the first of the microimages provide a first portion of the target image based on the first portion of the second of the microimages;
wherein the sensor unit (110) is adapted to provide, after providing the first portion of the target image, a second portion of the first of the microimages and a second portion of the second of the microimages, and wherein the target image providing unit (120) is configured based thereon to provide the second subregion of the first of the microimages and based on the second subregion of the second of the microimages to provide a second subregion of the target image, and
wherein the second portion of the first of the microimages is different from the first portion of the first of the microimages, wherein the second portion of the second of the microimages is different from the first portion of the second of the microimages, and wherein the second portion of the target image is different from the first portion Target image is different.

Description

The application relates to image interweaving, and more particularly to an image interleaving apparatus and method.

Multi-aperture imaging systems depict a scene with multiple optical channels. For example, there are systems in which each channel on an image sensor creates its own image of the scene, a microimage that does not connect to its neighbors. The complete picture of the scene is created by electronic combination of all microimages (see [1]).

It may make sense that the visual fields of the channels do not connect to each other, but overlap. In the overlap areas, a scene of multiple channels is displayed.

So will in 2a an object "A" scanned by nine channels. The circles with numbers indicate the entangled scan pattern of the nine channels. This entangled scan pattern results in a finer sample (supersampling).

2 B shows a second representation of the scenery, already from 2a is represented, in which the object of 2a is scanned by 9 channels. In 2 B each of the nine channels is represented by one of the numbers 1 to 9 (see 2 B bottom right). Each of the circles is characterized by the channel number of the channel to which the respective circle is assigned. Each of the circles in 2 B on the left indicates a point, which is scanned by the respective channel whose number carries the circle. So the 25 circles in 2 B on the left, labeled "1", indicate the 25 areas sampled by channel 1. The 25 circles labeled "2" indicate the 25 areas sampled by channel 2, etc. In 2 B top left are the resulting nine fields of view (English: "field of view") of the nine channels indicated.

By fine-tuning the channel viewing directions, the scene can be sampled finer in these overlap areas than by a single channel (supersampling).

In order for this fine scan to be effective in the reconstructed image, the microimages are not merely juxtaposed, but entangled or intertwined ("braiding").

In prior art transformations, all channels are mapped to a common contiguous pixel matrix (see 3 showing micro images on a conventional image sensor).

In order for the microimages to be available for processing, according to the prior art, this pixel matrix must first be completely read out. The resulting raw image must be cached therein for processing according to the prior art. The memory must be so large that it can completely absorb the raw image.

Finally, according to the prior art, for recombination, pixels are read from the stored raw image and copied into the target image. Each individual pixel is copied. A pixel is usually represented by a small data word, for example by a byte. Pixels that are next to each other in the target image come from different areas of the stored raw image. To assemble the target image, therefore, many small amounts of data have to be copied from widely separated memory areas.

Such prior art methods have inefficiencies. On the one hand, a lot of memory is needed, on the other hand, the existing memory bandwidth is exploited very unfavorable.

The required memory is at least twice as large as the target image (memory for the raw image plus memory for the target image.) If the raw image is larger than the target image, for example due to unused areas between the channels, the memory buffer has to be correspondingly larger in normal image processing chains Camera systems, however, are cached only a few lines of an image.

Further, current storage systems achieve full speed only when larger contiguous storage areas are read or written in one piece. This is due to the word width of memory systems, which is usually several bytes (32, 64 or even 128 are common) Bit wide words). On the other hand, it is due to the fact that in SDRAM, the request and transmission of a specific memory address costs time, which is added to the transmission of the actual memory content. The fastest is therefore the reading or writing of consecutive memory blocks (English: "burst transfer"). The writing and reading of individual bytes (English: "random transfer"), however, happens only with a fraction of the maximum possible memory bandwidth. For example, a storage system that can handle 1 Gbyte / s in burst mode may well break to 50 Mbytes / s or less when single bytes of random distribution are read and written. (When braiding, the distribution of the bytes to be read is not entirely random, it follows certain patterns.) Modern memory controllers can detect these patterns within certain limits and thus provide the required bytes faster ("cache preloading") one.)

The described effort for the reconstruction can not be realized within the image sensor itself. This is mainly due to the required amount of memory. Image sensors are manufactured in special CMOS processes that are optimized for the opto-electronic requirements of pixels. These processes use larger feature sizes than current CMOS processes for digital electronics (eg memory or CPUs). The space requirement of storage in this process is very high.

On the other hand, it is customary to integrate certain image processing functions in the image sensor, such as a color contrast adjustment or a JPEG compression. Many of these functions make sense to operate only on a complete, merged image. The image reconstruction must therefore take place before further processing. But if the image reconstruction takes place outside of the image sensor, the further processing must be outsourced. The integration of image processing in the image sensor is therefore not possible for most functions.

All the disadvantages listed increase the cost of the overall system. Since microscopic multi-aperture imaging systems are particularly advantageous for cost-sensitive consumer applications (cameras in mobile devices) and should reduce costs here through innovative optical production and integration, this is counterproductive.

In addition to the described method of copying pixels, there are also more complex methods that account for sub-pixel shifts of pixels, such as those caused by geometric distortion or parallax. For a good reconstruction then simple copying of pixels is not enough; instead each pixel in the target image is calculated by interpolating several pixels from the raw image. Such a method describes [2].

In a further variant, although a sensor is used with a split image field; Nevertheless, all image fields must first be read out and buffered in order to reconstruct an overall image. The memory requirement does not change as a result.

The object of the present invention is to provide improved concepts for image interlacing.

An apparatus according to claim 1 is provided.

A device for generating a target image is provided. The apparatus comprises a sensor unit for providing a plurality of microimages, wherein each of the microimages respectively images a field of view of a plurality of fields of view, and wherein each field of view of the plurality of fields of view overlaps at least one other field of view of the plurality of fields of view. Furthermore, the device comprises a target image providing unit. The sensor unit is configured to provide a first subarea of a first of the microimages and a first subarea of a second of the microimages. The target image providing unit is configured to generate a first partial area of the target image based on the first partial area of the first of the microimages and based on the first partial area of the second of the microimages. Furthermore, the sensor unit is set up to provide a second subarea of the first of the microimages and a second subarea of the second of the microimages after provision of the first subarea of the target image. The target image providing unit is configured to provide a second partial region of the target image based on the second partial region of the first of the microimages and based on the second partial region of the second of the microimages.

In embodiments, the camera system is a multi-aperture system. The device (the image sensor) has a split image field. The device outputs a complete, reconstructed image. Furthermore, the device does not have large memory structures.

In one embodiment, the sensor unit may be configured to provide a first subregion of three or more of the microimages, wherein the target image providing unit may be configured to allocate the first subregion of the target image based on the respective first subregion of the three or more microimages produce. In this case, the sensor unit may be configured, after the provision of the first subarea of the target image, to read in each case a second subarea of the three or more of the microimages, wherein the target image providing unit may be arranged based on the respectively second subarea of the three or more of the Micro images to create the second portion of the target image. For each of the microimages, the second subregion of this microimage is different from the first subregion of this microimage.

According to one embodiment, each of the microimages may comprise a plurality of pixels and the target image may comprise a plurality of pixels. In this case, the sensor unit may be configured to provide the first subarea of the first of the microimages to include one or more of the pixels of the first of the microimages, and to provide the first subarea of the second of the microimages to be one or more of the pixels of the second of the microimages. Further, the sensor unit may be configured to provide the second subregion of the first of the microimages to include one or more further pixels of the pixels of the first of the microimages, and to provide the second subregion of the second of the microimages to be one or more further Pixel of the pixels of the second of the microimages. Further, the target image providing unit may be configured to provide the first portion of the target image to include one or more pixels of the pixels of the target image and to provide the second portion of the target image to include one or more further pixels of the pixels of the target image Includes target image.

In one embodiment, the sensor unit may be configured to provide just a first pixel of the pixels of the first of the microimages as the first portion of the first of the microimages and to provide exactly one first pixel of the pixels of the second of the microimages as the first portion of the second of the microimages , In this case, the target image providing unit may be configured to generate, based on the exactly one first pixel of the first of the microimages and based on the exactly one first pixel of the second of the microimages, at least a first pixel of the target image as the first partial area of the target image. Furthermore, the sensor unit may be configured, after the provision of the exactly one first pixel of the target image, to provide exactly a second pixel of the first of the microimages as the second partial region of the first of the microimages, and exactly one second pixel of the second of the microimages as the second partial region of the second of the microimages. Further, the target image providing unit may be configured to provide at least a second pixel of the target image as the second portion of the target image based on the exactly one second pixel of the first of the microimages and based on the exactly one second pixel of the second of the microimages.

According to another embodiment, the sensor unit may be configured to provide a first pixel row of the first of the microimages as the first portion of the first of the microimages, and to provide a first pixel row of the second of the microimages as the first portion of the second of the microimages, the first row of pixels the first of the microimages comprises two or more pixels of the first of the microimages, the first pixel row of the second of the microimages comprising two or more pixels of the second of the microimages. Further, the target image providing unit may be configured to provide, based on the first pixel row of the first of the microimages and based on the first pixel row of the second of the microimages, a first pixel row of the target image as the first portion of the target image. Furthermore, the sensor unit may be configured to provide a second pixel row of the first of the microimages as the second portion of the first of the microimages after the provision of the first pixel row of the target image, and a second pixel row of the second of the microimages as the second portion of the second of the microimages wherein the second pixel row of the first of the microimages comprises a further two or more pixels of the first of the microimages and the second pixel row of the second of the microimages comprises a further two or more pixels of the second of the microimages. Further, the target image providing unit may be configured to provide, based on the second pixel row of the first of the microimages and based on the second pixel row of the second of the microimages, a second pixel row of the target image as the second partial area of the target image.

According to an embodiment, the target image providing unit may comprise a multiplexer, wherein the multiplexer may be arranged to decide which of the pixels of the first of the microimages is selected as a first pixel of the target image in the target image, which one of the pixels of the second of the microimages second pixel of the target image in the target image is selected, and wherein the multiplexer is arranged to decide which further of the pixels of the first of the microimages as a third pixel of the target image in Target image is selected, and which more of the pixels of the second of the microimages is selected as a fourth pixel of the target image in the target image.

According to an embodiment, the sensor unit may comprise a plurality of image fields, each of the image fields comprising a plurality of optical sensors, and wherein each of the image fields may be arranged to provide one of the microimages. In the following, the terms "channel" and "image field" have the same meaning.

In one embodiment, the sensor unit may comprise a plurality of analog-to-digital converters, wherein each of the image fields may be associated with one or more of the plurality of analog-to-digital converters for reading the pixels of the microimage provided by the image field.

According to one embodiment, the image field may be configured to provide the microimage by providing the image field the pixels of the microimage. In this case, each of the image fields can be assigned a pointer which designates that pixel of the microimage which is to be provided next from the image field.

In one embodiment, each of the frames may include a clock input, a first signal input, and a second signal input, wherein the analog-to-digital converter of each of the frames may have an output. Further, each of the image fields may be arranged for when the first signal input is activated to change the pointer of that image field such that the pointer from one pixel of the microimage provided by that image field to another pixel of the microimage provided by that image field is being converted. In this case, each of the image fields can be set up when the second signal input is activated to activate the output of the analog-to-digital converter which is assigned to this image field.

In one embodiment, the sensor unit may comprise a clock network for driving the image fields and / or the analog-to-digital converters of the image fields in order to control a readout of one or more of the pixels of the microimages.

According to one embodiment, the target image providing unit may be configured to adjacent one of the pixels of the first portion of the first of the microimages as a first of the pixels of the target image and one of the pixels of the first portion of the second of the microimages as a second of the pixels of the target image in the target image to arrange each other. In this case, the target image providing unit may be arranged, after arranging the first and second pixels of the target image, one of the pixels of the second subregion of the first of the microimages as a third of the pixels of the target image and one of the pixels of the second subregion of the second of the microimages as a fourth of the pixels of the target image in the target image adjacent to each other. Here, two of the pixels of the target image in the target image are adjacent to each other when no other of the pixels of the target image is arranged between them in the target image.

In one embodiment, the target image providing unit may be configured to determine the one or more pixels of the first portion of the target image and the one or more further pixels of the second portion of the target image in response to an offset associated with two of the plurality of frames ,

According to an embodiment, the target image providing unit may be configured to determine the one or more pixels of the first portion of the target image and the one or more further pixels of the second portion of the target image depending on whether parallax is present or not.

• – Bereitstellen eines ersten Teilbereichs eines ersten der Mikrobilder und eines ersten Teilbereichs eines zweiten der Mikrobilder und Bereitstellen eines ersten Teilbereichs des Zielbildes basierend auf dem ersten Teilbereich des ersten der Mikrobilder und basierend auf dem ersten Teilbereich des zweiten der Mikrobilder.
• – Nach dem Bereitstellen des ersten Teilbereichs des Zielbildes, Bereitstellen eines zweiten Teilbereichs des ersten der Mikrobilder und eines zweiten Teilbereichs des zweiten der Mikrobilder, und Bereitstellen eines zweiten Teilbereichs des Zielbildes basierend auf dem zweiten Teilbereich des ersten der Mikrobilder und basierend auf dem zweiten Teilbereich des zweiten der Mikrobilder.

Further provided is a method of generating a target image by providing a plurality of microimages, wherein each of the microimages images each a field of view of a plurality of fields of view, and wherein each field of view of the plurality of fields of view overlaps at least one other field of view of the plurality of fields of view. The method comprises:

• Providing a first portion of a first of the microimages and a first portion of a second of the microimages and providing a first portion of the target image based on the first portion of the first of the microimages and based on the first portion of the second of the microimages.
• After providing the first subarea of the target image, providing a second subarea of the first of the microimages and a second subarea of the second of the microimages, and providing a second subarea of the target image based on the second subarea of the first of the microimages and based on the second subarea of the target image second of the microimages.

In this case, the second subregion of the first of the microimages is different from the first subregion of the first of the microimages, the second subregion of the second of the microimages is different from the first subregion of the second of the microimages, and the second subregion of the target image is from the first subregion of the target image different.

Furthermore, a computer program with a program code for carrying out the method described above is provided.

Embodiments of the invention can be found in the dependent claims.

Hereinafter, preferred embodiments of the invention will be described with reference to the drawings.

The drawings are described below:

1 shows an apparatus for generating a target image according to an embodiment,

2a shows an object scanned by 9 channels, the dots on the left show the entangled scan pattern of the 9 channels resulting in a finer scan, in a first illustration,

2 B shows a second representation of the scenery, already from 2a is represented, in which the object of 2a scanned by 9 channels,

3 shows microimages on a conventional image sensor,

4 shows a first arrangement for reading pixel values,

5 shows a second arrangement for reading pixel values,

6 represents nine microimages with 3 × 3 pixels each, and

7 represents a target image provided in accordance with an embodiment.

1 shows an apparatus for generating a target image according to an embodiment.

The device comprises a sensor unit 110 for providing a plurality of microimages, wherein each of the microimages respectively images a field of view of a plurality of fields of view, and wherein each field of view of the plurality of fields of view overlaps at least one other field of view of the plurality of fields of view.

Furthermore, the device comprises a target image providing unit 120 ,

The sensor unit 110 is configured to provide a first portion of a first of the microimages and a first portion of a second of the microimages. The destination image delivery unit 120 is configured to provide a first portion of the target image based on the first portion of the first of the microimages and based on the first portion of the second of the microimages.

Furthermore, the sensor unit 110 adapted to provide a second portion of the first of the microimages and a second portion of the second of the microimages after the provision of the first portion of the target image. The destination image delivery unit 120 is configured to provide a second portion of the target image based on the second portion of the first of the microimages and based on the second portion of the second of the microimages.

The device according to the invention is distinguished, for example, in one of the embodiments in that the device firstly provides a first subregion of the first microimage and a second subregion of the second microimage. Based on these two subareas, a first subarea of the target image is then provided. After this processing, it is no longer necessary to keep ready the first subarea of the first microimage and the first subarea of the second microimage. Extensive storage resources for the microimages are thus not required.

After the first subarea of the target image has been generated and the first subarea of the first of the microimages and the second subarea of the second of the microimages is no longer needed, the second subarea of the first of the microimages and the second subarea of the second of the microimages are provided.

Often more than two microimages are processed. In embodiments, the sensor unit 110 be configured to each provide a first portion of three or more of the microimages, wherein the target image providing unit 120 may be configured to generate the first subregion of the target image based on the respective first subregion of the three or more microimage images. In this case, the sensor unit 110 be configured to read out after the provision of the first portion of the target image in each case a second portion of the three or more of the microimages, and wherein the target image providing unit 120 may be configured to provide the second sub-area of the target image based on the respective second sub-area of the three or more of the micro-images. For each of the microimages, the second subregion of this microimage is different from the first subregion of this microimage.

The microimages and the target image will often comprise a plurality of pixels which are typically arranged in rows and columns, as is common practice. In one embodiment, each of the microimages may comprise a plurality of pixels and the target image may comprise a plurality of pixels. In this case, the sensor unit 110 be configured to provide the first portion of the first of the microimages to include one or more of the pixels of the first of the microimages, and to provide the first portion of the second of the microimages to include one or more of the pixels of the second of the microimages , Furthermore, the sensor unit 110 be configured to provide the second subregion of the first of the microimages to include one or more further pixels of the pixels of the first of the microimages, and to provide the second subregion of the second of the microimages to be one or more further pixels of the pixels of the second of the microimages. Furthermore, the target image providing unit 120 be arranged to provide the first portion of the target image to include one or more pixels of the pixels of the target image and to provide the second portion of the target image to include one or more further pixels of the pixels of the target image. In such an embodiment, therefore, only the respective pixels of the subarea of the microimages currently being considered are always kept in the memory or register, but not the pixels of the (one or more) other subregions of the microimages.

According to one embodiment, the processing may be done pixel by pixel. Thus, in such an embodiment, the sensor unit 110 be configured to provide just a first pixel of the pixels of the first of the microimages as the first portion of the first of the microimages and to provide exactly one first pixel of the pixels of the second of the microimages as the first portion of the second of the microimages. In this case, the destination image providing unit 120 be configured to provide at least a first pixel of the target image as the first portion of the target image based on the exactly one first pixel of the first of the microimages and based on the exact first pixel of the second of the microimages. Furthermore, the sensor unit 110 be arranged to provide, after the provision of the exactly one first pixel of the target image, exactly one second pixel of the first of the microimages as the second subregion of the first of the microimages and exactly one second pixel of the second of the microimages as the second subregion of the second of the microimages provide. Furthermore, the target image providing unit 120 be configured to provide at least a second pixel of the target image as the second portion of the target image based on the exactly one second pixel of the first of the microimages and based on the exactly one second pixel of the second of the microimages. In such an embodiment, the processing thus takes place pixel by pixel, and in each case only the pixel of the microimages currently being considered is kept in the memory or register, but not the pixels of the other pixels of the microimages.

According to a further embodiment, the processing can be done pixel by pixel line. So the sensor unit 110 be configured to provide a first pixel row of the first of the microimages as the first portion of the first of the microimages, and to provide a first pixel row of the second of the microimages as the first portion of the second of the microimages, wherein the first pixel row of the first of the microimages comprises two or more Pixel of the first of the microimages and the first pixel row of the second of the microimages comprises two or more pixels of the second of the microimages. Furthermore, the target image providing unit 120 be configured to provide, based on the first pixel row of the first of the microimages and the first pixel row of the second of the microimages, a first pixel row of the target image as the first portion of the target image. Furthermore, the sensor unit 110 be configured to provide, after providing the first pixel row of the target image, a second pixel row of the first of the microimages as the second portion of the first of the microimages and to provide a second pixel row of the second of the microimages as the second portion of the second of the microimages, the second pixel row of the first the microimages comprise further two or more pixels of the first of the microimages, and the second pixel row of the second of the microimages comprises further two or more pixels of the second of the microimages. Furthermore, the target image providing unit 120 be configured to generate, based on the second pixel row of the first of the microimages and based on the second pixel row of the second of the microimages, a second pixel row of the target image as the second partial region of the target image. In such an embodiment, the processing thus takes place pixel line by pixel line, and in each case only the pixel line of the microimages currently being considered is kept in the memory or register, but not the other pixel lines of the microimages.

The destination image delivery unit 120 For example, the multiplexer may be configured to decide which of the pixels of the first of the microimages is selected as a first pixel of the target image in the target image and which of the pixels of the second of the microimages as a second pixel of the target image in FIG Target image is selected, and wherein the multiplexer is arranged to decide which one of the pixels of the first of the microimages is selected as a third pixel of the target image in the target image, and which further of the pixels of the second of the microimages as a fourth pixel of the target image is selected in the destination image.

Additionally or instead, the destination image providing unit may 120 have a data bus, wherein on the data bus in each case that pixel of the first or second of the micro images is applied, which should be provided as the next pixel of the target image. Through this data bus, the destination image providing unit provides 120 then the subareas of the target image ready.

Furthermore, the target image providing unit 120 also simply consist of a line arrangement, wherein the lines of the line arrangement of the analog-to-digital converters of the sensor unit 110 lead to outputs of the device. At the outputs of the device, the pixels of the target image would then be provided by the line arrangement. In this way, the target image providing unit can also provide the pixels of the target image.

Alternatively or additionally, the destination image providing unit 120 also have a processing unit that operates a more complex processing. For example, such processing unit may post-process the pixels of the first and / or second of the microimages to obtain the pixels of the target image.

According to one embodiment, the sensor unit 110 a plurality of image fields, each of the image fields comprising a plurality of optical sensors, and wherein each of the image fields may be configured to provide one of the microimages.

In one embodiment, the sensor unit 110 a plurality of analog-to-digital converters (ADCs). wherein each of the image fields may be associated with one or more of the plurality of analog-to-digital converters for reading the pixels of the microimage provided by the image field.

According to one embodiment, the image field may be configured to provide the microimage by providing the image field the pixels of the microimage. In this case, each of the image fields can be assigned a pointer which designates that pixel of the microimage which is to be provided next from the image field.

In one embodiment, each of the frames may include a clock input, a first signal input, and a second signal input, wherein the analog-to-digital converter of each of the frames may have an output. Further, each of the image fields may be arranged for when the first signal input is activated to change the pointer of that image field such that the pointer from one pixel of the microimage provided by that image field to another pixel of the microimage provided by that image field is being converted. In this case, each of the image fields can be set up when the second signal input is activated to activate the output of the analog-to-digital converter which is assigned to this image field.

In one embodiment, the sensor unit 110 a clock network for driving the image fields and / or the analog-to-digital converter of the image fields in order to control a readout of one or more of the pixels of the microimages.

Thus, embodiments provide one of the devices described above as a customized image sensor that interleaves the microimages without the need for additional buffers.

This image sensor (ie a device described above according to an embodiment) initially has a possibility to read out the microimages independently of one another. It can be an image sensor with split image field. Each microimage then falls onto an image field (often referred to as a "pixel field") that is not connected to the adjacent image fields. For example, electronics such as line decoders, ADCs or image processing logic are located between the image fields.

Furthermore, each image field can be assigned one or more ADCs that provide the pixels of this microimage. The reading order within each microimage is the same as that of an ordinary image sensor: line by line, each line linearly pixel by pixel from one side to the other. Random access to the pixels within a frame is not necessary. The exact selection order (from bottom to top or vice versa, from right to left or vice versa) should be chosen in such a way that a possible optical inversion of the image is compensated.

Each frame has a pointer that marks the pixel within the frame to be read next. Each frame also has a clock input and two additional signal inputs, CLK_EN and OUTPUT_EN. If CLK_EN is activated, each pulse at the clock input increases the read pointer by one pixel. OUTPUT_EN activates the output of the ADC.

To control it, there is initially a clock network, which is connected to all groups. In addition, CLK_EN can be enabled or disabled individually for each group. Finally, a pixel value from any pixel group can be selected via a combination of line-by-line activation of OUTPUT_EN and column-by-column selection of the output signal (see FIG 4 and 5 ).

In this context shows 4 a first arrangement for reading pixel values.

5 shows a second arrangement for reading pixel values.

This control makes it possible to put the current pixel from any group on the output. Within a pixel group no random access is possible, but also not required. However, pixels can be skipped. These possibilities are sufficient for an intertwining of adjacent channels.

According to an embodiment, the target image providing unit 120 be arranged to arrange one of the pixels of the first portion of the first of the microimages as a first of the pixels of the target image and one of the pixels of the first portion of the second of the microimages as a second of the pixels of the target image in the target image adjacent to each other. In this case, the destination image providing unit 120 for arranging, after arranging the first and second pixels of the target image, one of the pixels of the second portion of the first of the microimages as a third of the pixels of the target image and one of the pixels of the second portion of the second of the microimages as a fourth of the pixels of the target image to arrange adjacent to each other in the target image. Here, two of the pixels of the target image in the target image are adjacent to each other when no other of the pixels of the target image is arranged between them in the target image.

In an embodiment, the target image providing unit 120 be arranged to determine the one or more pixels of the first portion of the target image and the one or more further pixels of the second portion of the target image in response to an offset associated with two of the plurality of frames.

The following provides detailed information about the wiring of the signals.

First, image entanglement without parallax correction will be explained.

9 channels with 3 × 3 pixels each are considered. These are nested so that each channel overlaps its neighbor by half a channel plus one pixel. Neighboring channels are nested, e.g. B. Pixels are used by neighbors in an alternating pattern.

Pixels on edges where no neighbors can be found are truncated.

In 6 are each designated 3 × 3 pixels of the nine microimages.

In 7 the pixels of the resulting target image are designated, the designations of the pixels of the target image indicating to which pixel of the microimages the respective pixel of the target image is due.

In the following, a simplified algorithm for image interlacing without parallax correction is explained.

It is assumed that each channel (each frame / "pixel field") can read pixels independently of the other channels. The reading of the pixels starts in the top left corner of each channel. Under a channel, an image field can be understood.

The reading direction starts with the top line first, from left to right, and so on, line by line (see the numbering above).

The reading of the pixels in each channel is only one-way, in one-pixel increments. No jumps or steps back needed.

A multiplexer unit decides which channel is read out for a particular output pixel.

For each output pixel, the multiplexer selects a particular channel and reads a pixel from it. The channel read is then shifted by one pixel (for example, by activating the corresponding clock signal).

• – Beginne mit dem Kanal (mit dem Bildfeld bzw. dem von dem Bildfeld bereitgestellten Mikrobild) in der oberen linken Ecke
• – Wähle alternierend Pixel von einem Kanal und seinem rechten Nachbarn aus
• – Wenn das Ende einer Zeile eines Kanals erreicht wird, gehe zum nächsten Kanal über (zum Beispiel: gehe zum nächsten derzeit nicht betrachteten Kanal dieser Zeile von Kanälen über; z. B. werden A und B betrachtet, und ist das Ende einer Zeile von A erreicht, gehe zu C über)
• – Wenn der letzte Kanal in einer Zeile von Kanälen erreicht ist, wähle den ersten Kanal der nächsten Zeile von Kanälen aus
• – Alterniere zwischen den Kanalreihen entsprechend
• – Wenn das Ende der Kanäle erreicht ist, vertausche die Zeilen, um die nächste Zeile zu erhalten

The channel selection follows the following pattern:

• - Start with the channel (with the image field or the microimage provided by the image field) in the upper left corner
• - Alternately select pixels from a channel and its right neighbor
• When the end of a line of a channel is reached, proceed to the next channel (for example: go to the next channel of this line of channels not currently considered, eg A and B are considered, and is the end of a line of A reaches, go to C)
• - When the last channel in a row of channels is reached, select the first channel of the next line of channels
• - Alternate between the channel rows accordingly
• - When the end of the channels is reached, swap the lines to get the next line

The handling of edge areas corresponding to the channels at the edge areas of the field requires special attention. Of these channels, only half (B, D, F, H) or only a quarter (A, C, G, I) of the pixels are used. Therefore, when reading the pixels (eg in a computer program, such as a C program), parts of the channels that are not used are to be omitted. To do this, move the pixel readout without actually reading the corresponding pixels.

For example, in a program, the pointer to the channel memory block is increased by half a line. In the image sensor, the corresponding channel (image field) has to receive the corresponding number of clock pulses.

Alternatively, corner channels can be cropped, removing unneeded pixels. In this case, no special treatment of the corner channels is required.

• – Beginne mit dem ersten Kanal, Kanal A; Der Zeiger für das Auslesen steht bei Pixel 5
• – Lese Pixel von Kanal A. Dieses ist A5. Schreibe das Pixel in den Ausgabestrom
• – Gehe einen Kanal vorwärts zu Kanal B; Der Zeiger für das Auslesen steht bei Pixel 4
• – Lese Pixel von Kanal B. Dieses ist B4. Schreibe das Pixel in den Ausgabestrom
• – Gehe einen Kanal zurück zu Kanal A; Der Zeiger für das Auslesen steht bei Pixel 6
• – Lese Pixel von Kanal A. Dieses ist A6. Schreibe das Pixel in den Ausgabestrom
• – Gehe einen Kanal vorwärts zu Kanal B; Der Zeiger für das Auslesen steht bei Pixel 5
• – Lese Pixel von Kanal B. Dieses ist B5. Schreibe das Pixel in den Ausgabestrom
• – Kanal A ist am Ende der Zeile, also wähle stattdessen Kanal C; Der Zeiger für das Auslesen steht bei Pixel 4
• – Lese Pixel von Kanal C. Dieses ist C4. Schreibe das Pixel in den Ausgabestrom ... und so weiter.

Again referring to 6 and 7 the procedure is as follows:

• - Start with the first channel, channel A; The pointer for reading is at pixel 5
• - Read pixels from channel A. This is A5. Write the pixel in the output stream
• - go one channel forward to channel B; The pointer for reading is at pixel 4
• - Read pixels from channel B. This is B4. Write the pixel in the output stream
• - Go back one channel to channel A; The pointer for reading is at pixel 6
• - Read pixels from channel A. This is A6. Write the pixel in the output stream
• - go one channel forward to channel B; The pointer for reading is at pixel 5
• - Read pixels from channel B. This is B5. Write the pixel in the output stream
• - Channel A is at the end of the line, so choose channel C instead; The pointer for reading is at pixel 4
• - Read pixels from channel C. This is C4. Write the pixel in the output stream ... and so on.

The following explains an algorithm with parallax correction.

The method described so far assumes that the arrangement of the pixels of the picture fields is invariable (scanning pattern in FIG 2a and 2 B ). In fact, this is only the case when the objects in the scene are far away from the recording system. Back the objects in a distance that is no longer large compared to the distance of the channels, the scanning move against each other. This effect is also called parallax.

This shift can be compensated with the described arrangement without changing the interconnection. According to one embodiment, only a detuning of the readout pointers of the channels relative to one another is necessary for this purpose. Details are described below.

Parallax is an effect that increases as the scene approaches the camera.

According to an embodiment, the target image providing unit 120 be configured to determine the one or more pixels of the first portion of the target image and the one or more further pixels of the second portion of the target image depending on whether a parallax is present or not.

The basic concept and the readout algorithm remain the same. It changes the offset between the channels (the offset between the image areas, English: offset). Without parallax, the offset is half a channel, so channel A starts in the middle (A5) and channel B starts at the left edge (B4). With increasing parallax, the offset becomes larger, which clearly corresponds to a displacement of the channels to the outside.

In the example described, there are now distances between the channels, so that for certain output pixels (pixels of the target image) there are no input pixels (pixels of one of the microimages). Since this is undesirable, it makes sense to provide additional pixels at the edge of the image field in each image field. These are not used if there is no parallax, but are used when parallax is present. For example, the image fields at each of the edges can be extended by an additional line. This can be read out to be used in the case of parallax. To compensate even larger parallaxes, more lines are needed at the edges.

• – Die Kanalgröße für den ausgewählten Objektabstand aus einer Tabelle auslesen, beispielsweise aus der nachfolgenden Tabelle, die den Zusammenhang zwischen dem Abstand zwischen Szenerie und Kamera, der Kanalgröße und dem Parallaxen-Versatz p erläutert:

Abstand d Parallaxen-Versatz p Kanalgröße 50 cm–unendlich 0 3×3 25–50 cm 1 5×5 12–25 cm 2 7×7

• – Nur den mittleren Bereich mit der ermittelten Größe von jedem Kanal verwenden. Um dies zu tun, muss der Algorithmus überwachen, wo er sich in jedem Kanal – bezogen auf Zeile und Spalte – befindet. Ein einfacher Zähler reicht nicht mehr aus.
• – Bei bekannter Position innerhalb eines Kanals sind dann bestimmte Pixel an den Kanten in Abhängigkeit von der Kanalgröße zu übergehen.

The parallax-modified algorithm must therefore:

• - Read the channel size for the selected object distance from a table, for example from the table below, which explains the relationship between the distance between scenery and camera, the channel size and the parallax offset p:

Distance d Parallax offset p channel size 50 cm infinity 0 3 x 3 25-50 cm 1 5 x 5 12-25 cm 2 7 x 7

• - Use only the middle range with the determined size of each channel. To do this, the algorithm must monitor where it is in each channel - in terms of row and column. A simple counter is no longer enough.
• If the position within a channel is known, then certain pixels at the edges must be passed depending on the channel size.

According to the embodiments provided here, no memory is necessary for the interlacing, but only the said arrangement and interconnection. In addition to the memory itself eliminates the memory controller. The space requirement is consequently much lower. It also eliminates the power requirements for the refresh of the RAM (with the necessary amounts of memory typically dynamic RAM) and the Memory transfer. If the prior art entanglement has been carried out in an external CPU, this is relieved, which also saves power and memory bandwidth available for other tasks.

Furthermore, according to the above embodiments, the interlacing can now be performed completely in the image sensor. This allows further operations directly in the image sensor. This change also saves space, especially in split-field sensors. In these sensors, gaps between image fields can now be filled with logic.

Any measures that reduce the footprint of an integrated circuit significantly reduce unit costs. Reducing power consumption on mobile devices increases battery life and reduces waste heat.

The solutions described can be used inter alia in devices (image sensors) with a split image field. In principle, these image sensors can be used wherever conventional image sensors are used, especially in applications where low construction height is important. This is the case for example in cameras in entertainment and communication electronics (mobile phones, laptops, tablets) and in the material and component examination in small, cramped spaces, eg. B. slots and holes.

Although some aspects have been described in the context of a device, it will be understood that these aspects also constitute a description of the corresponding method, so that a block or a component of a device is also to be understood as a corresponding method step or as a feature of a method step. Similarly, aspects described in connection with or as a method step also represent a description of a corresponding block or detail or feature of a corresponding device. Some or all of the method steps may be performed by a hardware device (or using a hardware device). Apparatus), such as a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some or more of the most important method steps may be performed by such an apparatus.

Depending on particular implementation requirements, embodiments of the invention may be implemented in hardware or in software. The implementation may be performed using a digital storage medium such as a floppy disk, a DVD, a BluRay disc, a CD, a ROM, a PROM, an EPROM, an EEPROM or FLASH memory, a hard disk, or other magnetic or optical Memory are stored on the electronically readable control signals are stored, which can cooperate with a programmable computer system or cooperate such that the respective method is performed. Therefore, the digital storage medium can be computer readable.

Thus, some embodiments according to the invention include a data carrier having electronically readable control signals capable of interacting with a programmable computer system such that one of the methods described herein is performed.

In general, embodiments of the present invention may be implemented as a computer program product having a program code, wherein the program code is operable to perform one of the methods when the computer program product runs on a computer.

The program code can also be stored, for example, on a machine-readable carrier.

Other embodiments include the computer program for performing any of the methods described herein, wherein the computer program is stored on a machine-readable medium. In other words, an embodiment of the method according to the invention is thus a computer program which has a program code for performing one of the methods described herein when the computer program runs on a computer.

A further embodiment of the inventive method is thus a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program is recorded for carrying out one of the methods described herein.

A further embodiment of the method according to the invention is thus a data stream or a sequence of signals, which represent the computer program for performing one of the methods described herein. The data stream or the sequence of signals may be configured, for example, to be transferred via a data communication connection, for example via the Internet.

Another embodiment includes a processing device, such as a computer or a programmable logic device, that is configured or adapted to perform one of the methods described herein.

Another embodiment includes a computer on which the computer program is installed to perform one of the methods described herein.

Another embodiment according to the invention comprises a device or system adapted to transmit a computer program for performing at least one of the methods described herein to a receiver. The transmission can be done for example electronically or optically. The receiver may be, for example, a computer, a mobile device, a storage device or a similar device. For example, the device or system may include a file server for transmitting the computer program to the recipient.

In some embodiments, a programmable logic device (eg, a field programmable gate array, an FPGA) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, in some embodiments, the methods are performed by any hardware device. This may be a universal hardware such as a computer processor (CPU) or hardware specific to the process, such as an ASIC.

The embodiments described above are merely illustrative of the principles of the present invention. It will be understood that modifications and variations of the arrangements and details described herein will be apparent to others of ordinary skill in the art. Therefore, it is intended that the invention be limited only by the scope of the appended claims and not by the specific details presented in the description and explanation of the embodiments herein.

literature

• [1] Andreas Brückner, Jacques Duparré, Robert Leitel, Peter Dannberg, Andreas Bräuer and Andreas Tünnermann: Thin-wafer-level camera lenses inspired by insect compound eyes. Optics Express, Vol. 18, no. 24
• [2] A. Oberdorster, F. Wippermann, A. Brückner: Image pickup device and method for taking a picture, German Patent Application 10 2010 031 535 A1

Claims (15)

Apparatus for generating a target image, comprising: a sensor unit ( 110 ) for providing a plurality of microimages, wherein each of the microimages images each one field of view of a plurality of fields of view, and wherein each field of view of the plurality of fields of view overlaps at least one other field of view of the plurality of fields of view, and a target image providing unit ( 120 ), wherein the sensor unit ( 110 ) is arranged to provide a first subarea of a first of the microimages and a first subarea of a second of the microimages, and wherein the target image providing unit ( 120 ) is adapted to provide, based on the first subregion of the first of the microimages and based on the first subregion of the second of the microimages, a first subregion of the target image, wherein the sensor unit ( 110 ) is arranged to provide a second portion of the first of the microimages and a second portion of the second of the microimages after the provision of the first portion of the target image, and wherein the target image providing unit ( 120 ) is arranged to provide a second portion of the target image based on the second portion of the first of the microimages and based on the second portion of the second of the microimages wherein the second portion of the first of the microimages is different from the first portion of the first of the microimages, wherein the second portion of the second of the microimages is different from the first portion of the second of the microimages, and wherein the second portion of the target image is different from the first portion Target image is different. Apparatus according to claim 1, wherein the sensor unit ( 110 ) is arranged to respectively provide a first portion of three or more of the microimages, and wherein the target image providing unit ( 120 ) is adapted to provide, based on the respective first subarea of the three or more of the microimages, the first subarea of the target image, and wherein the sensor unit ( 110 ) is adapted to read out a second subarea of the three or more of the microimages after the provision of the first subarea of the target image, and wherein the target image providing unit ( 120 ) is adapted to provide, based on the respective second subarea of the three or more of the microimages, the second subarea of the target image, and wherein for each of the microimages the second subarea of this microimage is different from the first subarea of this microimage. Apparatus according to claim 1 or 2, wherein each of the microimages comprises a plurality of pixels, the target image comprising a plurality of pixels, the sensor unit (10). 110 ) is arranged to provide the first portion of the first of the microimages to include one or more of the pixels of the first of the microimages to provide the first portion of the second of the microimages to include one or more of the pixels of the second of the microimages to provide the second portion of the first of the microimages to include one or more further pixels of the pixels of the first of the microimages and to provide the second portion of the second of the microimages to include one or more further pixels of the pixels of the second of the microimages and wherein the target image providing unit ( 120 ) is arranged to provide the first portion of the target image to include one or more pixels of the pixels of the target image and to provide the second portion of the target image to include one or more further pixels of the pixels of the target image. Apparatus according to claim 3, wherein the sensor unit ( 110 ) is arranged to provide just a first pixel of the pixels of the first of the microimages as the first portion of the first of the microimages, and to accurately provide a first pixel of the pixels of the second of the microimages as the first portion of the second of the microimages, and wherein the target image Provisioning unit ( 120 ) is arranged to provide at least a first pixel of the target image as the first portion of the target image based on the exactly one first pixel of the first of the microimages and based on the exactly one first pixel of the second of the microimages, and wherein the sensor unit ( 110 ) is arranged to provide, after providing the exactly one first pixel of the target image, exactly one second pixel of the first of the microimages as the second portion of the first of the microimages, and exactly one second pixel of the second of the microimages as the second portion of the second of the microimages and the target image providing unit ( 120 ) is arranged to provide at least a second pixel of the target image as the second portion of the target image based on the exactly one second pixel of the first of the microimages and based on the exactly one second pixel of the second of the microimages. Apparatus according to claim 3, wherein the sensor unit ( 110 ) is arranged to provide a first pixel row of the first of the microimages as the first portion of the first of the microimages, and to provide a first pixel row of the second of the microimages as the first portion of the second of the microimages, the first row of pixels of the first of the microimages being two or three several Pixel of the first of the microimages, wherein the first pixel row of the second of the microimages comprises two or more pixels of the second of the microimages, and wherein the target image providing unit ( 120 ) is arranged to provide, based on the first pixel row of the first of the microimages and based on the first pixel row of the second of the microimages, a first pixel row of the target image as the first subregion of the target image, and wherein the sensor unit ( 110 ) is arranged to provide a second pixel row of the first of the microimages as the second portion of the first of the microimages after providing the first pixel row of the target image, and to provide a second pixel row of the second of the microimages as the second portion of the second of the microimages second pixel row of the first of the microimages comprises further two or more pixels of the first of the microimages, the second pixel row of the second of the microimages comprising further two or more pixels of the second of the microimages, and wherein the target image providing unit ( 120 ) is arranged to provide, based on the second pixel row of the first of the microimages and based on the second pixel row of the second of the microimages, a second pixel row of the target image as the second partial area of the target image. Device according to one of claims 3 to 5, wherein the target image providing unit ( 120 ) comprises a multiplexer, the multiplexer being arranged to decide which of the pixels of the first of the microimages is selected as a first pixel of the target image in the target image, which of the pixels of the second of the microimages is selected as a second pixel of the target image in the target image and wherein the multiplexer is arranged to decide which one of the pixels of the first one of the microimages is selected as a third pixel of the target image in the target image and which one of the pixels of the second of the microimages is selected as a fourth pixel of the target image in the target image , Device according to one of claims 3 to 6, wherein the target image providing unit ( 120 ) comprises a data bus for providing one or more of the pixels of the target image. Device according to one of claims 3 to 7, wherein the sensor unit ( 110 ) comprises a plurality of image fields, each of the image fields comprising a plurality of optical sensors, and wherein each of the image fields is arranged to provide one of the microimages. Apparatus according to claim 8, wherein the sensor unit ( 110 ) comprises a plurality of analog-to-digital converters and wherein each of the image fields is associated with one or more of the plurality of analog-to-digital converters for reading the pixels of the microimage provided by the image field. Device according to claim 9, wherein the image field is arranged to provide the microimage by the image field providing the pixels of the microimage, and wherein each of the image fields is assigned a pointer indicating the pixel of the microimage to be next provided by the image field. Device according to claim 10, wherein each of the picture fields has a clock input, a first signal input and a second signal input, wherein the analog-to-digital converter of each of the picture fields has an output, wherein each of the image fields is arranged, when the first signal input is activated, to change the pointer of that image field so that the pointer from one pixel of the microimage provided by that image field to another pixel of the microimage provided by that image field , is rearranged, and wherein each of the image fields is arranged to enable the second signal input to activate the output of the analog-to-digital converter associated with that image field. Device according to one of claims 9 to 11, wherein the sensor unit ( 110 ) has a clock network for driving the image fields and / or the analog-to-digital converter of the image fields in order to control a readout of one or more of the pixels of the microimages. Apparatus according to any one of claims 8 to 12, wherein the target image providing unit ( 120 ) is arranged to arrange one of the pixels of the first portion of the first of the microimages as a first of the pixels of the target image and one of the pixels of the first portion of the second of the microimages as a second of the pixels of the target image in the target image adjacent to each other, and wherein the target image Provisioning unit ( 120 ), after arranging the first and second pixels of the target image, one of the pixels of the second portion of the first of the microimages as a third of the pixels of the target image and one of the pixels of the second portion of the second of the microimages as a fourth of the pixels of the target image in the target image adjacent to each other, wherein two of the pixels of the target image in the target image are adjacent to each other when no other of the pixels of the target image is arranged between them in the target image. Apparatus according to any one of claims 8 to 13, wherein the target image providing unit ( 120 ) is arranged to determine the one or more pixels of the first portion of the target image and the one or more further pixels of the second portion of the target image in response to an offset associated with two of the plurality of frames. Apparatus according to any one of claims 8 to 14, wherein the target image providing unit ( 120 ) is arranged to determine the one or more pixels of the first portion of the target image and the one or more further pixels of the second portion of the target image depending on whether parallax is present or not.

Images