WO2022117914A1

WO2022117914A1 - Occluding object detection

Info

Publication number: WO2022117914A1
Application number: PCT/FI2021/050803
Authority: WO
Inventors: Matti OKKONEN; Pekka SUOPAJÄRVI
Original assignee: Teknologian Tutkimuskeskus Vtt Oy
Priority date: 2020-12-01
Filing date: 2021-11-24
Publication date: 2022-06-09
Also published as: FI20206224A1

Abstract

According to an example aspect of the present invention, there is provided a method for occluding object detection, comprising: receiving (200) a first frame (300) of a video frame sequence comprising a set of frames, and defining (220) an occlusion area (312) in a display panel portion in the first frame occluded by an object (30), the defining of the occlusion area comprising: comparing a first pattern in the first frame to a second pattern in a reference frame, and identifying the occlusion area on the basis of difference in the patterns.

Description

OCCLUDING OBJECT DETECTION

FIELD

Various example embodiments relate to detection of occluding objects, and in particular to occluding object detection in video frame portions, such as digital video broadcasts.

BACKGROUND

Digital broadcasts, such as football match broadcasts, may include portions that need to be replaced with targeted broadcast portions, such as different language versions or advertisements. For example, it may be desirable to replace advertisements visible based on a led banner of the stadium to a set of local or targeted advertisements for a given countryspecific broadcast version.

Objects, such as players or equipment, between the camera and a display panel, may occlude parts of the panel and thus appear in the image frame portion to be replaced. However, it is often desirable to avoid removing or replacing the object appearing in the frame portion, e.g. to avoid replacing an image portion representative of a player by an advertisement.

US20010026319 discloses a system enhancing a preselected target for a broadcast of a live event, including accounting for occluding objects.

SUMMARY

The scope of protection sought for various embodiments of the invention is set out by the independent claims. Some specific embodiments are defined in the dependent claims. The embodiments/examples and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.

According to a first aspect, there is provided an apparatus, comprising at least one processor, at least one memory including computer program code, the at least one memory and the computer program code being configured to, with the at least one processor, cause the apparatus for: receiving a first frame of a video frame sequence comprising a set of frames, and defining an occlusion area of a display panel portion in the first frame occluded by an object, the display panel portion being indicative of a pattern of light sources of a display panel, wherein the defining of the occlusion area comprises: comparing a first pattern in the first frame to a second pattern in a reference frame, and identifying the occlusion area on the basis of difference in the patterns. The first pattern and the second pattern are based on the light sources of the display panel.

According to a second aspect, there is provided a method, comprising: receiving a first frame of a video frame sequence comprising a set of frames, and defining an occlusion area of a display panel portion in the first frame occluded by an object, the display panel portion being indicative of a pattern of light sources of a display panel, wherein the defining of the occlusion area comprises: comparing a first pattern in the first frame to a second pattern in a reference frame, and identifying the occlusion area on the basis of difference in the patterns. The first pattern and the second pattern are based on the light sources of the display panel.

There is also provided an apparatus comprising means for causing the apparatus at least to perform the method, or any embodiment thereof. The apparatus may comprise a receiver or receiver circuitry for receiving the first frame, a processor or processing circuitry to define to occlusion area. The processor/processing circuitry, or another processor/processing circuitry may be configured to perform at least some of the further features defined in the claims or the detailed description, such as generating the occlusion mask, modifying the frame(s) based on the occlusion mask, generate broadcast streams with differing replacement image data, etc.

According to still further aspects, there are provided a computer program, a computer-readable medium, or a non-transitory computer-readable medium, comprising code configured, when executed in a data processing apparatus, to cause the data processing apparatus to perform the method or an embodiment thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

Some example embodiments will now be described with reference to the accompanying drawings. FIGURE 1 illustrates an example system in accordance with at least some embodiments;

FIGURE 2 illustrates a method in accordance with at least some embodiments;

FIGURES 3a to 3c illustrate example video sequence frame contents and processing thereof;

FIGURE 4 illustrates application of a calibration frame;

FIGURE 5 illustrates Moire effect;

FIGURE 6 illustrates detection of occlusion area based on Moire affected patterns;

FIGURE 7 illustrates an image processing method according to some embodiments;

FIGURE 8 illustrates application of a multiband filter for processing frames;

FIGURES 9a and 9b illustrate a system and procedure for processing frames for different broadcast data streams; and

FIGURE 10 illustrates an example apparatus capable of supporting at least some embodiments.

EMBODIMENTS

Figure 1 illustrates an example system, comprising a processing device (PD) 10 configured for at least processing frames of a video frame sequence captured by at least one camera 20, 22. The PD 10 may be connected to further devices or systems, in some embodiments a broadcast system (BS) 50 comprising one or more devices for generating and/or transmitting at least one broadcast transmission, and/or further networks 60.

The PD 10 may be configured to process received frames to generate or facilitate generation of modified frames based on input frames of the sequence as further illustrated below. In some embodiments, the PD 10 may be configured to process the frames to prepare frames for multiple different broadcast transmissions and/or to generate control information for controlling generation of such multiple different broadcast transmissions based on the single initial or input video frame sequence. The PD may comprise a plurality of units or devices, such as (image) processing unit, a communications unit, a user interface, etc.

As further illustrated in Figure 1, the camera 20, 22 may be configured to capture (image) frames of an area comprising a display panel 40. In some embodiments, the display panel 40 is a led panel comprising a sequence of led chips. The led chips may be positioned at substantially equal distance from each other. The led chips in the set may comprise a diffusor to further scatter the light.

There may be object(s) 30, such as persons between the camera and the display panel 40. Such objects may be stationary or mobile, and may also be referred to as obstacles. The mobility state of the object 30 may change, and the object may block different portions of the display panel being visible in a frame captured by the camera 20, 22. A portion of an image frame blocked or occluded by an (occluding) object may be referred to as an occluded area or an occlusion area. There is a need for robust identification of an occluded area, referred herein also as occlusion area, of a display panel portion in the image frame to facilitate image processing for the non-occluded portions of the display panel portion.

There is now provided an improved solution addressing occluding object detection and facilitating selective image frame portion replacement, which does not require specifically configured display panel or additional devices for the panel.

Figure 2 illustrates a method in accordance with at least some embodiments. The method may be performed in an image processing apparatus or a controller thereof, such as the processing device 10 or a unit thereof.

The method comprises receiving 200 a first frame of a video frame sequence comprising a set of frames. For example, the PD 10 may receive the frame from the camera 20.

A display panel portion in the first frame may be detected 210. The display panel portion is indicative of a pattern of light sources of a display panel, such as the panel 40. The display panel portion may thus generally refer to a portion in the video frame indicative of light pattern from the physical panel display. The display panel portion may be searched for in the first frame, or it may be (implicitly) detected based on detecting a first pattern in the first frame. The first pattern may be based on and indicate a pattern in the display panel, such as one or more sequences of light sources in the display panel 40. It is to be noted that block 210 is not necessary in all embodiments, since in some implementations the display panel portion does not need to be specifically actively identified or detected after block 210. For example, image analysis area may be beforehand configured to an area corresponding to the display panel portion.

An occlusion area of the display panel portion in the first frame occluded by an object is defined 220. The occlusion area may generally represent an area of the display panel portion at or in which visibility of the light sources is prevented by the occluding object. The defining of the occlusion area comprises:

- comparing a first pattern in the first frame to a second pattern in a reference frame, and

- identifying the occlusion area on the basis of difference in the patterns.

The first pattern and the second pattern are based on the light sources of the display panel. The frames may thus be processed to detect differences in the patterns (detected in the display panel portion). There may be a further block of receiving the reference frame, preceding block 220, before or after receiving 200 the first frame. The reference frame may be received from a memory or a specific camera. When the display panel portion is first detected, the first pattern as available in the display panel portion may be compared to the second pattern in the reference frame. Threshold value(s) may be configured for the differences, and if an assessed area or pixel exceeds such threshold value, it may be identified as occluded area or pixel. It is to be noted that there are various embodiments available for processing the frames in order to perform block 220 and obtain the occlusion area based on differences in patterns of the first frame and the reference frame, some of which are illustrated further below.

Figure 3a illustrates a simple example of video sequence frame contents. A first frame 300 in a set of frames 302 may comprise a display panel portion 310 indicative of a pattern 314 of light sources. The display panel portion 310 may comprise an occlusion area 312 caused by an occluding object, such as object 30.

As illustrated in the examples of Figure 3b and 3c, an occlusion mask 320 may be generated based on the occlusion area 312 defined in block 220. The first frame, or a second frame captured (substantially at the same time as the first frame) with another camera than the first frame and having contents similar as contents of the first frame, may then be modified based in the occlusion mask. Replacement image data may thus be included in the first/second frame to replace the display panel portion in the first frame excluded by the detected occlusion area. Hence, the occluded portion of the display panel portion indicative of the object remains in an output data stream comprising the first/second frame. Depending on the applied implementation, different mask utilization options are available. For example, the occlusion mask 320 of the example of Figure 3b may (exclude the occlusion area and) be applied to prevent overlaid on the occlusion area. In another example embodiment illustrated in Figure 3c, pixels in indicated mask area may be modified but the excluded area is not modifiable.

In a further example embodiment, based on the detected differences in the first pattern of the first frame and the second pattern of the reference frame, pixels belonging to the occlusion area 312 are detected, and defined or marked as occluded. Remaining pixels of the display panel portion 310 are thus (and may be defined as) non-occluded and may be modified. The method illustrated in Figure 2 may be repeated for each consecutive frame in the frame sequence and the occlusion mask may be generated for each of these frames.

In another embodiment, the method is not performed and the occlusion mask is not generated for each frame, but the occlusion mask is applied for a selected number, two or more consecutive frames. The method may thus be repeated after the mask has been applied for the selected number of frames. Thus, amount of processing and delay caused by the occlusion area detection may be reduced. This may be advantageous e.g. in case of high frame rate video frame sequences and/or when there is not much movement in the image area. When an occlusion mask update threshold value, which may be between 3 to 5 frames, for example, is reached, the method of Figure 2 is repeated and a new occlusion mask is generated. The threshold value may be selected depending on the allowed amount of movement, and may be dynamically adapted. For example, the occlusion area needs to be updated much more frequently in case of passing bicycles than in case of golf players.

In some embodiments, the reference frame is a calibration frame. The calibration frame may be a pre-stored frame indicative of at least the panel portion without occluding objects, that is, the second pattern may correspond to complete pattern originating from the display panel. The calibration frame may be captured by the same camera as the first frame. The calibration frame may be captured during a system calibration phase and stored in a memory in or accessible by the processing device 10. The calibration frame is received from the memory for block 220 and is at least partially (at least for the display panel portion) compared to the first frame in block 220.

The system may be configured to update the calibration frame during live frame sequence processing event, to adjust to potential environment changes. This may be performed for the entire display panel portion at a time, as long as it is ensured that there are no obstacles at least in the display panel portion area when capturing the calibration frame,. In another example embodiment, the calibration frame is updated partially. This may be performed selectively only for those (sub-)portions (of the display panel portion) which are non-occluded at the time of updating. The calibration frame update may be adaptive or iterative, for example reference frame REF may be adapted over time t based on:

REF( t ) = a*REF( t -1 ) + b*IM( t ) wherein IM represents new image used to update the reference frame, a and b represent weight factor (the new image can thus be used to weight e.g. by 20%), and a + b = 1. This enables the reference frame to adapt to changing external conditions.

The occlusion area may be defined 220 on the basis of detected difference(s) between the panel display portion of the first frame and the panel display portion of the calibration frame. The occlusion area may be defined on the basis of (and to correspond to) an area in the display portion of the first frame detected to lack a pattern similar as in the second pattern in the display panel portion of the second frame, such as the calibration frame. Thus, the first frame may be analysed to detect coverage area of the first pattern, and the calibration frame may be analysed to detect coverage area of the second pattern. The coverage areas may then be compared and the differing (sub-)areas are forming or indicative of the occlusion area.

In an example embodiment, at least the coverage areas are compared pixel by pixel. A set of pixels in the first frame may be compared to a set of pixels in the calibration frame in the corresponding area. If the patterns are the same, associated pixels may be marked or classified as non-occluded. If the patterns differ, an occlusion may be marked for associated pixels, and the replacement image data is not overlaid to such occlusion-marked pixels.

Instead of comparing single pixel values, the comparison may be performed based on local properties, for example based on computing a contrast value for each pixel, or based on computing amount of local gradient or other characteristics characterizing the light source pattern and which normally does not exist in the occlusion area. The comparison may be performed based on environment of an assessed pixel, i.e. a set of pixels around the assessed pixel. For example, pixel values of an n x n window or area of pixels around the assessed pixel in the first frame and the reference/calibration frame are processed. This processing may output a first reference value for the first frame and a second reference value for the reference/calibration frame. Such reference values may be texture or contrast values, for example. These reference values are compared and (occlusion indication triggering) difference for the assessed pixel may be detected in response to difference of the reference values exceeding an associated threshold value. Such operation may be repeated for each pixel of the coverage areas and the occlusion area may be defined based on differing pixels.

Figure 4 illustrates application of a reference frame, such as a calibration frame. Image frame 400 illustrates an ideal image of a display panel portion and light source pattern, such as that of a led board, where individual light sources 402 of the pattern are visible and distinguished without optical distortion. Frame 410 is a reference frame, where the light source pattern is visible without any occlusion as a reference. Further, in this example no optical distortion appears. Frame 420 illustrates a captured frame (of a real-time frame sequence) for which occlusion detection needs to take place. An obstacle is visible 422 and occluding a part of the display panel portion/light source pattern.

Frames 410 and 420 are processed, to compare at least the light source patterns of the frames to detect occlusion area(s). The processing may comprise computing differences in the frames and the detect differences in the light source patterns of these frames. Texture values, or other type of mathematical pattern descriptors, for assessed pixels may be computed and compared, pixel-by-pixel, instead of directly applying pixel intensity values. Texture values may be computed for each pixel using a respective pixel’s local neighbourhood. Image 430 illustrates an example output of the processing, which may be utilized as the occlusion mask.

With reference also to Figure 1, in an example embodiment the first frame is received from a first camera 20 and the reference frame is received from a second camera 22. The frames are in this embodiment captured substantially at the same time. The cameras are closely positioned and may have a different angle (in horizontal plane towards the panel 40) to the display panel. The display panel portion may be detected in the first frame generated by the first camera and in the reference frame generated by the second camera. The first pattern is detected in the first frame and the second pattern is detected in the reference frame. The patterns are compared and the occlusion area is defined based on differences in the first pattern and the second pattern, for example by applying the embodiment below.

In some embodiments, the occlusion area detection 220 is based on applying Moire effect. Figure 5 illustrates two examples of Moire effect emerging on overlapping frames. Theoretically Moire effect is an interference related visual phenomena when two repeated patterns are overlaid. A pattern affected by Moire effect may be referred to as a Moire pattern. Effectively Moire pattern is an additional low frequency wave pattern appearing in front of a view. In a camera and led board case it can be seen as a sampling issue, where camera sensor and the led board are the two “overlaid patterns” that affect each other.

Configuration of the camera(s) 20, 22 used for detecting the occlusion area based on Moire patterns is adapted to enable and intensify forming of Moire pattern in the frames. This may include adapting resolution of the camera(s) close to resolution of the display panel 40. Camera parameters may be adapted such that the imaging resolution is close to distance between light sources of the panel. Alternatively, Moire pattern is introduced by resampling the images comprising the light source patterns (by reducing the resolution).

Figure 6 illustrates detection of occlusion area based on Moire patterns. Again, image frame 400 illustrates an ideal image of a display panel portion and light source pattern. Frame 610 illustrates an image of the light source pattern, where Moire pattern appears due to spatial sampling frequency/resolution. Frame 610 is without occlusion and may serve as the reference frame, captured e.g. as the calibration frame. Frame 620 illustrates the (first) frame for which occlusion detection needs to take place. Image 630 illustrates example output of processing of frames 610 and 620. Eack of Moire pattern is detected within the occlusion area 622, on the basis of which an occlusion mask may be generated for replacing the visible or non-occluded part of the display panel portion with replacement image data.

In some embodiments, the occlusion area detection is based on comparison of the first frame 620 and the reference frame 610, both comprising Moire affected area(s)/pattems. The first frame and the reference frame, or only the display panel portions identified therein, may be processed in block 220 to detect or identify difference(s) in the Moire (affected) patterns thereof. The occlusion area may be defined 220 on the basis of the differences in the Moire patterns. Since the Moire pattern is of low-frequency, the comparison of the frames may be performed pixel-by-pixel.

The apparatus performing the method of Figure 2 may be configured to identify display panel portion areas including a Moire affected pattern and/or areas lacking a Moire pattern. The occlusion area may be defined 220 on the basis of identifying lack of Moire pattern. Thus, the occlusion area may be defined to correspond to an area in the display panel portion detected to lack the Moire pattern. It is to be noted that the first frame (and particularly the display panel portion 310 thereof) and the reference frame may be processed to (directly) identify portion(s) or area(s) in the first frame lacking the Moire pattern (instead of first identifying the Moire pattern). In another example embodiment, based on the processing of the frames, (first) sub-portion(s) of the display panel portion of the first frame 620 with Moire effected are detected, and remaining (second) sub-portion(s) of the display panel portion are defined as the occlusion area.

Figure 7 illustrates an example image processing method, based on processing the first frame and the reference frame, captured by separate cameras, e.g. the cameras 20, 22 illustrated in Figure 1. The method may be applied in the apparatus configured to perform the method of Figure 2, or in another image processing apparatus or unit.

Further processing features illustrated in connection with Figure 7 may be applied at least in occlusion area detection based on the Moire affected patterns. It will be appreciated that Figure 7 is merely for illustrative purpose and at least some of the presently disclosed images processing features may be applied for various other processing configurations to define the occlusion area.

The method comprises detecting 700 differing features between the first frame and the reference frame, such as frame 702 captured by the first camera 20 and frame 704 captured by the second camera 22. This may be performed by e.g. a speeded up robust features (SURF) detector, or another type of established image descriptor. Based on the captured frames 702, 704, alignment difference may be identified in block 700 (e.g. by SURF detection) and a transformation 710 may be performed to align the frames, into aligned frames 706 and 708, respectively. Image alignment enables comparing the differences of the frames. The non-occluded areas of the panel may be detected by difference in the Moire patterns, induced by the view angle difference. Differences between the aligned frames 706 and 708 may be further analyzed to generate 720 a difference image or frame 712. For example, there should be a difference in the Moire patterns in upper portion of the dot matrix, but not on areas that are not indicative of the light source pattern, such as background and occluding objects. The occlusion area is detected 730 on the basis of the difference image. An occlusion mask 714 may then be generated 740 on the basis of the occlusion area.

Further processing steps may be performed based on a difference image generated based on the first frame and the reference frame, in an example embodiment in or before block 730 for the difference image 712 as generated in block 720. A dilation operation may be performed to the difference image. Gaussian blur may be added to the difference image. A threshold operation may be performed to the difference image. An erosion operation may be performed to the difference image. At least one largest connected component is searched for and found in the difference image. The occlusion area is determined on the basis of the at least one largest connected component.

In some embodiments, at least some of the above illustrated image processing features, such as the features illustrated above for the Moire effect analysis based embodiments, may be implemented by operational modules performed by software in one or more data processing devices, such as the PD 10. However, specifically configured hardware, comprising e.g. field-programmable gate array (FPGA) circuitry, may be applied for performing at least some of such modules.

In some embodiments, a multiband filter, such as a band-stop filter, is applied. The filter may be applied to retain frequencies in which the first and second patterns in the frames for the occlusion area detection exist, and the comparison may be performed on these retained frequencies. The filter may be applied to attenuate or remove bands actively emitted by the light sources of the display panel 40. Such filtering may be performed before performing the analysis of the frames for detecting the occlusion area, e.g. between blocks 200 and 210.

This enables to address a challenge regarding exposure control, and how to maximize contrast for dark and lit panel areas. If exposure is adjusted according to dark areas, bright areas may saturate so that the dot pattern is not recognized. If adjusted according to bright areas, pattern in dark areas may have too low contrast. This challenge may be addressed and adequate contrast for pattern detection may be achieved by the filtering of the bands actively emitted by the panel and appropriate exposure configuration based on external illumination, reflected by the panel.

As mentioned, the display panel 40 may comprise a diffusor or reflector elements to reflect and scatter external light, thus separating from other portions and background material of the panel and forming a pattern. A customized multi band-stop filter or combination of single channel band stop filters may be applied, configured to filter the bands actively emitted by the panel light sources. For example, the filtering may be configured to filter out R, G, and B bands emitted by led chips. Instead of applying multi band-stop filter, a multi band-pass filter may be applied.

Figure 8 illustrates use of multi-band filter 820, which removes (RGB) wavelength regions, one of which is indicated by reference 800, emitted by led chips of the display panel 40. The continuous wave 840 indicates led board active emission spectrum, emission as a function of emission wavelength. Reference 850 indicates emission spectrum (white led) of continuous ambient illumination. By applying the multi-band filter indicated by reference 820, wavelength areas, one which is indicated by reference 810, without active illumination remain for the further processing. These remaining wavelength areas 810 are thus indicative of the pattern of the (led) light sources display panel portion 310 and may be applied for block 210 and 220 and further features thereof.

At least some of the above-illustrated features may be applied in various video frame processing applications. One such application area includes live video broadcast transmissions that need to be modified for different purposes, such as separate national or regional transmissions, transmissions with different language versions, and/or transmissions of different broadcast classes or types. For example, advertisements or other national or regional dependent content may be included, or there may be multiple transmissions prepared of the initial video frame sequence to include different replacement and/or supplementary data according to a subscriber class.

A set of broadcast data streams may be generated on the basis of the video frame sequence. Each of the broadcast data streams may be adapted to comprise different replacement (image) data, e.g. of the type illustrated in the previous paragraph. The replacement data may be included, on the basis of the occlusion area definition on nonoccluded areas of frame display panel portions of frames in sequences. Thus, after defining 220 the occlusion area for the first frame, remaining area(s) of the display panel portion of the first frame may be replaced, overlaid or otherwise modified, depending on the applied processing method, to include replacement data specific to the respective broadcast data stream. The occlusion mask may be applied to include the replacement data. The replacement data may be selected for each broadcast data stream on the basis of broadcast data stream specific configuration data.

Figure 9a illustrates an example system, in which a third camera 900, which may be referred to as a broadcast camera, is applied to generate ‘raw’ broadcast frame sequence, on the basis of which broadcast data transmissions with customized contents are generated by a broadcast (preparation) system or unit 910. Cameras 20 and 22 are configured to capture frame sequences from the similar area (including the display panel 40) as the broadcast camera 9OO.The cameras may be positioned to image the display panel 40 from slightly different angles. These cameras 20, 22 may be arranged for occlusion area detection, and may be referred to as occlusion area detection or mask cameras, for example. The cameras are connected to an occlusion computation unit (OCU) 920. The OCU 920 may be configured to perform at least some of the features illustrated above, including the method of Fig. 3. In another example embodiment, input for the occlusion detection to the OCU 920 is obtained from a single optical axis by means of beam splitter(s). For example, mask camera(s) may be added on the same optical axis as the broadcast camera 900 by a beam splitter.

The object 30 appearing in front the panel 40 may be visible with similar color/gray level intensity as the light source pattern from a camera, whereby light source pattern is not necessary distinguished. Specific occlusion detection system may thus be preferable, configured to detect the light source pattern or the Moire pattern. Since it is generally desirable to minimize Moire effect, separate/specific occlusion detection or ‘Moire’ cameras may be used, and a further (broadcast) camera is configured to generate input frame sequence for the broadcast, with minimized Moire effect. In such example embodiment, the cameras 20, 22 referred to as Moire cameras, for example.

As further illustrated in Figure 9b, the OCU 920 may be configured to generate 930 the occlusion mask, in an example embodiment based on the Moire effect. The OCU may then transmit 940 the occlusion mask to the broadcast unit 910, which generates broadcast- specific frames (A, B, C) by including respective broadcast- specific content in a frame from the broadcast camera by using the occlusion mask. Hence, the content A, B, or C 950 is included in non-occluded portions of the display panel portions of respective broadcast- specific frames 960. By repeating this to the frames of the frame sequence from the broadcast camera, broadcast- specific frame sequences or streams 970 are generated for transmission to respective broadcast systems.

An electronic device comprising electronic circuitries may be an apparatus for realizing at least some embodiments of the present invention. The apparatus may be, or may be comprised in a computer, a laptop, a tablet computer, a server, a machine to machine, a network node or element or any other apparatus provided with communication and image processing capability. In another embodiment, the apparatus carrying out the abovedescribed functionalities is comprised in such a device, e.g. the apparatus may comprise a circuitry, such as a chip, a chipset, a microcontroller, or a combination of such circuitries in any one of the above-described devices.

Figure 10 illustrates an example apparatus capable of supporting at least some embodiments of the present invention. Illustrated is a device 1000, which may comprise, for example, the PD 10 or the OCU 920. The device may include one or more controllers configured to perform operations in accordance with at least some of the embodiments illustrated above, such as some or more of the features illustrated above in connection with Figures 2 to 9b. The device may be configured to operate as the apparatus configured to carry out the method of Figure 2.

Comprised in the device 1000 is a processor 1002, which may comprise, for example, a single- or multi-core processor wherein a single-core processor comprises one processing core and a multi-core processor comprises more than one processing core. The processor 1002 may comprise more than one processor. The processor may comprise at least one application- specific integrated circuit, ASIC. The processor may comprise at least one FPGA. The processor may be means for performing method steps in the device. The processor may be configured, at least in part by computer instructions, to perform actions.

The device 1000 may comprise memory 1004. The memory may comprise random-access memory and/or permanent memory. The memory may comprise at least one RAM chip. The memory may comprise solid-state, magnetic, optical and/or holographic memory, for example. The memory may be at least in part comprised in the processor 1002. The memory 1004 may be means for storing information. The memory may comprise computer instructions that the processor is configured to execute. When computer instructions configured to cause the processor to perform certain actions are stored in the memory, and the device in overall is configured to run under the direction of the processor using computer instructions from the memory, the processor and/or its at least one processing core may be considered to be configured to perform said certain actions. The memory may be at least in part comprised in the processor. The memory may be at least in part external to the device 1000 but accessible to the device. For example, control parameters affecting operations related to the occlusion area definition may be stored in one or more portions of the memory and used to control operation of the apparatus.

The device 1000 may comprise a transmitter 1006. The device may comprise a receiver 1008. The transmitter and the receiver may be configured to transmit and receive, respectively, information in accordance with at least one cellular or non-cellular standard. The transmitter may comprise more than one transmitter. The receiver may comprise more than one receiver. The transmitter and/or receiver may be configured to operate in accordance with global system for mobile communication, GSM, wideband code division multiple access, WCDMA, long term evolution, LTE, 3GPP new radio access technology (N-RAT), wireless local area network, WLAN, and/or Ethernet standards, for example. The device 1000 may comprise a near-field communication, NFC, transceiver 1010. The NFC transceiver may support at least one NFC technology, such as NFC, Bluetooth, Wibree or similar technologies.

The device 1000 may comprise user interface, UI, 1012. The UI may comprise at least one of a display, a keyboard, a touchscreen, a pointer device, a speaker, and a microphone, or interface(s) to at least one of such UI devices. A user may be able to operate the device via the UI, for example to configure frame processing.

The processor 1002 may be furnished with a transmitter arranged to output information from the processor, via electrical leads internal to the device 1000, to other units comprised in the device. Such a transmitter may comprise a serial bus transmitter arranged to, for example, output information via at least one electrical lead to memory 1004 for storage therein. Alternatively to a serial bus, the transmitter may comprise a parallel bus transmitter. Likewise the processor may comprise a receiver arranged to receive information in the processor, via electrical leads internal to the device 1000, from other devices comprised in the device 1000. Such a receiver may comprise a serial bus receiver arranged to, for example, receive information via at least one electrical lead from the receiver 1008 for processing in the processor. Alternatively to a serial bus, the receiver may comprise a parallel bus receiver.

The device 1000 may comprise further devices not illustrated in Figure 10. For example, the device may comprise at least one digital camera. The device may comprise a fingerprint sensor arranged to authenticate, at least in part, a user of the device. In some embodiments, the device lacks at least one device described above. For example, some devices may lack the NFC transceiver 1010.

The processor 1002, the memory 1004, the transmitter 1006, the receiver 1008, the NFC transceiver 1010, and/or the UI 1012 may be interconnected by electrical leads internal to the device 1000 in a multitude of different ways. For example, each of the aforementioned devices may be separately connected to a master bus internal to the device, to allow for the devices to exchange information. However, as the skilled person will appreciate, this is only one example and depending on the embodiment various ways of interconnecting at least two of the aforementioned devices may be selected without departing from the scope of the present invention.

It is to be understood that the embodiments of the invention disclosed are not limited to the particular structures, process steps, or materials disclosed herein, but are extended to equivalents thereof as would be recognized by those ordinarily skilled in the relevant arts. It should also be understood that terminology employed herein is used for the purpose of describing particular embodiments only and is not intended to be limiting.

Reference throughout this specification to one embodiment or an embodiment means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Where reference is made to a numerical value using a term such as, for example, about or substantially, the exact numerical value is also disclosed.

As used herein, a plurality of items, structural elements, compositional elements, and/or materials may be presented in a common list for convenience. However, these lists should be construed as though each member of the list is individually identified as a separate and unique member. Thus, no individual member of such list should be construed as a de facto equivalent of any other member of the same list solely based on their presentation in a common group without indications to the contrary. In addition, various embodiments and example of the present invention may be referred to herein along with alternatives for the various components thereof. Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the preceding description, numerous specific details are provided, such as examples of lengths, widths, shapes, etc., to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the invention.

The verbs “to comprise” and “to include” are used in this document as open limitations that neither exclude nor require the existence of also un-recited features. The features recited in depending claims are mutually freely combinable unless otherwise explicitly stated. Furthermore, it is to be understood that the use of "a" or "an", that is, a singular form, throughout this document does not exclude a plurality.

Claims

CLAIMS:

1. An apparatus, comprising at least one processor, at least one memory including computer program code, the at least one memory and the computer program code being configured to, with the at least one processor, cause the apparatus for:

- receiving (200) a first frame (300) of a video frame sequence comprising a set of frames, and

- defining (220) an occlusion area (312) of a display panel portion in the first frame occluded by an object (30), the display panel portion being indicative of a pattern of light sources of a display panel (40), wherein the defining of the occlusion area comprises: o comparing a first pattern in the first frame to a second pattern in a reference frame (410), wherein the first pattern and the second pattern are based on the light sources of the display panel, and o identifying the occlusion area on the basis of difference in the patterns.

2. The apparatus of claim 1 , wherein the at least one memory and the computer program code is configured to, with the at least one processor, further cause the apparatus for:

- generating (740) an occlusion mask (320) on the basis of the defined occlusion area (312), and

- modifying, on the basis of the occlusion mask, the first frame or a second frame captured with another camera than the first frame and having contents similar as contents of the first frame, by including replacement image data to replace the display panel portion in the first frame excluded by the defined occlusion area.

3. The apparatus of claim 1 or 2, wherein the at least one memory and the computer program code is configured to, with the at least one processor, cause the apparatus to define (220) the occlusion area on the basis of identifying lack of Moire effected pattern.

4. The apparatus of any preceding claim, wherein the at least one memory and the computer program code is configured to, with the at least one processor, further cause the apparatus for detecting the display panel portion (310) in the first frame, and comparing the first pattern in the display panel portion to the second pattern in the reference frame to define (210) the occlusion area (312) of the display panel portion. The apparatus of any preceding claim, wherein the reference frame (410) is a calibration frame comprising the display panel portion without obstacles. The apparatus of any preceding claim 1 to 4, wherein the first frame (300) is from a first camera (20) and the reference frame is from a second camera (22), wherein the cameras are closely positioned. The apparatus of claim 6, wherein the at least one memory and the computer program code is configured to, with the at least one processor, cause the apparatus for:

- detecting (700) differing features between the first frame (300) and the reference frame (410),

- aligning (710) the first frame and the reference frame,

- generating (720) a difference image (712) on the basis of the aligned first frame and the reference frame, and

- identifying (730) the occlusion area on the basis of the difference image. The apparatus of any preceding claim, further comprising a multiband filter (820) configured to attenuate bands actively emitted by light sources of the display panel (40). The apparatus of any preceding claim, wherein the at least one memory and the computer program code is configured to, with the at least one processor, further cause the apparatus for generating a set of broadcast data streams (970), wherein each of the broadcast data streams has different replacement image data included after defining (220) the occlusion area on non-occluded areas of the frame display panel portion (310), and replacement image data for each broadcast data stream is selected on the basis of broadcast data stream specific configuration data.

10. The apparatus of any preceding claim, wherein the display panel (40) is a led panel comprising a sequence of led chips.

11. A method for occluding object detection, comprising:

- defining (220) an occlusion area (312) of a display panel portion in the first frame occluded by an object (30), the display panel portion being indicative of a pattern of light sources of a display panel (40), wherein the defining of the occlusion area comprises: o comparing a first pattern in the first frame to a second pattern in a reference frame, wherein the first pattern and the second pattern are based on the light sources of the display panel, and o identifying the occlusion area on the basis of difference in the patterns.

12. The method of claim 11, further comprising:

- generating an occlusion mask (320) on the basis of the defined occlusion area (312), and

13. The method of claim 11 or 12, wherein the occlusion area is defined (220) on the basis of identifying lack of Moire effected pattern.

14. The method of any preceding claim, wherein the reference frame (410) is a calibration frame comprising the display panel portion without obstacles, or the first frame (300) is received from a first camera (20) and the reference frame is received from a second camera (22), wherein the cameras are closely positioned, the method further comprising:

- detecting (700) differing features between the first frame (300) and the reference frame (410), - aligning (710) the first frame and the reference frame,

- identifying (730) the occlusion area on the basis of the difference image.

15. A computer program comprising code for, when executed in a data processing apparatus (1000), to cause the data processing apparatus to perform the method of any one of claims 11 to 14.