EP2640063A2 - Bildverarbeitungsvorrichtung und -verfahren - Google Patents

Bildverarbeitungsvorrichtung und -verfahren Download PDF

Info

Publication number
EP2640063A2
EP2640063A2 EP13158870.9A EP13158870A EP2640063A2 EP 2640063 A2 EP2640063 A2 EP 2640063A2 EP 13158870 A EP13158870 A EP 13158870A EP 2640063 A2 EP2640063 A2 EP 2640063A2
Authority
EP
European Patent Office
Prior art keywords
eye
image
strip
panoramic image
eye strip
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP13158870.9A
Other languages
English (en)
French (fr)
Other versions
EP2640063A3 (de
EP2640063B1 (de
Inventor
Joon Hyuk Cha
Young Su Moon
Young Sun Jeon
Shi Hwa Lee
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of EP2640063A2 publication Critical patent/EP2640063A2/de
Publication of EP2640063A3 publication Critical patent/EP2640063A3/de
Application granted granted Critical
Publication of EP2640063B1 publication Critical patent/EP2640063B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/156—Mixing image signals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/128—Adjusting depth or disparity
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H04N13/211—Image signal generators using stereoscopic image cameras using a single 2D image sensor using temporal multiplexing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H04N13/218—Image signal generators using stereoscopic image cameras using a single 2D image sensor using spatial multiplexing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture

Definitions

  • One or more example embodiments relate to an apparatus and method for generating a three-dimensional (3D) panoramic image using a single camera.
  • a general panoramic image generating method may connect stereo panoramas corresponding to a left eye and a right eye, based on the assumption that a user photographs a plurality of images while horizontally panning, a camera, i.e., rotating the camera horizontally to obtain a panoramic image, without shaking the camera.
  • a camera i.e., rotating the camera horizontally to obtain a panoramic image, without shaking the camera.
  • minute hand shaking or trembling may occur by the photographer.
  • a left-eye panorama and a right-eye panorama with respect to the same point are different images having disparity and thus, quality may be degraded. In one aspect of the invention these disadvantages shall be overcome.
  • an image processing apparatus including a reference generator to generate a two-dimensional (2D) reference panoramic image using a central area of each of a plurality of input image frames, and a strip extractor to extract a left-eye strip and a right-eye strip when a similarity calculated in the plurality of input image frames based on the 2D reference panoramic image has a maximum value.
  • This has the advantage that by using the central area of the input image, it is possible to decrease a failure probability of a left-eye panoramic image and a right-eye panoramic image.
  • the reference panoramic image may further be employed as a source to generate a high quality 3D panoramic image by performing local alignment.
  • the reference generator may generate the 2D reference panoramic image by blending a predetermined central area in each of the plurality of input image frames. Accordingly, the 2D reference panoramic image may be generated through blending of the central areas and may be used as a reference image for either a strip that is extracted from a left area of an input image or a strip that is extracted from a right area of the input image, or both. Further, using a central area can minimize the effect of rotation. Blending means that the respective predetermined central areas in the plurality of input image frames are combined to the 2D reference panoramic image.
  • the strip extractor may include a left-eye strip extractor to extract a left-eye strip of when a similarity calculated from a predetermined left-eye strip extraction position to a vertical direction based on the 2D reference panoramic image has a maximum value, and a right-eye strip extractor to extract a right-eye strip of when a similarity calculated from a predetermined right-eye strip extraction position to a vertical direction based on the 2D reference panoramic image has a maximum value.
  • the left-eye strip and the right-eye strip may be separately extracted.
  • the left-eye strip extractor may extract a left-eye strip having a maximum correlation value through template matching of the predetermined left-eye strip extraction position in an input image frame corresponding to the 2D reference panoramic image.
  • the right-eye strip extractor may extract a right-eye strip having a maximum correlation value through template matching of the predetermined right-eye strip extraction position in the input image frame corresponding to the 2D reference panoramic image. Template matching is a particularly accurate procedure to calculate the correlation value.
  • the strip extractor may extract the left-eye strip and the right-eye strip based on correlation between a color value of a predetermined left-eye strip extraction position and a color value of a predetermined right-eye strip extraction position in an input image frame corresponding to the 2D reference panoramic image. This provides for a very efficient calculation of the correlation value.
  • the strip extractor may extract the left-eye strip and the right-eye strip having the same width in an input image frame corresponding to the 2D reference panoramic image.
  • a left-eye panoramic image and a right-eye panoramic image may be generated that have similar size.
  • the image processing apparatus may further include a panoramic image generator to generate a left-eye panoramic image and a right-eye panoramic image based on the left-eye strip and the right-eye strip.
  • the image processing apparatus may further include a three-dimensional (3D) panoramic image generator to generate a 3D panoramic image based on a disparity between the left-eye panoramic image and the right-eye panoramic image.
  • 3D three-dimensional
  • the image processing apparatus may further include a processing unit to perform spherical transformation of an image that is input from a photographing device, and to perform image matching by matching a feature point between the spherically transformed images, a calculating unit to calculate a translation amount between the spherically transformed images, a first determining unit to determine whether the translation amount satisfies a predetermined target, and a second determining unit to store the spherically transformed image when the translation amount satisfies the predetermined target, and to determine whether the number of stored spherically transformed images satisfies a value that is used to generate a 3D panoramic image. Accordingly, only a minimum required number of images may be used to generate a 3D panoramic image.
  • an image processing apparatus including a global alignment unit to perform image matching of a plurality of input image frames by matching a feature point between the plurality of input image frames, and a local alignment unit to locally align a right-eye strip and a left-eye strip into a vertical direction in each of the plurality of input image frames using a 2D reference panoramic image. Therefore, further a global alignment of the input image frames can be achieved.
  • the 2D reference panoramic image may be generated based on a central area of each of the plurality of input image frames.
  • the local alignment unit may extract a left-eye strip and a right-eye strip of when a similarity calculated in the plurality of input image frames based on the 2D reference panoramic image has a maximum value.
  • the local alignment unit may generate a left-eye panoramic image and a right-eye panoramic image based on the left-eye strip and the right-eye strip.
  • an image processing method including generating a 2D reference panoramic image using a central area of each of a plurality of input image frames, and extracting a left-eye strip and a right-eye strip of when a similarity calculated in the plurality of input image frames based on the 2D reference panoramic image has a maximum value.
  • the generating may include generating the 2D reference panoramic image by blending a predetermined central area in each of the plurality of input image frames.
  • the extracting may include extracting a left-eye strip of when a similarity calculated from a predetermined left-eye strip extraction position to a vertical direction based on the 2D reference panoramic image has a maximum value, and extracting a right-eye strip of when a similarity calculated from a predetermined right-eye strip extraction position to a vertical direction based on the 2D reference panoramic image has a maximum value.
  • the extracting may include extracting a left-eye strip having a maximum correlation value through template matching of the predetermined left-eye strip extraction position in an input image frame corresponding to the 2D reference panoramic image, and extracting a right-eye strip having a maximum correlation value through template matching of the predetermined right-eye strip extraction position in the input image frame corresponding to the 2D reference panoramic image.
  • the extracting may include extracting the left-eye strip and the right-eye strip based on correlation between a color value of a predetermined left-eye strip extraction position and a color value of a predetermined right-eye strip extraction position in an input image frame corresponding to the 2D reference panoramic image.
  • the image processing method may further include generating a left-eye panoramic image and a right-eye panoramic image based on the left-eye strip and the right-eye strip.
  • the foregoing and/or other aspects are achieved by providing a method of creating a three-dimensional (3D) panoramic image from a plurality of sequentially input image frames.
  • the method includes generating a two-dimensional (2D) reference panoramic image using a center portion of each of the sequentially input image frames, extracting, from an input image frame of the plurality of sequentially input image frames, a left-eye strip and a right-eye strip having an similarity of the input image frame compared to the 2D reference panoramic image has a maximum value, and generating, by way of a processor, a left-eye panoramic image and a right-eye panoramic image using the extracted left-eye strip and right-eye strip.
  • the foregoing and/or other aspects are achieved by providing a camera to create a three-dimensional (3D) panoramic image from a plurality of sequentially input image frames.
  • the camera includes a processor to control one or more processor-executable units, a reference generator to generate a two-dimensional (2D) reference panoramic image using a center portion of each of the sequentially input image frames, and a strip extractor to extract, from an input image frame of the plurality of sequentially input image frames, a left-eye trip and a right-eye strip having an similarity of the input image frame compared to the 2D reference panoramic image has a maximum value.
  • a three-dimensional (3D) panoramic image may be generated based on two images, for example, a left-eye panoramic image and a right-eye panoramic image having different viewpoints.
  • a left-eye panoramic image and a right-eye panoramic image having different viewpoints.
  • quality degradation resulting in dizziness on the part of a viewer may occur.
  • an input strip extraction position of the left-eye panoramic image may be different from an input strip extraction position of the right-eye panoramic image. Due to a time difference between the left-eye panoramic image and the right-eye panoramic image, local motion is highly likely to occur. Therefore, quality degradation resulting in dizziness may occur at a predetermined portion of the 3D panoramic image.
  • precise local alignment may also be used with respect to the left-eye panoramic image and the right-eye panoramic image.
  • precise local alignment may be more effective than global alignment using translation information between input images.
  • FIG. 1 illustrates a sequence of an image photographed using a single camera.
  • Global translation information may be obtained by performing spherical transformation and image matching using feature point matching with respect to a plurality of consecutive input images. That is, translation information of a photographed subject may be obtained from an input image.
  • an input frame having the same or similar amount of translation may need to be selected as an input of a panoramic image.
  • the shaking may be expressed in the form of rotational translation together with horizontal and vertical translation in an input image.
  • the shaking may be calibrated during a global alignment process such as that performed by spherical transformation and image matching using feature point matching of consecutive input images.
  • a rotation is present in an input image due to the rotational translation, it may be difficult to align the left-eye panoramic image and the right-eye panoramic image. Dizziness on the part of a viewer of the image may occur due to difficulty in aligning an area where the rotation is present.
  • FIG. 1A illustrates an ideal input image photographed by panning a camera, in which no rotation of the camera occurs and hence no vertical translation of the input image occurs.
  • FIG. 1B when rotation does occur in input images 110 due to shaking of the camera, the rotation may also occur in a left-eye panoramic image 120 and a right-eye panoramic image 130 that are generated based on the input images 110. Even though the images are obtained by photographing the same subject, mismatch may occur in the images due to the rotation and thus, a viewer may experience dizziness and thereby feel inconvenienced.
  • FIG. 2 illustrates a change in a position of a subject when a rotation occurs in an input image.
  • a translation value of a left-eye strip that is an input of a left-eye panoramic image and a translation value of a right-eye strip that is an input of a right-eye panoramic image may vary.
  • Image 210 shows a case in which shaking of a camera is absent during the photographing.
  • Image 220 shows a case in which the camera is tilted to the right during the photographing.
  • Image 230 shows a rotation of a subject captured by the camera. It can be seen that, compared to image 210, a right area 235 and a left area 231 are significantly rotated upward and downward, respectively, in comparison with central area 233. That is, serious change has occurred in the right area 235 of the image and the left area 231 of the image compared to the central area 233 of the image.
  • Global translation information between input image frames is measured as the average of the entire image. Therefore, even though the rotation occurs, a translation value need not greatly vary.
  • the vertical translation of a right-eye strip and a vertical translation of a left-eye strip may occur in opposite directions such as with left-eye strip 231 and right-eye strip 235 of FIG. 2 .
  • the measured global translation information may not vary significantly since it is measured as an average.
  • a variation of the average horizontal, for example, x-axial translation value by the rotation may be significantly smaller than a variation of the average vertical, for example, y-axial translation value.
  • the average translation in the horizontal direction (x axis) caused by the rotation does not vary at a significant level.
  • FIG. 3 illustrates an image processing apparatus according to an embodiment.
  • the image processing apparatus may include, for example, a reference generator 310, a strip extractor 320, a controller 330, a panoramic image generator 340, and a 3D panoramic image generator 350.
  • the reference generator 310 may generate a 2D reference panoramic image using a central area of each of a plurality of input image frames.
  • the central area may correspond to a central area when an input image frame is divided into three areas, for example, a left area, the central area, and a right area, based on a predetermined criterion.
  • the criterion may be set in the image processing apparatus. The criterion may be adjusted based on a user setting, and may also be determined based on a translation amount of rotation.
  • the plurality of input image frames may indicate images that are photographed using a photographing device or camera at predetermined time intervals.
  • the predetermined time intervals may be of equal or different durations, depending on the embodiment.
  • the reference generator 310 may generate the 2D reference panoramic image by blending a predetermined central area in each of the plurality of input image frames.
  • a rotation translation amount may be relatively small in the central area in comparison to the left area or the right area.
  • the 2D reference panoramic image generated through blending of the central areas may be used as a reference image for either a strip that is extracted from a left area of an input image or a strip that is extracted from a right area of the input image, or both.
  • the strip extractor 320 may extract a left-eye strip and a right-eye strip when a similarity calculated in the plurality of input image frames based on the 2D reference panoramic image has a maximum value.
  • the similarity may be calculated through template matching.
  • the strip extractor 320 may extract, from the input image, the left-eye strip and the right-eye strip when the similarity calculated through template matching has a maximum value.
  • the similarity may be calculated by matching feature points.
  • the schemes of matching feature points may vary depend on the kinds of feature points.
  • the similarity may be calculated by matching a pattern.
  • the similarity may be calculated by matching a histogram.
  • the strip extractor 320 may compare translation amounts of each of the plurality of input image frames, may estimate an input image frame having the same translation amount as the 2D reference panoramic image, and may determine a similarity between the 2D reference panoramic image and the input image frame having the same translation amount.
  • the strip extractor 320 may include a left-eye strip extractor 321 and a right-eye strip extractor 323.
  • the left-eye strip extractor 321 may extract a left-eye strip when a similarity calculated form a predetermined left-eye strip extraction position to a vertical direction based on the 2D reference panoramic image has a maximum value.
  • the left-eye strip may indicate a slit of an image that is viewed with a left eye of a user.
  • a horizontal position may be determined through a previous process of a process of generating the 2D reference panoramic image.
  • the horizontal position may be determined while performing image matching between the plurality of input image frames.
  • the horizontal extraction position of the left-eye strip may be set as an initial value of a photographing device or a setting value of the photographer.
  • the left-eye strip extractor 321 may calculate the similarity while matching an input image frame with a template of the 2D reference panoramic image with respect to a vertical direction, i.e., up and down.
  • the left-eye strip extractor 321 may extract, from an input image frame, a left-eye strip of a position having a similarity closest to a predetermined similarity, or may extract a left-eye strip of a position having a maximum similarity. For example, when the predetermined similarity is 90%, the left-eye strip extractor 321 may extract a left-eye strip of a position having a similarity closest to 90%.
  • the right-eye strip extractor 323 may extract a right-eye strip when a similarity calculated from a predetermined right-eye strip extraction position to a vertical direction based on the 2D reference panoramic image has a maximum value.
  • the right-eye strip may indicate a slit of an image that is viewed with a right eye of the user.
  • a horizontal position may be determined through a previous process of a process of generating the 2D reference panoramic image.
  • the horizontal position may be determined while performing image matching between the plurality of input image frames.
  • the horizontal extraction position of the right-eye strip may be set as an initial value of the photographing device or a setting value of the photographer.
  • the right-eye strip extractor 323 may calculate the similarity while matching an input image frame with a template of the 2D reference panoramic image with respect to a vertical direction, i.e., up and down.
  • the right-eye strip extractor 323 may extract, from an input image frame, a right-eye strip of a position having a similarity closest to a predetermined similarity, or may extract a right-eye strip of a position having a maximum similarity. For example, when the predetermined similarity is 90%, the right-eye strip extractor 323 may extract a right-eye strip of a position having a similarity closest to 90%.
  • the left-eye strip extractor 321 may extract a left-eye strip having a maximum correlation value through template matching of the predetermined left-eye strip extraction position in an input image frame corresponding to the 2D reference panoramic image.
  • a correlation value may be calculated through template matching.
  • the correlation value may be calculated by matching feature points. The schemes of matching feature points may vary depend on the kinds of feature points.
  • the correlation value may be calculated by matching a pattern.
  • the correlation value may be calculated by matching a histogram.
  • the right-eye strip extractor 323 may extract a right-eye strip having a maximum correlation value through template matching of the predetermined right-eye strip extraction position in the input image frame corresponding to the 2D reference panoramic image.
  • a correlation value may be calculated through template matching.
  • the correlation value may be calculated by matching feature points. The schemes of matching feature points may vary depend on the kinds of feature points.
  • the correlation value may be calculated by matching a pattern.
  • the correlation value may be calculated by matching a histogram.
  • the strip extractor 320 may extract the left-eye strip and the right-eye strip based on correlation between a color value of a predetermined left-eye strip extraction position and a color value of a predetermined right-eye strip extraction position in an input image frame corresponding to the 2D reference panoramic image.
  • the strip extractor 320 may calculate a correlation value by comparing a red, green, blue (RGB) value of a template of the 2D reference panoramic image with the input image frame.
  • the strip extractor 320 may extract a strip of a position having a maximum correlation value in the input image frame.
  • the strip extractor 320 may extract the left-eye strip and the right-eye strip having the same width in an input image frame corresponding to the 2D reference panoramic image.
  • the strip extractor 320 may extract strips having the same width. For example, left strips and right strips extracted from the input image frame may have the same width.
  • the panoramic image generator 340 may generate a left-eye panoramic image and a right-eye panoramic image based on the left-eye strip and the right-eye strip extracted by the strip extractor 320.
  • the panoramic image generator 340 may generate a left-eye panoramic image by blending left strips.
  • the panoramic image generator 340 may generate a right-eye panoramic image by blending right strips.
  • the 3D panoramic image generator 350 may generate a 3D panoramic image based on a disparity between the left-eye panoramic image and the right-eye panoramic image.
  • the 3D panoramic image generator 350 may employ any technology for generating a 3D image based on a disparity used in the image processing field.
  • the 3D panoramic image generator 350 may employ any technology for generating a 3D image that is based on a 2D stereo image.
  • a processing unit 360 may perform spherical transformation of an image that is input from a photographing device or camera, and may perform image matching by matching a feature point between the spherically transformed images.
  • An input image is photographed from a circular baseline and thus, may be distorted.
  • the processing unit 360 may perform spherical transformation of a distorted image by considering a distance and an angle based on a position at which the input image is photographed.
  • the processing unit 360 may perform feature point matching between spherically transformed images.
  • a feature point may be determined based on a shape, a color, and the like of a subject in the input image.
  • a calculating unit 370 may calculate a translation amount between the spherically transformed images.
  • the calculating unit 370 may calculate a translation amount between input images based on the image matching result. For example, the calculating unit 370 may calculate a distance of the subject that has moved in the input image.
  • the calculating unit 370 may obtain translation information of the input image through calculation of the translation amount.
  • a horizontal position of a left-eye strip and a horizontal position of a right-eye strip may be determined based on the translation information.
  • a first determining unit 380 may determine whether the translation amount satisfies a predetermined target. When the calculated translation amount does not satisfy the predetermined target, the first determining unit 380 may not store a spherically transformed image and may request that the image be re-photographed.
  • a second determining unit 390 may store the spherically transformed image when the translation amount satisfies the predetermined target, and may determine whether the number of stored spherically transformed images satisfies the number of images that are used to generate a 3D panoramic image. For example, the number of images that are used to generate the 3D panoramic image may be determined based on a resolution of the 3D panoramic image, a capacity, and a processing rate of the image processing apparatus.
  • the stored spherically transformed images may be input images that are used to generate the 2D reference panoramic image.
  • the controller 330 may perform overall control of the image processing apparatus, and may perform functionalities of the reference generator 310, the strip extractor 320, the panoramic image generator 340, the 3D panoramic image generator 350, the processing unit 360, the calculating unit 370, the first determining unit 380, and the second determining unit 390.
  • the above constituent elements are separately illustrated in FIG. 3 to distinctively describe functions of the respective constituent elements. Accordingly, when actually configuring a product, the above constituent elements may also be configured to be processed in the controller 330, or alternatively only a portion thereof may be configured to be processed in the controller 330.
  • FIG. 4 illustrates an image processing apparatus according to another embodiment.
  • the image processing apparatus may include, for example, a global alignment unit 410 and a local alignment unit 420.
  • the global alignment unit 410 may perform image matching of a plurality of input image frames by matching a feature point between the plurality of input image frames.
  • the global alignment unit 410 may perform spherical transformation of each of the input image frames.
  • the global alignment unit 410 may obtain translation information between the input image frames through image matching.
  • the local alignment unit 420 may locally align a right-eye strip and a left-eye strip in a vertical direction in each of the plurality of input image frames based on a 2D reference panoramic image.
  • the 2D reference panoramic image may be generated based on a central area of each of the plurality of input image frames.
  • the local alignment unit 420 may generate the 2D reference panoramic image by blending predetermined central areas of the input image frames.
  • the local alignment unit 420 may calculate a correlation value through vertical template matching at a right-eye strip extraction position and a left-eye strip extraction position of the input image frame based on the 2D reference panoramic image.
  • the right-eye strip extraction position and the left-eye strip extraction position may indicate a horizontal position in the input image frame.
  • the horizontal position may be determined based on translation information obtained by the global alignment unit 410.
  • the correlation value may be calculated by matching feature points.
  • the schemes of matching feature points may vary depend on the kinds of feature points.
  • the correlation value may be calculated by matching a pattern.
  • the correlation value may be calculated by matching a histogram.
  • the local alignment unit 420 may determine a vertical position in the input image frame through template matching.
  • the local alignment unit 420 may extract a right-eye strip and a left-eye strip at a position having a maximum correlation value.
  • the local alignment unit 420 may extract a left-eye strip and a right-eye strip when a similarity calculated in the plurality of input image frames based on the 2D reference panoramic image has a maximum value.
  • the similarity may be determined based on a correlation value of template matching. That is, when the correlation value is great, the similarity may be determined to be high.
  • the correlation value may be calculated by matching feature points. The schemes of matching feature points may vary depend on the kinds of feature points.
  • the correlation value may be calculated by matching the pattern.
  • the correlation value may be calculated by matching a histogram.
  • the local alignment unit 420 may generate a left-eye panoramic image and a right-eye panoramic image based on the left-eye strip and the right-eye strip.
  • the local alignment unit 420 may extract a left-eye strip and a right-eye strip having a high similarity based on the 2D reference panoramic image, may generate a left-eye panoramic image by blending left-eye strips, and may generate a right-eye panoramic image by blending right-eye strips, thereby align the left-strips and the right-strips.
  • the local alignment unit 420 may generate the 3D panoramic image using the left-eye panoramic image and the right-eye panoramic image.
  • FIG. 5 illustrates a process of generating a 2D reference panoramic image according to an embodiment.
  • both a left-eye strip and a right-eye strip may have an incorrect vertical position information. Therefore, when aligning one of the left-eye strip and the right-eye strip based on the other of the left-eye strip and the right-eye strip, a new error may arise.
  • a central area of an input image that may be least affected by the rotation may be used as a reference to align both the left-eye strip and the right-eye strip.
  • a 2D reference panoramic image 520 may be generated using central areas 501, 503, 505, 507, 509, 511, 513, and 515 of input images.
  • FIG. 6 illustrates a process of extracting a left-eye strip and a right-eye strip according to an embodiment.
  • areas determined to have a maximum similarity to a 2D reference panoramic image 610 through template matching are extracted.
  • the areas are extracted in the form of a left-eye strip and a right-eye strip and by comparing the extracted left-eye strip and right-eye strip to a 2D reference panoramic image 610, the left-eye strip and the right-eye strip may be vertically aligned in correspondence with shaking of a photographing device or camera.
  • left-eye strips 621 and 631 may be extracted from input image frames 620 and 630, respectively, based on the 2D reference panoramic image 610.
  • Right-eye strips 641, 651, and 661 may be extracted from input image frames 640, 650, and 660, respectively.
  • the left-eye strip 621 extracted from the input image frame 620 and the left-eye strip 631 extracted from the input image frame 630 may have the same horizontal position but different vertical positions.
  • vertical positions of the left-eye strips 621 and 631 may be determined through template matching.
  • vertical positions of the left-eye strips 621 and 631 may be determined by matching feature points. The schemes of matching feature points may vary depend on the kinds of feature points.
  • vertical positions of the left-eye strips 621 and 631 may be determined by matching a pattern.
  • vertical positions of the left-eye strips 621 and 631 may be determined by matching a histogram. The same method may be applied to the right-eye strips 641, 651, and 661.
  • the right-eye strip 641 extracted from the input image frame 640, the right-eye strip 651 extracted from the input image frame 650, and the right-eye strip 661 extracted from the input image frame 660 may have the same horizontal position but different vertical positions. Vertical positions of the right-eye strips 641, 651, and 661 may be determined through template matching.
  • Left-eye strips and right-eye strips may be extracted by the number of input image frames corresponding to the 2D reference panoramic image 610.
  • a left-eye panoramic image may be generated by blending the left-eye strips 621 and 631.
  • a right-eye panoramic image may be generated by blending the right-eye strips 641, 651, and 661.
  • FIGS. 7A and 7B illustrate images before and after local alignment has been performed according to an embodiment.
  • FIG. 7A shows an example in which a left-eye panoramic image and a right-eye panoramic image mismatch occurs in the vertical direction with respect to the same subject before local alignment is vertically performed.
  • FIG. 7B shows an example in which the left-eye panoramic image and the right-eye panoramic image are aligned after the local alignment has been vertically performed.
  • FIG. 7A a downward rotation has occurred.
  • FIG. 7B the downward rotation of FIG. 7A has been aligned after local alignment is performed in the vertical direction.
  • FIG. 8 illustrates an image processing method according to an embodiment.
  • an image processing apparatus may sequentially photograph an image while panning a camera in one direction.
  • the image processing apparatus may receive a sequentially photographed image.
  • the image processing apparatus may perform spherical transformation of a photographed image or an input image. Through the spherical transformation, the image processing apparatus may compensate for any distortion occurring due to the rotation. The image processing apparatus may perform image matching through feature point matching between the spherically transformed images.
  • the image processing apparatus may calculate a translation amount between the spherically transformed images.
  • the image processing apparatus may calculate a translation amount between input images based on the image matching result.
  • the image processing apparatus may determine whether the translation amount satisfies a predetermined target. When the calculated translation amount does not satisfy the predetermined target, the image processing apparatus may elect not to store a spherically transformed image and may request photographing of the image again.
  • the image processing apparatus may store the spherically transformed image in operation 820.
  • the image processing apparatus may determine whether the number of stored spherically transformed images satisfies the number of images that are desired to generate a 3D panoramic image. For example, the number of images that are desired to generate the 3D panoramic image may be determined based on any one or more of a resolution of the 3D panoramic image, a capacity, and a processing rate of the image processing apparatus.
  • the image processing apparatus may generate a 2D reference panoramic image.
  • the image processing apparatus may generate the 2D reference panoramic image by blending central areas of spherically transformed images.
  • the image processing apparatus may extract a left-eye strip.
  • the image processing apparatus may extract a left-eye strip in an input image frame having the same or similar translation amount in comparison with the 2D reference panoramic image.
  • the image processing apparatus may extract the left-eye strip when a similarity calculated through vertical template matching in the input image frame has a maximum value.
  • the similarity may be calculated by matching feature points.
  • the schemes of matching feature points may vary depend on the kinds of feature points.
  • the similarity may be calculated by matching a pattern.
  • the similarity may be calculated by matching a histogram.
  • the image processing apparatus may extract a right-eye strip.
  • the image processing apparatus may extract a right-eye strip in the input image frame having the same translation amount as the 2D reference panoramic image.
  • the image processing apparatus may generate a left-eye panoramic image.
  • the image processing apparatus may generate the left-eye panoramic image by blending left-eye strips.
  • the image processing apparatus may generate a right-eye panoramic image.
  • the image processing apparatus may generate the right-eye panoramic image by blending extracted right-eye strips.
  • the image processing apparatus may generate a 3D panoramic image.
  • the image processing apparatus may generate the 3D panoramic image based on a disparity between the left-eye panoramic image and the right-eye panoramic image.
  • the image processing apparatus may generate the 3D panoramic image by optimizing the horizontal disparity between the left-eye panoramic image and the right-eye panoramic image.
  • FIG. 9 illustrates a camera including an image processing apparatus according to an embodiment.
  • the camera 900 may include, for example, a controller 901 and an image processing apparatus 902.
  • the camera 900 may be in the form of a digital camera or a digital video camera, although other kinds of cameras are equally possible.
  • the controller 901 may generate one or more control signals to control the camera 900 and the image processing apparatus 902.
  • the controller 901 may include one or more processors.
  • the image processing apparatus 902 may be used to generate a 3-D panoramic image for the camera 900 and may include, for example, a reference generator, a strip extractor, a controller, a panoramic image generator, and a 3D panoramic image generator. None of the preceding items are illustrated in FIG. 9 , however, each of these units may correspond to similarly named units discussed herein, for example with respect to FIG. 3 , and therefore need not be discussed again here.
  • the image processing apparatus 902 may be installed internally within the camera 900, may be attached to the camera 900, or may be separately embodied from the camera 900. Regardless of its physical configuration, the image processing apparatus 902 has all of the capabilities discussed herein.
  • the image processing apparatus 902 may include one or more internal processors or may be controlled by the one or more processors included within the multi- image processing apparatus 902 such as the one or more processors of controller 901.
  • an image processing apparatus may be applied to a camera related product and an image processing method may be employed as software.
  • the image processing apparatus and the image processing method may be applied to a product such as a digital camera and a camera image processing processor, for example.
  • an image processing apparatus may generate a 2D reference panoramic image using a central image of an input image that is less susceptible to a variation in rotation caused by shaking of a photographing device, and may generate a left-eye panoramic image and a right-eye panoramic image based on the 2D reference panoramic image.
  • the central image may be employed as a source to generate a high quality 3D panoramic image by performing local alignment.
  • an image processing apparatus may prevent a partial alignment failure of a left-eye panoramic image and a right-eye panoramic image that may occur due to a different photographing point in time between the left-eye panoramic image and the right-eye panoramic image with respect to the same subject and a time difference in the photographing moment.
  • the image processing apparatus may generate a 2D reference panoramic image capable of minimizing the effect of rotation using a central area of an input image.
  • the image processing method may be recorded in non-transitory computer-readable media including program instructions to implement various operations embodied by a computer.
  • the media may also include, alone or in combination with the program instructions, data files, data structures, and the like.
  • Examples of non-transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM disks and DVDs; magneto-optical media such as optical discs; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like.
  • Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter.
  • the described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described embodiments, or vice versa. Any one or more of the software modules described herein may be executed by a controller such as a dedicated processor unique to that unit or by a processor common to one or more of the modules.
  • the described methods may be executed on a general purpose computer or processor or may be executed on a particular machine such as the image processing apparatus described herein.
EP13158870.9A 2012-03-15 2013-03-13 Bildverarbeitungsvorrichtung und -verfahren Active EP2640063B1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
KR1020120026412A KR101804205B1 (ko) 2012-03-15 2012-03-15 영상 처리 장치 및 방법

Publications (3)

Publication Number Publication Date
EP2640063A2 true EP2640063A2 (de) 2013-09-18
EP2640063A3 EP2640063A3 (de) 2017-05-03
EP2640063B1 EP2640063B1 (de) 2019-12-25

Family

ID=48044532

Family Applications (1)

Application Number Title Priority Date Filing Date
EP13158870.9A Active EP2640063B1 (de) 2012-03-15 2013-03-13 Bildverarbeitungsvorrichtung und -verfahren

Country Status (5)

Country Link
US (1) US9378544B2 (de)
EP (1) EP2640063B1 (de)
JP (1) JP6289811B2 (de)
KR (1) KR101804205B1 (de)
CN (1) CN103313081B (de)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10414271B2 (en) 2013-03-01 2019-09-17 Delbert Tesar Multi-speed hub drive wheels
CN103854624B (zh) * 2014-03-17 2016-04-27 深圳市华星光电技术有限公司 一种液晶显示装置的驱动方法和一种液晶显示装置
US9613575B2 (en) 2014-03-17 2017-04-04 Shenzhen China Star Optoelectronics Technology Co., Ltd Liquid crystal display device and method for driving the liquid crystal display device
KR102264591B1 (ko) * 2015-02-27 2021-06-15 삼성전자주식회사 영상 처리 방법 및 이를 지원하는 전자 장치
CN104994272A (zh) * 2015-06-12 2015-10-21 惠州Tcl移动通信有限公司 一种移动终端及其实现图像扫描的方法
CN105898269A (zh) * 2015-12-27 2016-08-24 乐视致新电子科技（天津）有限公司 一种视频播放方法及装置
CN105657397B (zh) * 2015-12-30 2018-03-30 深圳Tcl数字技术有限公司 3d视频播放调整方法及系统
JP6494060B2 (ja) * 2016-02-29 2019-04-03 富士フイルム株式会社 情報処理装置、情報処理方法、及びプログラム
US10257501B2 (en) 2016-04-06 2019-04-09 Facebook, Inc. Efficient canvas view generation from intermediate views
CN106097245B (zh) * 2016-07-26 2019-04-30 北京小鸟看看科技有限公司 一种全景3d视频图像的处理方法和装置
US10339627B2 (en) 2016-10-10 2019-07-02 Gopro, Inc. Apparatus and methods for the optimal stitch zone calculation of a generated projection of a spherical image
US11290644B2 (en) * 2017-03-29 2022-03-29 Christopher Carmichael Image system with 360 and 3-d stitching
US10373362B2 (en) * 2017-07-06 2019-08-06 Humaneyes Technologies Ltd. Systems and methods for adaptive stitching of digital images
JP7075252B2 (ja) * 2018-03-23 2022-05-25 キヤノン株式会社 情報処理装置およびその制御方法、プログラム
KR102260519B1 (ko) * 2021-02-25 2021-06-03 (주)에이피피커뮤니케이션즈 3차원 입체영상 변환방법

Family Cites Families (42)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6205259B1 (en) * 1992-04-09 2001-03-20 Olympus Optical Co., Ltd. Image processing apparatus
US6714689B1 (en) * 1995-09-29 2004-03-30 Canon Kabushiki Kaisha Image synthesizing method
US6078701A (en) 1997-08-01 2000-06-20 Sarnoff Corporation Method and apparatus for performing local to global multiframe alignment to construct mosaic images
US6545701B2 (en) 1997-08-13 2003-04-08 Georgia Tech Research Corporation Panoramic digital camera system and method
US6424752B1 (en) * 1997-10-06 2002-07-23 Canon Kabushiki Kaisha Image synthesis apparatus and image synthesis method
JPH11164326A (ja) * 1997-11-26 1999-06-18 Oki Electric Ind Co Ltd パノラマステレオ画像生成表示方法及びそのプログラムを記録した記録媒体
US7477284B2 (en) 1999-09-16 2009-01-13 Yissum Research Development Company Of The Hebrew University Of Jerusalem System and method for capturing and viewing stereoscopic panoramic images
US6831677B2 (en) * 2000-02-24 2004-12-14 Yissum Research Development Company Of The Hebrew University Of Jerusalem System and method for facilitating the adjustment of disparity in a stereoscopic panoramic image pair
US7064783B2 (en) 1999-12-31 2006-06-20 Stmicroelectronics, Inc. Still picture format for subsequent picture stitching for forming a panoramic image
US8896725B2 (en) * 2007-06-21 2014-11-25 Fotonation Limited Image capture device with contemporaneous reference image capture mechanism
US20050089213A1 (en) 2003-10-23 2005-04-28 Geng Z. J. Method and apparatus for three-dimensional modeling via an image mosaic system
KR20060120052A (ko) 2003-10-28 2006-11-24 코닌클리케 필립스 일렉트로닉스 엔.브이. 파노라마 또는 모자이크 기능을 구비한 디지털 카메라
US7746404B2 (en) 2003-11-10 2010-06-29 Hewlett-Packard Development Company, L.P. Digital camera with panoramic image capture
KR101009832B1 (ko) 2004-06-18 2011-01-19 삼성전자주식회사 디지털 카메라에서 이미지 합성 장치 및 방법
JP4189761B2 (ja) 2004-09-30 2008-12-03 株式会社ビューマジック パノラマ立体画像方式
JP2006310975A (ja) 2005-04-26 2006-11-09 Mitsubishi Electric Corp ビデオモザイキング装置、パノラマ映像送受信システム、パノラマ映像作成送信装置、パノラマ映像受信装置、パノラマ映像用ストリップ作成送信装置およびパノラマ映像用ストリップ受信装置
JP4595733B2 (ja) 2005-08-02 2010-12-08 カシオ計算機株式会社 画像処理装置
KR100790890B1 (ko) 2006-09-27 2008-01-02 삼성전자주식회사 파노라마 영상 생성장치 및 방법
US7995861B2 (en) * 2006-12-13 2011-08-09 Adobe Systems Incorporated Selecting a reference image for images to be joined
KR100869952B1 (ko) * 2007-02-14 2008-11-24 삼성전자주식회사 파노라마 사진 촬영 방법 및 장치
US8593506B2 (en) * 2007-03-15 2013-11-26 Yissum Research Development Company Of The Hebrew University Of Jerusalem Method and system for forming a panoramic image of a scene having minimal aspect distortion
US8200039B2 (en) * 2007-04-05 2012-06-12 Adobe Systems Incorporated Laying out multiple images
KR100866278B1 (ko) 2007-04-26 2008-10-31 주식회사 코아로직 파노라마 영상 생성 장치, 방법 및 상기 방법을프로그램화하여 수록한 컴퓨터로 읽을 수 있는 기록매체
JP5097480B2 (ja) * 2007-08-29 2012-12-12 株式会社トプコン 画像測定装置
KR100919247B1 (ko) 2008-03-12 2009-09-30 중앙대학교 산학협력단 파노라마 영상 생성장치 및 방법, 그리고 이를 이용한 객체추적장치 및 방법
WO2010025309A1 (en) 2008-08-28 2010-03-04 Zoran Corporation Robust fast panorama stitching in mobile phones or cameras
JP4962460B2 (ja) 2008-10-03 2012-06-27 ソニー株式会社 撮像装置、撮像方法、およびプログラム
JP5409107B2 (ja) * 2009-05-13 2014-02-05 任天堂株式会社 表示制御プログラム、情報処理装置、表示制御方法、および情報処理システム
JP5347716B2 (ja) * 2009-05-27 2013-11-20 ソニー株式会社 画像処理装置、情報処理方法およびプログラム
KR101616874B1 (ko) 2009-09-23 2016-05-02 삼성전자주식회사 다중 영상 합성 방법 및 그 장치
JP2011082918A (ja) 2009-10-09 2011-04-21 Sony Corp 画像処理装置および方法、並びにプログラム
JP5359783B2 (ja) * 2009-10-28 2013-12-04 ソニー株式会社 画像処理装置および方法、並びにプログラム
JP2011130327A (ja) * 2009-12-21 2011-06-30 Sony Corp 画像処理装置および方法、並びにプログラム
JP2011135246A (ja) 2009-12-24 2011-07-07 Sony Corp 画像処理装置、撮像装置、および画像処理方法、並びにプログラム
JP5287702B2 (ja) 2009-12-25 2013-09-11 ソニー株式会社 画像処理装置および方法、並びにプログラム
JP2011147067A (ja) 2010-01-18 2011-07-28 Sony Corp 画像処理装置及び方法、並びにプログラム
JP5467993B2 (ja) 2010-03-30 2014-04-09 富士フイルム株式会社 画像処理装置、複眼デジタルカメラ、及びプログラム
JP5390707B2 (ja) 2010-06-24 2014-01-15 富士フイルム株式会社 立体パノラマ画像合成装置、撮像装置並びに立体パノラマ画像合成方法、記録媒体及びコンピュータプログラム
JPWO2012002046A1 (ja) 2010-06-30 2013-08-22 富士フイルム株式会社 立体パノラマ画像合成装置及び複眼撮像装置並びに立体パノラマ画像合成方法
KR101663321B1 (ko) * 2010-07-30 2016-10-17 삼성전자주식회사 파노라마 사진 촬영 방법
JP2012085252A (ja) * 2010-09-17 2012-04-26 Panasonic Corp 画像生成装置、画像生成方法、プログラムおよびプログラムを記録した記録媒体
JP2012151796A (ja) * 2011-01-21 2012-08-09 Sony Corp 画像処理装置と画像処理方法およびプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None

Also Published As

Publication number Publication date
EP2640063A3 (de) 2017-05-03
CN103313081A (zh) 2013-09-18
CN103313081B (zh) 2016-09-28
EP2640063B1 (de) 2019-12-25
US9378544B2 (en) 2016-06-28
JP6289811B2 (ja) 2018-03-07
KR20130104691A (ko) 2013-09-25
US20130242039A1 (en) 2013-09-19
JP2013198155A (ja) 2013-09-30
KR101804205B1 (ko) 2017-12-04

Similar Documents

Publication Publication Date Title
US9378544B2 (en) Image processing apparatus and method for panoramic image using a single camera
JP7163345B2 (ja) 映像補正方法及び装置
US10540806B2 (en) Systems and methods for depth-assisted perspective distortion correction
US11838606B2 (en) Methods and systems for large-scale determination of RGBD camera poses
US9401041B2 (en) Multi view synthesis method and display devices with spatial and inter-view consistency
CN108886611B (zh) 全景立体视频系统的拼接方法和装置
KR102013978B1 (ko) 이미지들의 융합을 위한 방법 및 장치
US8810635B2 (en) Methods, systems, and computer-readable storage media for selecting image capture positions to generate three-dimensional images
US9635348B2 (en) Methods, systems, and computer-readable storage media for selecting image capture positions to generate three-dimensional images
TWI547901B (zh) 模擬立體圖像顯示方法及顯示設備
US9948913B2 (en) Image processing method and apparatus for processing an image pair
US8750600B2 (en) Apparatus and method for generating three-dimensional (3D) zoom image of stereo camera
US20130093839A1 (en) Apparatus and method of generating three-dimensional (3d) panoramic image
US20120105601A1 (en) Apparatus and method for creating three-dimensional panoramic image by using single camera
JP2022174085A (ja) 場面の積層深度データを生成するための方法
EP3507986B1 (de) Bestimmung von mit einer mehrkameravorrichtung aufgenommenen zwischenansichtsvorhersagebereichen in bildern
KR20140077315A (ko) 3차원 모델뷰 기반의 멀티카메라 모니터링 시스템 및 방법
WO2013081576A1 (en) Capturing a perspective-flexible, viewpoint-synthesizing panoramic 3d image with a multi-view 3d camera
Paradiso et al. A novel interpolation method for 3D view synthesis
Wang et al. A common feature-based disparity control strategy in stereoscopic panorama generation
Cho et al. Stereo panoramic image stitching with a single camera
Tanger et al. Depth/disparity creation for trifocal hybrid 3d system
Peer et al. Towards a real time panoramic depth sensor

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 13/02 20060101ALI20170329BHEP

Ipc: H04N 5/232 20060101AFI20170329BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20170714

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20180312

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602013064269

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: H04N0005232000

Ipc: H04N0013106000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 13/211 20180101ALI20190617BHEP

Ipc: H04N 13/106 20180101AFI20190617BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190725

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602013064269

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1218443

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200115

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200326

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200325

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200520

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200425

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602013064269

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1218443

Country of ref document: AT

Kind code of ref document: T

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

26N No opposition filed

Effective date: 20200928

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20200331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200313

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200313

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200331

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200331

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200331

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200331

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191225

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20230221

Year of fee payment: 11

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230220

Year of fee payment: 11

Ref country code: DE

Payment date: 20230220

Year of fee payment: 11

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20240221

Year of fee payment: 12