KR20200052206A - Plenoptic image processing apparatus, system having the same, and object segmentation method thereof - Google Patents

Abstract

According to the present invention, an object division method of a plenoptic image processing device can comprise the steps of: selecting a first viewpoint image from plenoptic images; displaying a user area in a foreground or background in the first viewpoint image to generate user display information; finding the displayed user area in different viewpoint images among the plenoptic images and reflecting the user display information to the different viewpoint images; and dividing an object from the foreground or background by using the reflected user display information.

Description

Plenoptic image processing apparatus, a plenoptic image processing system including the same, and a method for dividing objects thereof {PLENOPTIC IMAGE PROCESSING APPARATUS, SYSTEM HAVING THE SAME, AND OBJECT SEGMENTATION METHOD THEREOF}

The present invention relates to a plenoptic image processing apparatus, a plenoptic image processing system including the same, and a method for segmenting the object thereof.

In general, compared to the conventional image processing technology that relies on 2D camera image information, a Plenoptic or Light Field image provides information of light traveling in an arbitrary direction in space. That is, since it provides light intensity and color information for a sampled direction for each pixel constituting a 2D image, it is well suited to realistically describing objects in real space. The plenoptic image information has spatial domain information of the existing 2D image and angular domain information by direction information at the same time. Depth information of the spatial area and additional direction of light rays by each region By using the information, various image processing such as perspective viewing change, refocusing and 3D depth of field extraction can be performed.

Published Patent: 10-2016-0120533, Published Date: October 18, 2016, Title: Object segmentation method in light field image. US Patent Publication: US 2016-0371304, Published Date: December 22, 2016, Title: METHOD AND APPARTUS FOR DATA RETRIEVAL IN A LIGHTFIELD DATABASE.

The object of the present invention is that, in a plenoptic image, if a user displays both a foreground and a background or a foreground or a background area in a first viewpoint image, the user does not determine where to place the epipolar line in the second viewpoint image. It is to provide a plenoptic image processing apparatus that automatically generates display information desired by a user by automatically reflecting it on all viewpoint images, a plenoptic image processing system including the same, and an object segmentation method thereof.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following descriptions.

An object segmentation method of a plenoptic image processing apparatus according to an embodiment of the present invention includes: selecting a first viewpoint image from a plenoptic image; Displaying a user area in a foreground or background in the first viewpoint image to generate user display information; Finding the displayed user area in different viewpoint images among the plenoptic images and reflecting the user display information in the different viewpoint images; And dividing an object from the foreground or the background using the reflected user display information.

In an embodiment, the method may further include acquiring a plenoptic image from the plenoptic camera.

In an embodiment, the similarity (SIM) of color values of pixels is used to find the displayed user area in the other viewpoint images.

을 이용하여 계산하는 단계를 더 포함하고, 여기서,

는 상기 제 1 시점 영상에서 사용자가 표시한 영역의 각 픽셀의 컬러값이고,

는 상기 제 1 시점 영상을 제외한 i번째 영상에서 상기 제 1 시점 영상에서 상기 사용자가 표시한 영역의 각 픽셀을 변환 행렬(

)에 의해 변환시킨 위치의 픽셀의 컬러값인 것을 특징으로 한다.In an embodiment, the step of reflecting on the other viewpoint images may include the similarity (SIM) as an equation.

Further comprising the step of calculating, wherein,

Is a color value of each pixel in a region displayed by the user in the first viewpoint image,

In the i-th image excluding the first view image, each pixel of a region displayed by the user in the first view image is transformed matrix (

Characterized in that the color value of the pixel at the position converted by).

In an embodiment, the step of reflecting on the other viewpoint images may further include generating a bounding box based on an area corresponding to a background after the user area is displayed on the first viewpoint image.

In an embodiment, the step of reflecting to the other viewpoint images may further include setting the bounding box at a position corresponding to the object in the different viewpoint images.

A plenoptic image processing apparatus according to an embodiment of the present invention includes: at least one processor; And a memory for storing at least one instruction executed by at least one processor, wherein the at least one instruction comprises: selecting a first viewpoint image from among the plenoptic images; Displaying a user area in a foreground or background in the first viewpoint image to generate user display information; Find the displayed user area in different viewpoint images among the plenoptic images and reflect the user display information in the different viewpoint images; And at least one processor to divide an object from the foreground or the background using the reflected user display information.

In an embodiment, the plenoptic image is characterized by being a multi-view image.

In an embodiment, when the user area is displayed on the first viewpoint image, a user area corresponding to the remaining viewpoint images is displayed.

In an embodiment, a user area corresponding to the remaining viewpoint images is searched using a similarity of color values of pixels in the user area.

In an embodiment, when the user area is displayed on the first viewpoint image, a bounding box is generated based on an area corresponding to a background, and a bounding box is set at a position corresponding to the object in the remaining viewpoint images. It is characterized by.

A plenoptic image processing system according to an embodiment of the present invention includes: a plenoptic image acquisition device for acquiring a multi-view plenoptic image from a plenoptic camera; And a plenoptic image processing apparatus for segmenting the plenoptic image into a four-dimensional object, wherein the plenoptic image processing apparatus selects a first viewpoint image from among the plenoptic images; Displaying a user area in a foreground or background in the first viewpoint image to generate user display information; Find the displayed user area in different viewpoint images among the plenoptic images and reflect the user display information in the different viewpoint images; And dividing the object from the foreground or the background using the reflected user display information.

In an embodiment, when the user area is displayed on the first viewpoint image, the plenoptic image processing apparatus generates a bounding box based on an area corresponding to a background, and attaches the object to the other viewpoint images. It is characterized by setting a bounding box at a corresponding position.

In an embodiment, the apparatus may further include a storage device that stores the result of dividing the object.

A plenoptic image processing apparatus according to an embodiment of the present invention, a plenoptic image processing system including the same, and a method of dividing an object thereof, divide an object from an entire plenoptic image by a user displaying a foreground and a background in one image You can do

The accompanying drawings are provided to help understand the present embodiment, and provide embodiments with detailed description. However, the technical features of the present embodiment are not limited to specific drawings, and the features disclosed in each drawing may be combined with each other to form a new embodiment.
1 is a diagram illustrating a plenoptic image processing system 10 according to an embodiment of the present invention.
2 is a flowchart exemplarily showing a method of object segmenting a plenoptic image of the plenoptic image processing system 10 according to an embodiment of the present invention.
3 is a flowchart exemplarily showing a method of object segmenting a plenoptic image of a general plenoptic image processing system.
4 is a diagram illustrating a result of automatically reflecting user display information in another viewpoint area in a plenoptic processing apparatus according to an embodiment of the present invention.
5 is a diagram illustrating a result of a user manually reflecting user display information in a typical plenoptic processing apparatus.
6 exemplarily shows a plenoptic image processing apparatus 1000 according to an embodiment of the present invention.

Hereinafter, the contents of the present invention will be described clearly and in detail so that those skilled in the art of the present invention can easily implement the drawings using the drawings.

The present invention can be applied to various changes and may have various forms, and specific embodiments will be illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to a specific disclosure form, and it should be understood as including all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms may be used for the purpose of distinguishing one component from other components. For example, the first component may be referred to as the second component without departing from the scope of the present invention, and similarly, the second component may also be referred to as the first component. When an element is said to be "connected" or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

Terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "include" or "have" are intended to indicate the presence of a feature, number, step, action, component, part or combination thereof carried out, one or more other features or numbers. It should be understood that it does not preclude the existence or addition possibilities of, steps, actions, components, parts or combinations thereof. Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains. Terms, such as those defined in the commonly used dictionary, should be interpreted as meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. .

In general, object segmentation of an image is an operation of dividing an object desired by a user from an image. In the conventional two-dimensional image, semi-automatic object segmentation methods are used to divide an object based on the information when the user wants to divide the object, which is both the foreground and the background, or a foreground or background area.

In order to divide an object from a multi-view-based plenoptic image, if a user displays both the foreground and the background or displays one area among the foreground or background in all viewpoint images, the existing object segmentation method can be applied as it is. However, as the number of viewpoints increases, the user needs to intervene. For this reason, overall execution time or user friendliness for object segmentation is deteriorated. In order to efficiently perform such object segmentation, the user selects an arbitrary viewpoint image (first viewpoint image) to display both the foreground and the background, or after displaying one area among the foreground or background, the user views another viewpoint image ( Once the information displayed by the user is confirmed in the first view image only once in the first view image), the user displays the first view image in all other view images except the first view image and the second view image, and the second view Information determined by the user in the image may be automatically applied.

When the user displays both the foreground and the background or one area of the foreground or background on the first viewpoint image, and applies the information displayed by the user from the first viewpoint image to the same position of the second viewpoint image, the first viewpoint image and the second Due to the time difference between the viewpoint images, information drawn on the second viewpoint images is misaligned. At this time, the first viewpoint image and the second viewpoint image are corrected (Calibration) and aligned (Rectification) so that the position candidate group in the second viewpoint image corresponding to the information displayed by the user in the first viewpoint image is the epipolar line (Epipolar). Line). Here, the epipolar line means a straight line connected to the first viewpoint and projected on the second viewpoint image.

When the user determines where the position of the epipolar line is located in the second viewpoint image, the user automatically displays the first viewpoint image at the desired position in the remaining viewpoint images except the first viewpoint image and the second viewpoint image. Information is reflected. However, in this process, more user intervention (determining where to locate on the epipolar line in the second viewpoint image) is required compared to object segmentation in the existing two-dimensional image.

1 is a diagram illustrating a plenoptic image processing system 10 according to an embodiment of the present invention. Referring to FIG. 1, the plenoptic image processing system 10 may include a plenoptic image acquisition device 100 and a plenoptic image processing device 200.

The plenoptic image acquisition apparatus 100 may be implemented to receive an array image of a multi-viewpoint for an object from the camera array 11.

The plenoptic image processing apparatus 200 may be implemented to perform segmentation of an object desired by a user from the obtained plenoptic image. Here, the divided plenoptic image may be converted to a storage format and then stored in the storage device 12.

In the plenoptic image processing system 10 according to an embodiment of the present invention, when a user displays both a foreground and a background in a first viewpoint image in a plenoptic image, or when one of the foreground and background is displayed, the user views the second viewpoint image In the display, the display information desired by the user can be automatically generated by automatically reflecting the areas displayed on all the viewpoint images as well as the second viewpoint images, without determining where to place them on the epipolar line. Accordingly, the plenoptic image processing system 10 according to an embodiment of the present invention may perform a 4D object segmentation by expanding a dimension of an existing 2D object segmentation.

2 is a flowchart exemplarily showing a method of object segmenting a plenoptic image of the plenoptic image processing system 10 according to an embodiment of the present invention. 1 and 2, a process of dividing an object desired by a user in a plenoptic image is as follows.

A multi-view-based plenoptic image may be obtained through a plenoptic camera photographing (S110). The plenoptic image obtained here may be a multi-view image. The user can select one arbitrary viewpoint image from the plenoptic images (S120). The viewpoint image selected by the user will be referred to as a first viewpoint image.

In the first viewpoint image, the user may display one area among the foreground and background or the foreground or background (S130). The displayed area is user display information.

The area displayed by the user may exist at a different location due to parallax in different viewpoint images. The user display information displayed in the first view image may be searched from the other view image, and the user display information found in the other view image may be reflected (S140). In order to find user display information of the first viewpoint image from another viewpoint image, the following equation may be used. Find a transformation matrix that maximizes the similarity (SIM) of the color values of pixels displayed by the user, and transforms the transformation matrix.

은 제 1 시점 영상에서 사용자가 표시한 영역의 각 픽셀의 컬러값이고,

은 제 1 시점 영상을 제외한 i번째 영상에 제 1 시점 영상에서 사용자가 표시한 영역의 각 픽셀을 변환 행렬(

) 에 의해 변환한 위치의 픽셀의 컬러값이다.here

Is a color value of each pixel in a region displayed by the user in the first viewpoint image,

Transforms each pixel in the area displayed by the user in the first view image to the i-th image except the first view image.

) Is the color value of the pixel at the position converted by.

Meanwhile, a method of calculating the similarity (SIM) may be various. Below, as an example, similarity is calculated using the following equation.

As described above, after reflecting the user display information of the first viewpoint image to the remaining viewpoint images (after S140), an object segmentation desired by the user may be performed by applying a 4D object segmentation method (S150). Finally, the result of the foreground / background segmentation can be derived. In this case, object segmentation may be performed using a region-based graph cut. Object segmentation using region-based graph cut extracts the region of interest from the input original image, performs filtering on the extracted region of interest, and groups the pixels of the filtered region of interest into groups and consists of grouped pixels Weights can be calculated for each region, and the original image can be divided into a region of interest and a background region using the calculated weights. Meanwhile, it should be understood that the object segmentation of the present invention is not limited to area-based graph cuts.

Meanwhile, an operation of focusing multiple views of the plenoptic image after object division may be additionally performed.

3 is a flowchart exemplarily showing a method of object segmenting a plenoptic image of a general plenoptic image processing system. Referring to FIG. 3, processes (S210, S220, and S230) shown in FIG. 2 are performed by the user displaying the foreground and background on the first viewpoint image or displaying one area among the foreground or background. same.

Thereafter, the user may select an arbitrary viewpoint image except for the first viewpoint image (S240). The selected image is a second viewpoint image. The information displayed by the user in the first viewpoint image may appear as an epipolar line in the second viewpoint image. In the first viewpoint image, a position corresponding to one pixel displayed by the user exists on the line in the second viewpoint image. Therefore, a determination process is needed to determine where the user is located on the line appearing on the second viewpoint image (S250). When this determination process is performed, user display information is automatically reflected to images other than the first and second viewpoint images (S260). Through this, 4D object segmentation is performed (S270).

4 is a diagram illustrating a result of automatically reflecting user display information in another viewpoint area in a plenoptic processing apparatus according to an embodiment of the present invention. Referring to FIG. 4, when a user displays a foreground and a background in a first viewpoint image as shown in FIG. 4 (a), a bounding box based on an area corresponding to FIG. 4 (b) and a garting background Can be generated. Then, the bounding box may be automatically set at a position corresponding to the object in another viewpoint image.

5 is a diagram illustrating a result of a user manually reflecting user display information in a typical plenoptic processing apparatus. As shown in (a) of FIG. 5, a user displays a foreground and a background on a first viewpoint image, and a bounding box may be generated as shown in (b) of FIG. 5. As shown in (c) of FIG. 5, after the bounding box is generated, the user should move the bounding box based on the epipolar line in the second viewpoint image to place the bounding box at the position including the object in the second viewpoint image. do.

6 exemplarily shows a plenoptic image processing apparatus 1000 according to an embodiment of the present invention. Referring to FIG. 6, the plenoptic image processing apparatus 1000 may include at least one processor 1100, a network interface 1200, a memory 1300, a display 1400, and an input / output device 1500. .

The processor 1100 may include at least one device through FIGS. 1 to 5, or may be implemented by at least one method described above through FIGS. 1 to 5. As described above, the processor 1100 selects a first viewpoint image from among the plenoptic images, displays a user area in the foreground or background in the first viewpoint image to generate user display information, and is different from the plenoptic images. Instructions may be executed to find a user area displayed in viewpoint images, reflect user display information to other viewpoint images, and divide an object from the foreground or background using the reflected user display information.

The processor 1100 may execute a program and control the plenoptic image processing apparatus 1000. The plenoptic image processing apparatus 1000 may be connected to an external device (for example, a personal computer or a network) through the input / output device 1500 and exchange data. The generating device 1000 is a mobile device such as a mobile phone, a smart phone, a PDA, a tablet computer, a laptop computer, a computing device such as a personal computer, a tablet computer, a netbook, or an electronic product such as a television, a smart television, a security device for gate control, etc. It can include various electronic systems.

The network interface 1200 may be implemented to communicate with an external network by various wired / wireless methods.

The memory 1300 may include instructions readable by a computer. The processor 1100 may perform the aforementioned operations as the instructions stored in the memory 1300 are executed in the processor 1100. The memory 1300 may be a volatile memory or a nonvolatile memory. The memory 1300 may include a storage device to store user data. The storage device may be an embedded multimedia card (eMMC), a solid state drive (SSD), or universal flash storage (UFS). The storage device may include at least one nonvolatile memory device. Non-volatile memory devices include NAND Flash Memory, Vertical NAND Flash (VNAND), NOR Flash Memory, Resistive Random Access Memory (RRAM), and Phase Change Memory (Phase-Change Memory: PRAM), Magnetoresistive Random Access Memory (MRAM), Ferroelectric Random Access Memory (FRAM), Spin Transfer Torque Random Access Memory (STT-RAM), etc. Can be

The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware components and software components. For example, the devices, methods, and components described in the embodiments include, for example, a processor, a controller, an Arithmetic Logic Unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate (FPGA). Arrays, Programmable Logic Units (PLUs), microprocessors, or any other device capable of executing and responding to instructions can be implemented using one or more general purpose computers or special purpose computers. The processing device may perform an operating system (OS) and one or more software applications running on the operating system.

In addition, the processing device may access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the corresponding technical field may use a processing device to process a plurality of processing elements and / or a plurality of types. It can be seen that it can contain elements. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and may be recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for an embodiment or may be known and available to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. Includes hardware devices specifically configured to store and execute program instructions such as magneto-optical media, ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

In the related art, a user independently displays a foreground and a background to perform multi-view based plenoptic image object segmentation in all viewpoint images, or displays a foreground and a background in a first viewpoint image, and a first in a second viewpoint image. There were many user interventions, such as performing an operation to correct the position of the portion displayed in the viewpoint image. On the other hand, according to the present invention, the user intervention used in the existing 2D object segmentation method, that is, the user can perform object segmentation in the entire plenoptic image by only displaying the foreground and background by the user in one image.

Meanwhile, the above-described contents of the present invention are only specific embodiments for carrying out the invention. The present invention will include technical ideas that are abstract and conceptual ideas that can be utilized as future technologies, as well as specific and practically available means themselves.

10: Plenoptic image processing system
100: plenoptic image acquisition device
200, 1000: Plenoptic image processing device

Claims (14)

In the object segmentation method of the plenoptic image processing apparatus,
Selecting a first viewpoint image from the plenoptic images;
Displaying a user area in a foreground or background in the first viewpoint image to generate user display information;
Finding the displayed user area in different viewpoint images among the plenoptic images and reflecting the user display information in the different viewpoint images; And
And dividing an object from the foreground or the background using the reflected user display information. According to claim 1,
And obtaining a plenoptic image from the plenoptic camera. According to claim 1,
An object segmenting method characterized by using a similarity (SIM) of color values of pixels to find the displayed user area in the other viewpoint images. The method of claim 3,
The step of reflecting on the other viewpoint images,
The similarity (SIM) is expressed by the equation

Further comprising the step of calculating using,
here,

Is a color value of each pixel in a region displayed by the user in the first viewpoint image,

In the i-th image excluding the first view image, each pixel of a region displayed by the user in the first view image is transformed matrix (

Object color separation method, characterized in that the color value of the pixel at the position converted by. According to claim 1,
The step of reflecting on the other viewpoint images,
And generating a bounding box based on an area corresponding to a background after the user area is displayed on the first viewpoint image. The method of claim 5,
The step of reflecting to the other viewpoint images,
And setting the bounding box at a position corresponding to the object in the other viewpoint images. In the plenoptic image processing apparatus:
At least one processor; And
And a memory storing at least one instruction executed by at least one processor,
The at least one instruction,
A first viewpoint image is selected from the plenoptic images;
Displaying a user area in a foreground or background in the first viewpoint image to generate user display information;
Find the displayed user area in different viewpoint images among the plenoptic images and reflect the user display information in the different viewpoint images; And
Plenoptic image processing apparatus characterized in that it is executed in at least one processor to divide the object from the foreground or the background using the reflected user display information. The method of claim 7,
The plenoptic image processing apparatus is characterized in that the multi-view image. The method of claim 7,
When the user area is displayed on the first viewpoint image, a user area corresponding to the remaining viewpoint images is displayed. The method of claim 9,
A plenoptic image processing apparatus, characterized in that a user region corresponding to the remaining viewpoint images is searched by using the similarity of color values of pixels in the user region. The method of claim 9,
When the user area is displayed on the first viewpoint image, a bounding box is generated based on the area corresponding to the background,
A plenoptic image processing apparatus characterized in that a bounding box is set at a position corresponding to the object in the remaining viewpoint images. A plenoptic image acquisition device for acquiring a multi-view plenoptic image from a plenoptic camera; And
And a plenoptic image processing apparatus for dividing the plenoptic image into 4D objects,
The plenoptic image processing apparatus,
A first viewpoint image is selected from the plenoptic images;
Displaying a user area in a foreground or background in the first viewpoint image to generate user display information;
Find the displayed user area in different viewpoint images among the plenoptic images and reflect the user display information in the different viewpoint images; And
A plenoptic image processing system, characterized in that the object is segmented from the foreground or the background using the reflected user display information. The method of claim 12,
The plenoptic image processing apparatus,
When the user area is displayed on the first viewpoint image, a bounding box is generated based on an area corresponding to a background, and a bounding box is set at a position corresponding to the object in the other viewpoint images. Plenoptic image processing system. The method of claim 12,
A plenoptic image processing system further comprising a storage device that stores the result of dividing the object.

Images