KR101649752B1 - Generating method for multiview image using interface with image matrix and generating system for multiview image - Google Patents

Abstract

A multi-view image generating method using an image matrix interface includes the steps of: capturing a moving image of the same target object by a plurality of cameras disposed at different positions; receiving a moving image by a computer device connected to the plurality of cameras; The apparatus according to claim 1, wherein the device classifies the moving image frame by frame according to a position and a photographing time of the camera, and the computer device has an image object corresponding to a frame of the moving image classified on the display device, Dimensional matrix having a different shooting time, the computer device receiving input of a command to select the object displayed in the two-dimensional matrix, and the computer device selecting a series Frames are synthesized to generate multi-view images It includes the steps:

Description

TECHNICAL FIELD [0001] The present invention relates to a multi-view image generation method and a multi-view image generation system using an image matrix interface,

The techniques described below relate to a method and system for generating multi-view images.

A multi-view image is a field of image processing that provides users with various viewpoints in various directions through spatial synthesis of images photographed by one or more cameras.

US Patent No. 7,312,821

In order to generate a multi-view image using a plurality of cameras, a plurality of cameras individually select and synthesize a photographic image photographed by each of the plurality of cameras. Conventional multi-view image generation has a disadvantage that much time and effort are required.

In the technique described below, a source moving image captured by a plurality of cameras is provided on a frame basis in a computer device as a two-dimensional interface according to time and a position of a camera, and a user is allowed to produce a multi-view image using a two-dimensional interface .

The solutions to the technical problems described below are not limited to those mentioned above, and other solutions not mentioned can be clearly understood by those skilled in the art from the following description.

A method for generating a multi-view image using a video matrix interface in a computer device includes the steps of capturing a moving image of the same target object by a plurality of cameras disposed at different positions, receiving a moving image by a computer device connected to the plurality of cameras, Wherein the device classifies the moving picture frame by frame according to the position and the photographing time of the camera, and the computer device displays the image object corresponding to the frame of the moving picture classified in the display device as the position of the camera on one axis, Dimensional matrix, the method comprising the steps of: receiving a command for a computer device to select an object displayed in a two-dimensional matrix; and generating a multi-view image by synthesizing a series of frames selected by the computer device in response to the command .

In the axis having the photographing time, the image object is outputted in the order of frames having all the frame order or the reference interval according to the time flow of the camera.

An image object is output in an axis having a camera position according to a camera order in a linear direction or in a rotation direction with respect to the target object.

The multi-view image generation system includes a plurality of cameras disposed at different positions and capturing moving pictures of the same object, a plurality of cameras connected to the plurality of cameras, and the moving pictures are classified according to the position and shooting time of the camera in frame units A computer apparatus for synthesizing a selected frame according to a command input from an input device to generate a multi-view image, an image object corresponding to a frame of the classified moving image as a single axis, Dimensional matrix, and an input device for receiving a command for selecting an image object displayed on the two-dimensional matrix.

The computer device generates a multi-view image by synthesizing the frames corresponding to the image object selected in the input device according to the selected order.

In the technique described below, an image of the same object is photographed as a moving image, all the cameras acquire images at the time the camera is located, and various kinds of multi-view images can be generated using the captured images. In addition, the technique described below can provide all of the images taken by a plurality of cameras in a matrix form on the display device according to the frame order, so that the user can easily generate various multi-view images.

The effects of the techniques described below are not limited to those mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the following description.

FIG. 1 is an example showing a scene in which a specific object is photographed by a plurality of multi-viewpoint cameras.
2 is an example in which a plurality of multi-viewpoint cameras are arranged.
FIG. 3 shows an example in which images photographed by a plurality of cameras are arranged in a two-dimensional matrix form.
4 is an example of a flowchart for a multi-view image generation method using an image matrix interface.
5 is an example of an image matrix interface output to the display device.
6 is another example of the image matrix interface output to the display device.
7 shows an example in which a user selects an image object using an image matrix interface in the form of a two-dimensional matrix.
8 is an example of a block diagram showing a configuration of a multi-view image generating system.

The following description is intended to illustrate and describe specific embodiments in the drawings, since various changes may be made and the embodiments may have various embodiments. However, it should be understood that the following description does not limit the specific embodiments, but includes all changes, equivalents, and alternatives falling within the spirit and scope of the following description.

The terms first, second, A, B, etc., may be used to describe various components, but the components are not limited by the terms, but may be used to distinguish one component from another . For example, without departing from the scope of the following description, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

As used herein, the singular " include "should be understood to include a plurality of representations unless the context clearly dictates otherwise, and the terms" comprises & , Parts or combinations thereof, and does not preclude the presence or addition of one or more other features, integers, steps, components, components, or combinations thereof.

Before describing the drawings in detail, it is to be clarified that the division of constituent parts in this specification is merely a division by main functions of each constituent part. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more functions according to functions that are more subdivided. In addition, each of the constituent units described below may additionally perform some or all of the functions of other constituent units in addition to the main functions of the constituent units themselves, and that some of the main functions, And may be carried out in a dedicated manner. Accordingly, the existence of each component described in the present specification should be interpreted as a function. For this reason, the multi-view image generation method 200 and multi-view image generation system 300 using the image matrix interface ) Can be different from the corresponding drawings within the scope of achieving the object of the following description.

Also, in performing a method or an operation method, each of the processes constituting the method may take place differently from the stated order unless clearly specified in the context. That is, each process may occur in the same order as described, may be performed substantially concurrently, or may be performed in the opposite order.

First, a multi-view camera to which the following description is applied will be described. FIG. 1 is an example showing a scene in which a specific object is photographed by a plurality of multi-viewpoint cameras. A specific object to be photographed is called a target object.

The plurality of cameras 50 shown in FIG. 1 correspond to multi-view cameras having different viewpoints. Hereinafter, a device referred to as a camera means a multi-view camera having different viewpoints such as the camera shown in FIG. In FIG. 1, reference numerals from camera a (50a) to camera j (50j) are separately given in connection with the following description.

In the following description, the camera includes a camera for capturing a still image and a camera for capturing a moving image. It should be noted that the following description assumes that the camera described below is basically a digital imaging device such as a digital camcorder since all the plurality of cameras 50 must use successive images of the target object.

Fig. 1 shows a case where a plurality of cameras 50 are arranged in a circle around a target object 5 to be imaged. More precisely, a plurality of cameras are arranged in an arc shape on a virtual horizontal plane having a constant height. Literally, an arc means a portion defined by two points on a circumference or other curve. The camera may be disposed circumferentially in the form of surrounding the target object 5 located in the central region of the circle, or may be disposed at a constant portion of the circumference. Fig. 1 shows an example in which a plurality of cameras are arranged at regular intervals on a portion indicated by a dashed line in a circumference.

A plurality of cameras (50) are placed on the pedestal device (30) where the camera is fixed. Each of the cameras may be disposed in a device such as a separate tripod or a plurality of cameras 50 may be successively disposed on one circular arc like the pedestal device 30 shown in FIG.

The same camera apparatus as shown in Fig. 1 can generate an image having the effect of rotating the target object 5. [ For example, the camera apparatus of FIG. 1 can generate an image representing the motion of a person while rotating in a static state in a movie 'matrix'. This video technique is also called time slicing.

The computer device 80 is a device that controls a plurality of cameras 50 and receives images photographed by the camera 50. [ The computer device 80 may generate a multi-view image using the two-dimensional matrix interface described below. It is needless to say that the computer device for generating the multi-view image need not necessarily be a device connected to the plurality of cameras 50 as shown in FIG. A multi-view image may be generated in a separate computer device using the image data collected by the computer device 80 of FIG. For convenience of explanation, it is assumed that the computer device 80, which has once received an image from a plurality of cameras 50, generates a multi-view image.

In addition, the technique described below is not necessarily applied to the arranged multi-view camera as shown in Fig. It is sufficient for the multi-viewpoint camera to photograph the same object or adjacent object at the same time. 2 is an example in which a plurality of multi-viewpoint cameras are arranged.

2 (a) shows a case where the camera 50 is arranged in an arc shape and has a certain distance from the object 5 as an object of photography as shown in Fig. Fig. 2 (b) shows a case where the cameras 50 are arranged in a straight line and their distances from the object 5 are different from each other. 2 (c) shows a case where a plurality of cameras 50 are randomly arranged.

The technique described below is for generating a multi-view image using an image captured by a plurality of cameras over the same time, and thus the arrangement state of the camera is not substantially related. Depending on the placement of the camera, the size or location of the object in the image can vary, but this is the object that needs to be corrected, if necessary, through a separate task.

Hereinafter, the multi-view image generation method 200 and the multi-view image generation system 300 using the image matrix interface will be described in detail with reference to the drawings.

3 shows an example in which images photographed by a plurality of cameras 50 are arranged in a two-dimensional matrix form. FIG. 3 shows an example in which images photographed by the plurality of cameras 50 shown in FIG. 1 are arranged.

The plurality of cameras 50 are arranged according to a constant reference and order, respectively. For example, in FIG. 1, a plurality of cameras 50 are arranged in an arc shape surrounding a target object, and the order can be determined based on a specific camera. The order may be determined from camera a 50a to camera j 50j or vice versa. Since the cameras 50 arranged in an arc form photograph the target object at different angles, the target object is photographed at a point of slightly different angle according to the sequence of the camera. If the cameras are arranged in a line, the order may be determined in a linear direction with respect to a certain direction.

3, the vertical axis (y axis) corresponds to the camera position. That is, the camera is identified according to the camera position, and the image captured by the camera is arranged. The vertical axis in Fig. 3 has items from camera a 50a to camera j 50j. Since the camera position is the viewpoint of the image, the vertical axis of FIG. 3 is a sequence in which the viewpoint changes continuously when the viewpoint is changed with respect to the image.

3, the horizontal axis (x axis) corresponds to the flow of time. And image frames obtained in the time course of each camera 50 are arranged in the vertical axis direction. 3 assumes that the camera 50 acquires an image of 20 frames per second. 3 illustrates an arrangement in which all the frames photographed by each camera 50 are arranged. The "image object" in Fig. 3 corresponds to one frame.

4 is an example of a flowchart for a multi-view image generation method 200 using an image matrix interface.

A plurality of cameras 50 disposed at different positions shoot a movie for the same target object (210). A plurality of cameras 50 capture a moving image for a predetermined time with respect to the same target object, acquire an image photographed by all cameras 50 at different viewpoints, and generate a multi-view image using the captured images. At this time, the acquired image corresponds to the source image, and the user can generate various kinds of multi-view images using the image matrix interface provided by the computer device 80. In other words, it is necessary to acquire a continuous image for every point of time by taking a single shot without repeatedly shooting the target object.

The computer device 80 connected to the plurality of cameras 50 receives the moving image 220 and classifies the moving image received by the computer device 80 on a frame-by-frame basis according to the position and the photographing time of the camera 230. When the computer device 80 receives an image from the plurality of cameras 50, the computer device 80 classifies and stores the acquired image frames for each camera 50 in order. The classification operation means that a received image is received from a certain camera, an identifier is inserted so that the received image can be distinguished in the order of reception, a table is generated for the data, or the stored data is stored in a predetermined place in the storage device. Various techniques can be used for dividing the images received by the computer device 80 into the respective cameras and the images belonging to the respective cameras in chronological order.

Thereafter, the computer device 80 outputs the image object corresponding to the frame of the moving picture classified on the separate display device, as a two-dimensional matrix having the camera position on one axis and the photographing time on the other axis (240 ). Here, the image object does not mean the moving image frame itself, but means a separate display object corresponding to the moving image frame. For example, the image object may be a thumbnail image object for a frame, a rectangular object for which no image is displayed, a different shaped object, or a specific icon. The computer device 80 outputs the images photographed by the plurality of cameras 50 to the display device in the form of a two-dimensional matrix based on the photographing time and the position of the camera.

The computer device 80 then receives a command to select an object displayed in the two-dimensional matrix (250). That is, the user selects a frame to synthesize a multi-view image from a source image using a two-dimensional matrix displayed on a display device. For the user, the two-dimensional matrix displayed on the display device corresponds to one interface for generating a multi-view image. Hereinafter, the two-dimensional matrix interface displayed on the display device is referred to as an image matrix interface. Will be described later in detail in FIG. 5 and the related description below.

A user may input a selection command through various input means (mouse, keyboard, etc.) connected to the computer device 50, or may input a selection command through a touch input if the display device includes a touch panel.

Finally, the computer device 80 generates a multi-view image by synthesizing a series of frames selected according to the command selected by the user (260).

5 is an example of an image matrix interface output to the display device. 5, it is assumed that a source image having 20 frames per second exists as shown in FIG.

5 (a), the vertical axis represents the camera position and the horizontal axis represents the shooting time, as shown in FIG. 5 (a) shows an example in which all the images photographed by the camera 50 over time are displayed. The actual display device may not display the identifier or the time of the camera. 5 (a) shows an example in which the user selects an image object corresponding to a frame acquired in 0.45 sec from the image captured by the camera a. And a process of previewing a certain image object on a display device when the user selects a specific image object.

FIG. 5B shows an example in which only the corresponding image is displayed at intervals of 0.5 seconds without displaying the images of all the frames in the two-dimensional matrix. If the user generates a multi-view image using such a two-dimensional matrix, a video image that is slightly cut off will be generated. In some cases, such a multi-view image may be needed. Depending on the user setting, it may be possible to display all the frames at the shooting time or to display the frames at regular intervals. Furthermore, the same image can be created using the image matrix interface of FIG. 5 (a). If a user selects only an image corresponding to a dropped frame at a predetermined interval, an image with the same effect can be generated.

In summary, an image object can be output according to a camera sequence in a linear direction or a rotation direction with respect to a target object on an axis having a camera position. In addition, the image object may be output in the order of frames having all frame orders or reference intervals according to the time-series taken by the camera on the axis having the photographing time.

6 is another example of the image matrix interface output to the display device.

FIG. 6 shows a camera type along the vertical axis (y-axis) as shown in FIG. 3, and the horizontal axis represents time. In FIG. 3, all the frames are arranged on the horizontal axis. However, it is difficult to arrange all the frames for the actual video. Even a short video has a lot of frames.

In FIG. 6, a rectangular image object is illustrated according to time intervals, but this does not correspond to an actual frame. When the user selects a specific point on the horizontal axis, the time of the point can be displayed. FIG. 6 shows an example for 2.65 sec.

The user can change the frame position while dragging a pointer indicating a specific point on the horizontal axis. If a specific camera is selected in this state, the camera can view the image acquired in the corresponding frame. FIG. 6 shows an example of selecting an image acquired by the camera j at a time of 2.65 sec.

Of course, the horizontal axis may be displayed in a different way from FIG. As shown in FIG. 3, an image of all the frames may be displayed, or an image may be displayed by a frame having a constant time interval as shown in FIG. 5 (b). However, if the time of the corresponding frame is displayed or the corresponding position is separately displayed as in 6, the subsequent operation or editing can be smoothly performed.

7 shows an example in which a user selects an image object using an image matrix interface in the form of a two-dimensional matrix.

7 (a) shows an example in which the user continuously selects an image object in the image matrix interface. The user first selects an image frame from the first image frame of the camera a to 0.2 sec of the camera a (indicated by 1). ① The image is the image taken by the camera a for 0.2 sec from the start of the shooting. After that, the user selects images from camera a to camera j continuously in 0.2sec frame (marked ②). ② The image corresponds to the image having the effect of rotating the target object by changing the position (viewpoint) of the camera at the stopping time of 0.2 sec. Then, the user selects 0.5 second frame image from the 0.2 sec image frame of the camera j (indicated by?). ③ Image is the image taken by camera j from 0.2sec to 0.5sec.

Meanwhile, the display device can output the effect that the image object is selected when the user selects the image object in the image matrix interface displayed on the display device. For example, the color or brightness of the selected image object may be changed, or a line may be displayed for the image object as shown in FIG.

7 (b) is another example in which the user uses the image matrix interface. The user does not select an image object at a time, but moves the viewpoint completely in the middle to select another image object. The user first selects an image frame from the first image frame of the camera a to 0.15 sec of the camera a (indicated by 1). Then, in the image frame of 0.15 sec of the camera a, the image frame of 0.3 sec of the camera e is selected by a diagonal line (indicated by?). (2) Unlike FIG. 7 (a), the image has the effect of rotating the target object over time. Such an effect can not be expected in a multi-view image in which a still image is used. The user wants to create an image at a point different from the image ① and the image ② after the ② image, and selects the image frame from the first photographing point to 0.3 sec at the camera j (marked ③). In other words, the image provides images of the different viewpoints in the same time zone as the images (1) and (2). Then, the user selects 0.5 sec image frame of the camera g in the obliquely upward direction in the 0.3 sec image frame of the camera j (indicated by?). Then, the user selects an image frame moving from the camera g to the camera a at the stop time of 0.5 sec (indicated by ⑤). (5) The image has the effect of rotating in the opposite direction to the image (2) of Fig. 7 (a) in the stop state.

Although FIG. 7 shows some examples using the image matrix interface, it is possible to generate multi-view images of almost infinite types by using the image matrix interface. For example, if the flow of time is selected in reverse order, an image may be generated backwards in time, or a time interval for selecting an image object may be adjusted to generate a constantly fast rotating image while rotating slowly.

FIG. 8 is an example of a block diagram showing the configuration of the multi-view image generation system 300. FIG. The description of the multi-view image generation method 200 using the image matrix interface will be omitted or briefly explained.

The multi-view image generation system 300 includes a plurality of cameras 310 disposed at different positions and capturing moving pictures of the same object, a plurality of cameras 310 connected to the plurality of cameras to receive moving pictures, A computer device 320 for classifying the images according to the shooting time and composing the selected frames according to a command input from the input device to generate a multi-view image, an image object corresponding to the frame of the classified moving image, A display device 330 for outputting the image data in the form of a two-dimensional matrix having the other imaging axis as one axis, and an input device 340 receiving a command for selecting an image object displayed on the two-dimensional matrix.

The camera 310 is the same device as the camera 50 of Fig. 1 described above. The camera 310 transmits the photographed moving image to the computer device 320 via a wired or wireless network. The computer device 320 receives the moving image data through the communication module 321 included therein and stores the received moving image data in the storage device 322. The received video data is referred to as a source video. The network including the communication module 321 and connecting the camera 310 and the computer device 320 may be referred to as an interface device. The interface device may be a wired device or a wireless network device that transmits data using the same protocol between the camera 310 and the computer device 320. [ Various types of devices, protocols, etc. used in the related field can be used as the interface device.

The computing device that generates a signal at the computer device 320 to control another device and generates a multi-view image is the processor device 232. The processor unit 323 may generate a multi-view image by combining the image frames selected by the user and store the multi-view image in the storage unit 322.

An image object may be output in an axis having a position of a camera in the display device 330 according to a camera order in a linear direction or in a rotation direction with respect to the target object.

In the display device 330, the image object may be output in the order of frames having all frame orders or reference intervals according to the time taken by the camera, on the axis having the photographing time.

The computer device 320 combines the frames corresponding to the image object selected in the input device according to the selected order to generate a multi-view image. Furthermore, the user may determine the order of the image frames used by the user before generating the re-point, irrespective of the order in which the user first selects the images using the image matrix interface. As shown in FIG. 6 (b), not the sequence of (1) + (2) -> (3) + (4) + (5), but the sequence of (3) + (4) +

It should be noted that the present embodiment and the drawings attached hereto are only a part of the technical idea included in the above-described technology, and those skilled in the art will readily understand the technical ideas included in the above- It is to be understood that both variations and specific embodiments which can be deduced are included in the scope of the above-mentioned technical scope.

5: object 30: pedestal device
50: camera 80: control device

Claims (10)

The plurality of cameras disposed at different positions photographing a moving picture for the same target object;
Receiving the moving picture by a computer device connected to the plurality of cameras;
Classifying the moving picture according to a position and a photographing time of the camera on a frame-by-frame basis;
Outputting, in a form of a two-dimensional matrix having a first axis and a second axis, an image object corresponding to a frame of the moving picture classified by the display device;
Receiving a command for the computer device to select a plurality of the image objects displayed in the two-dimensional matrix; And
And generating a multi-view image by synthesizing a series of frames corresponding to the selected image object in order according to the instruction,
Wherein the first axis represents the order of the cameras adjacent to the point of time when the target object is photographed and the second axis represents the time flow of the camera photographed by the camera. The method according to claim 1,
Wherein the image object is a thumbnail image object for the frame or a separate image object representing the frame. The method according to claim 1,
And the second axis outputs the image object in a frame order having a frame order or a reference interval in accordance with a time sequence taken by the camera. delete The method according to claim 1,
The command to select
Wherein the computer device is input as a touch input on the display device or as an input from an input means connected to the computer device. The method according to claim 1,
When the computer device receives the command to select
A method of generating a multi-view image using an image matrix interface in a computer device displaying the image object selected by the command in the display device as a separate color or another object, or displaying a continuous line on the selected image object . A plurality of cameras disposed at different positions and photographing a moving picture for the same object;
And controlling the display device to output an image object corresponding to the classified moving image frame to the display device, and to display the image object corresponding to the frame of the moving image, A computer device for synthesizing selected frames in order according to a command input from an input device to generate a multi-view image;
A display device for outputting an image object corresponding to a frame of the moving image classified as a two-dimensional matrix having a first axis and a second axis; And
And an input device for receiving a command for selecting the image object displayed in the two-dimensional matrix,
Wherein the first axis represents the order of the adjacent cameras when the target object is photographed, and the second axis represents the time flow photographed by the camera. delete 8. The method of claim 7,
In the display device
And the second axis outputs the image object in a frame order having a frame order or a reference interval in accordance with a time sequence taken by the camera. 8. The method of claim 7,
The computer device
And generates a multi-view image by synthesizing the frames corresponding to the image object selected by the input device according to the selected order.

Images