KR20190102871A - Transmitting system for multi channel image and controlling method, and image processing method of multi channel image - Google Patents

Abstract

According to an embodiment of the present invention, a transmission system for multi-channel images comprises: a camera control unit controlling a subject including a plurality of standard points to be photographed during a predetermined time through a plurality of cameras and transmitting multi-channel images corresponding to the plurality of cameras through a communication network; and an image server setting standard points with regard to at least one image among the transmitted multi-channel images, a rotation point and a focus point corresponding to the standard points, estimating standard points, rotation points, and focus points of the residual images based on the set standard points, rotation point, and focus point, and transmitting the multi-channel images through the communication network in accordance with a request of a user terminal.

Description

Transmitting system for multi channel image and controlling method, and image processing method of multi channel image}

Embodiments relate to a transmission system for a multichannel image, a control method thereof, and a multichannel image processing method.

Recently, the public prefers to play video using mobile. In line with these preferences, companies are serving broadcast platforms such as V-app, AfreecaTV and Youtube Live. The public viewing these platforms is watching a video taken from a viewpoint, one camera. Recently, however, viewers want to watch videos taken in the desired space.

Currently, a video service that provides a user with a multi-channel image by geometrically correcting and synthesizing a plurality of images obtained by photographing a single subject in various channels with a plurality of cameras is disclosed. The multi-channel video provides a realistic image that goes beyond the concept of high definition, which makes users feel more immersed in the media and can greatly enhance the effect of delivering image information in the fields of advertising, education, medical care, defense, entertainment, etc. have.

In the conventional multichannel video, the channel / time switching is simply reproduced by a predetermined merge method at the time of producing the multichannel video. That is, conventionally, a plurality of frames are acquired from a plurality of cameras, some of the obtained frames are selected, and these frames are merged to produce one channel switching image. Since the channel switching image simply merges the frames of the channel predetermined by the producer at the time of making the image, when the image file is played, the merged frames have a channel switching effect indicating a single channel shift effect. According to such a multi-channel video of the prior art, the user merely views a previously produced channel switching effect, and the viewer views the video while turning the channel to a point in time where the viewer wants to play by manually operating the time switching or the channel switching. It was impossible to do.

In addition, multi-channel or camera synchronization techniques, image correction, and transmission methods are required for capturing images in various spaces, and the coordinates of the rotation point and the focus point must be exactly matched in a plurality of images acquired by each camera. There was a technical difficulty.

Embodiments provide a transmission system for a multichannel image, a control method thereof, and a multichannel image processing method.

The transmission system for a multi-channel image according to an embodiment controls to photograph a subject including a plurality of reference points for a predetermined time through a plurality of cameras, and transmits the multi-channel images corresponding to the plurality of cameras through a communication network. A camera controller; And setting reference points for at least one of the transmitted multichannel images, a rotation point and a focus point corresponding to the reference points, and rest on the basis of the set reference points, the rotation point, and the focus point. And an image server for estimating reference points, rotation points, and focus points of the images, and transmitting the multi-channel images through the communication network at the request of a user terminal.

The image server may transmit the multichannel images including the rotation point and the estimated rotation points of the multichannel images, the focus points and the estimated focus points of the multichannel images.

The video server groups and stores the multi-channel images by at least one criterion of a time, a channel, a time and a channel, and transmits the grouped images. Mix images of the first to Mth channels (M is two or more natural numbers) corresponding to the first to Nth times (N is a natural number of two or more), or the M to first channels corresponding to the first to Nth time. Characterized by mixing the images.

The video server groups and stores the multi-channel images by at least one criterion of a time, a channel, a time and a channel, and transmits the grouped images. Mixing the images of the first to M channel corresponding to the N to the first time, or mixing the images of the M to first channel corresponding to the N to the first time.

Grouping images corresponding to ± y (y is a natural number) hourly channel based on an event corresponding to any one of the predetermined time periods.

The camera controller transmits a photographing synchronization signal and a photographing driving control signal so that the plurality of cameras simultaneously photograph the subject, and compresses the multi-channel images corresponding to the plurality of cameras transmitted from the plurality of cameras. And a transmittable data transmission method according to the communication network.

The video server sequentially decodes the grouped video from the first channel when an event corresponding to the request of the user terminal is generated in a predetermined channel among the first to Mth channels.

The image server may perform rendering on the decoded images in a direction in which the event occurs from the channel where the event occurs.

A control method of a transmission system for a multi-channel image according to another embodiment includes controlling to photograph a subject including a plurality of reference points for a predetermined time through a plurality of cameras; Transmitting multi-channel images corresponding to the plurality of cameras through a communication network; Setting reference points, at least one rotation point corresponding to the reference points, and a focus point for at least one of the transmitted multichannel images; Estimating reference points, rotation points, and focus points of the remaining images based on the set reference points, rotation points, and focus points; And transmitting the multi-channel images through the communication network at the request of the user terminal.

According to another exemplary embodiment, there is provided a method of processing a multi-channel image, including: obtaining a multi-channel image by photographing a subject including a plurality of reference points for a predetermined time through a plurality of cameras; Setting a rotation point corresponding to reference points of any one channel image of the acquired multichannel images and a focus point adjacent to the rotation point; Estimating rotation points and focus points for the remaining channel images based on the set rotation point and focus point; And storing the multi-channel images including the set rotation point and the estimated rotation points and the set focus point and the estimated focus points.

Another embodiment includes a recording medium recording a method for executing the method on a computer.

The transmission system for a multi-channel image according to an embodiment may increase the convenience of data transmission of the multi-channel image and reduce the processing speed when moving space over time.

In addition, the transmission system for the multi-channel image, the control method and the multi-channel image processing method according to the embodiment can accurately estimate the coordinates of the rotation position and the focus position of the image for each channel.

1 is a schematic diagram of a transmission system 100 for a multi-channel image, according to an exemplary embodiment.
2 illustrates an example of generating a multichannel image.
3A through 3C are exemplary diagrams for describing reference points and rotation points in images acquired by a plurality of cameras.
4 is a schematic diagram of the image server 200 shown in FIG. 1.
5 is a schematic diagram of the coordinate setting unit 131 shown in FIG. 4.
6 to 8 are exemplary diagrams for describing setting a rotation point and a focus point according to an exemplary embodiment.
9 is a diagram for describing a control method of a transmission system for a multi-channel image, according to another exemplary embodiment.
10 is a diagram for describing a multichannel image processing method, according to another exemplary embodiment.

The terminology used in the present embodiments is a general term that is currently widely used as possible while considering the functions of the present embodiments, but may vary according to the intention or precedent of a person skilled in the art, the emergence of new technology, etc. . In addition, in certain cases, there is also a term arbitrarily selected, and in this case, the meaning will be described in detail in the description of the corresponding embodiment. Therefore, the terms used in the present exemplary embodiments should be defined based on the meanings of the terms and the contents throughout the present exemplary embodiments, rather than simply names of the terms.

In the description of the embodiments, when a part is connected to another part, this includes not only a case in which the part is directly connected, but also a case in which another part is electrically connected in between. In addition, when a part includes a certain component, this means that it may further include other components, without excluding other components unless otherwise stated. In addition, the term "... unit" described in the embodiments means a unit for processing at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

Terms such as “consisting of” or “comprising” as used in the present embodiments should not be construed as necessarily including all of the various elements or steps described in the specification, and some or some of them may be included. It is to be understood that the steps may not be included or may further include additional components or steps.

The description of the following embodiments should not be construed as limiting the scope of the right, it should be construed as belonging to the scope of the embodiments that can be easily inferred by those skilled in the art. Hereinafter, only exemplary embodiments will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a transmission system 100 for a multi-channel image, according to an exemplary embodiment.

Referring to FIG. 1, the transmission system 100 controls a plurality of cameras 111 to 113, a plurality of cameras 111 to 113, and processes multi-channel images captured from the plurality of cameras 111 to 113. It includes a camera control unit 110 for transmitting, and an image server 200 for processing and storing the multi-channel image transmitted from the camera control unit 110. The image server 200 receives a request for a multi-channel image, a request for a switching image, a request for a specific event from the user terminal 150, and transmits the stored multi-channel image to the user terminal 150. The image server 200 may include an image processing apparatus 130 and an image storage unit 140.

As shown in FIG. 2, a plurality of cameras 1 to N may be arranged to photograph a specific subject. A plurality of cameras receiving the subject from multiple angles are received by a plurality of cameras arranged in an array around the subject. The plurality of camera arrangement methods may be, for example, arranging N cameras in a line on substantially the same plane with respect to any camera. In this case, N cameras may be sequentially arranged on a circumference spaced by a certain distance from the subject, and the distances between the two cameras disposed on both sides and the subject may be matched with respect to the camera disposed at the center. The distance to the subject may be different. In addition, the subject may be a fixed subject or a moving subject.

The plurality of cameras 1 to N and the camera controller 110 may be capable of communicating by wire or wirelessly, and may include a plurality of camera controllers for controlling the plurality of cameras 1 to N. FIG.

The camera controller 110 may control the plurality of cameras 1 to N through a synchronization signal for synchronizing the plurality of cameras 1 to N. The camera controller 110 temporarily stores the images photographed from the plurality of cameras 1 to N, and reduces the size of the photographed image by changing the codec and enables fast transmission. The camera controller 110 controls to photograph a subject including four reference points for a predetermined time through the plurality of cameras 1 to N, and records the multi-channel images corresponding to the plurality of cameras through a communication network through the communication network. To transmit. The reference point indicates an identification mark for recognizing the same position in the captured images including the subjects photographed by the plurality of cameras. The reference point may have a rectangular shape including at least four reference coordinates, but is not limited to the number or shape. In order to recognize the reference coordinates, the identification member for recognizing the reference point may be positioned in the area including the subject before the photographing.

The image server 200 sets reference points for at least one of the multi-channel images transmitted from the camera controller 110, a rotation point located in the reference points, and a focus point. Here, the rotation point or the focus point is described as being located in the reference points, but is not limited thereto and may be located outside the reference points. Also, reference points, rotation points, and focus points of the remaining images are estimated based on the set reference points, rotation points, and focus points.

The image server 200 transmits multi-channel images including a rotation point and a focus point through a communication network at the request of the user terminal. In addition, the image server 200 groups and stores multi-channel images including a rotation point and a focus point by at least one criterion of a mixture of time, channel, time, and channel, and stores the grouped images according to a request of a user terminal. Transmission to the user terminal 150 through a communication network.

3A through 3C are exemplary diagrams for describing reference points and rotation points in images acquired by a plurality of cameras.

FIG. 3A is an image captured by the camera 1, and four reference points P1, Q1, R1, and S1 and a rotation point A1 are illustrated. FIG. 3B is an image captured by the camera 2, and four reference points P2, Q2, R2, and S2 and a rotation point A2 are illustrated. 3C is an image taken by the camera 3, and four reference points P3, Q3, R3, and S3 and a rotation point A3 are illustrated. As shown in FIGS. 3A to 3C, in the images photographed by the respective cameras, an accurate view may be identified through four reference points and rotation points. In other words, the four reference points (P1, Q1, R1, S1) and the rotation point (A1) of the image captured by the camera 1, and the four reference points (P2, Q2, R2, S2) of the image captured by the camera 2 The rotation point A2, the four reference points P3, Q3, R3, and S3 and the rotation point A3 of the image photographed by the camera 3 have the same coordinates in the actual subject area.

In an embodiment, four reference points (P1, Q1, R1, S1) and rotation point A1 of an image captured by camera 1 are preset, and the reference points in cameras 2 and 3 are reference points in camera 1 Based on the positional relationship between the camera 1 and the camera 2 and 3, for example, it can be calculated according to the installation coordinates of the camera 1 to 3 shown in Figure 2, and set the reference point in the images respectively captured by the camera 2 and 3 It may be.

In addition, when the rotation point A1 in the camera 1 is set in advance, the rotation point A2 in the camera 2 is located in the four reference points P1, Q1, R1, and S1 in the camera 1, and the camera 2 It can be estimated according to the coordinates located in the four reference points P2, Q2, R2, and S2 at Similarly, the rotation point A3 in camera 3 can also be estimated according to the coordinates located in the reference points of camera 1 or camera 2 and the position coordinates in the four reference points P3, Q3, R3, and S3 in camera 3. .

4 is a schematic diagram of the image server 200 shown in FIG. 1.

Referring to FIG. 4, the image server 200 includes an image processing apparatus 130 and an image storage unit 140, and the image processing apparatus 130 includes a coordinate setting unit 131, an image processing unit 132, and an image. The converter 133 and the transmitter 134 are included.

The coordinate setting unit 131 sets reference points, rotation points, and focus points for the multichannel images. In an embodiment, the coordinate setting unit 131 sets a reference point for one of the transmitted multi-channel images, and sets a rotation point or a focus point located within the reference point. The coordinate setting unit 131 estimates reference points, rotation points, and focus points of the remaining images based on the set reference points, rotation points, and focus points. 5 is a schematic diagram of the coordinate setting unit 131 shown in FIG. 4.

The coordinate setting unit 131 includes a reference point setting unit 510, a rotation point setting unit 520, a focus point setting unit 530, and a point coordinate estimating unit 540.

The reference point setting unit 510 sets reference points for the multichannel images. As shown in FIGS. 3A to 3C, reference points may be set for images captured by the cameras 1 to 3, and only reference points for images captured by the camera 1 illustrated in FIG. 3A may be set. In the reference point setting, the operator may input a reference point through a user interface provided in an operator terminal (not shown) of the image server 200, and automatically recognize the identification member in the captured image and then automatically set the reference points. Can also be set. 3A to 3C, the reference points are illustrated in four rectangular shapes, but various modifications are possible without being limited to the number or shape of the reference points.

The rotation point setting unit 520 sets a rotation point with respect to any one of the multi-channel images. When the rotation point is located within the reference points and the rotation point is set for one image, the rotation point of the remaining image may be estimated. In the rotating embodiment, the rotation point setting may be input through an operator terminal (not shown), or may be set by the user terminal 150.

The focus point setting unit 630 sets a focus point for any one of the multi-channel images. The focus point may be a point adjacent to the rotation point and may be a focus area of the image. Based on the rotation point and the focus point set for one image, the rotation point and the focus point for the other image may be estimated. Here, the focus point may be a specific point or may be a specific area including the focus point.

The point coordinate estimator 540 is configured to display the remaining images based on the reference points set by the reference point setter 510, the rotation point set by the rotation point setter 520, and the focus point set by the focus point setter 540. Reference points, rotation points and focus points are estimated.

3A to 3C, a method of estimating the rotation point will be described. It is assumed that the conversion matrix H1 between the image photographed by the camera 1 and the image photographed by the camera 2, and the conversion matrix H2 between the image photographed by the camera 2 and the image photographed by the camera 3 are equal to the following equations (1) and (2).

[Equation 1]

[Equation 2]

인 경우, 카메라 2에서 촬영된 영상의 회전 포인트(A2)는

이고, 카메라 3에서 촬영된 영상의 회전 포인트(A3)는

이다. 이와 같은 방식으로, 각각의 카메라에서 촬영된 영상의 포커스 포인트도 회전 포인트와의 이격된 방향 및 거리를 기초로 추정할 수 있다. The coordinates of the rotation point A1 of the image captured by the camera 1 shown in FIG.

, The rotation point A2 of the image captured by the camera 2

The rotation point A3 of the image captured by the camera 3 is

to be. In this manner, the focus point of the image photographed by each camera may also be estimated based on the direction and distance from the rotation point.

6 to 8 are exemplary diagrams for describing setting a rotation point and a focus point according to an exemplary embodiment.

6A and 6B, when the rotation points 601 and 603 and the focus points 602 and 604 of a specific player are set in a basketball game, as shown in FIGS. 6C to 6E, the rotation of the image captured by the camera 1 is performed. Point 605 and focus point 606 are set. Subsequently, rotation points 607 and 609 and focus points 608 and 610 of the images captured by the cameras 2 and 3 are estimated.

Referring to FIGS. 7A and 7B, when the rotation points 701 and 703 and the focus points 702 and 704 of a specific athlete are set in the figure skating competition, as shown in FIGS. 7C to 7E, the image captured by the camera 1 may be used. Rotation point 705 and focus point 706 are set. Subsequently, rotation points 707 and 709 and focus points 708 and 710 of the images captured by the cameras 2 and 3 are estimated.

Referring to FIG. 8A, when a focus point 802 including a rotation point 801 of a specific player and a movement of an adjacent player is set in a basketball game, as illustrated in FIGS. 8B to 8D, the image captured by the camera 1 may be used. Rotation point 803 and focus point 804 are set. Subsequently, rotation points 805 and 807 and focus points 806 and 808 of the image captured by the cameras 2 and 3 are estimated.

In an exemplary embodiment, when a multichannel image is generated using only the rotation point, the viewpoint may not be corrected or an accurate multichannel image may be generated. Can be. In addition, by estimating the focus point based on the rotation point, the rotation points and the focus points of the remaining images may be estimated based on the rotation point of one of the multi-channel images, thereby increasing computation and processing speed.

The image processor 132 performs image correction on the transmitted multichannel images, that is, the images photographed by the plurality of cameras. For example, since images captured by a plurality of cameras may be out of focus, image processing is performed so that the focus is the same between cameras. The image processor 132 corrects the transmitted multichannel images. Since geometric errors of the N camera arrays are visually shaken in the process of reproducing the multi-channel image, at least one of the size or the size, the tilt, or the center position of each image may be corrected in order to remove them.

The image converter 133 groups the multichannel images based on at least one of a mixture of time, channel, time and channel. The image converter 133 groups several spaces into one. The grouping method can be performed according to various criteria. The transmission system according to the embodiment may transmit only the necessary data to the user by transmitting the grouped images without transmitting data, without wasting data by transmitting all the image data in order to transmit an effective multi-channel image or switching image to the user terminal 150. To be. The image converting unit 133 may group and group +/- y (y is a natural number) hourly channel image around the t-time event. For example, in channel 1, the event may occur at t3. Here, the event may be a predetermined case, for example, a home run scene or an out scene in a baseball game, an event at the request of a user, or a case desired by the user.

The image converter 133 groups the multi-channel images by time, channel, or a mixture of time and channel, and stores the multi-channel images in the image storage unit 140. When the user terminal 150 requests, the image processing apparatus 130 extracts the images stored in the image storage unit 140 and transmits the images to the user terminal 150 through the transmission unit 134. Here, the transmission unit 134 may be a streaming device, but will be described as being included in the video server 130, but may be implemented as a separate device separate from the video server 130.

The transmitter 133 transmits the processed image or the stored image in real time. For example, it may be a device for real time streaming. The transmitter 133 transmits a message handler for performing session management and protocol management with the user terminal, an image to the user terminal, a streamer which is a set of images to be transmitted to the user terminal, and receives a signal from the user terminal in units of GOPs. The channel manager may include a channel manager that schedules and delivers the stream to the streamer.

9 is a diagram for describing a control method of a transmission system for a multi-channel image, according to another exemplary embodiment.

Referring to FIG. 9, in operation 900, a control is performed to photograph a subject including a plurality of reference points for a predetermined time through a plurality of cameras. Here, the plurality of reference points may be four, but is not limited to the number. In operation 902, multichannel images corresponding to a plurality of cameras are transmitted through a communication network. In step 904, reference points for at least one of the transmitted multichannel images, a corresponding rotation point and a focus point within the reference points are set. Here, the rotation point and the focus point may be located within or outside the reference points. In operation 906, reference points, rotation points, and focus points of the remaining images are estimated based on the set reference points, the rotation point, and the focus point. In operation 908, the multi-channel images are transmitted through the communication network at the request of the user terminal. In another embodiment, the multi-channel images including the rotation point and the focus point may be grouped and stored according to at least one criterion of time, channel, time and channel, and the grouped images may be stored at the request of the user terminal. It can also be sent via.

10 is a diagram for describing a multichannel image processing method, according to another exemplary embodiment.

Referring to FIG. 10, in operation 1000, multichannel images are obtained by photographing a subject including a plurality of reference points for a predetermined time through a plurality of cameras. In operation 1002, a rotation point corresponding to reference points of any one channel image among the acquired multichannel images and a focus point adjacent to the rotation point are set. In operation 1004, the rotation points and the focus points for the remaining channel images are estimated based on the set rotation point and the focus point. In operation 1006, multichannel images including the set rotation point and the estimated rotation points and the set focus point and the estimated focus points are stored. In another embodiment, the multi-channel images including the rotation point and the focus point may be grouped and stored according to at least one criterion of mixing time, channel, time and channel.

One embodiment may also be implemented in the form of a recording medium that includes instructions executable by a computer, such as a program module executed by the computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transmission mechanism, and includes any information delivery media.

The foregoing description is for illustrative purposes, and it will be understood by those skilled in the art that the present invention may be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is shown by the following claims rather than the above description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present invention. do.

Claims (10)

A camera controller which controls to photograph a subject including a plurality of reference points for a predetermined time through a plurality of cameras and transmits multi-channel images corresponding to the plurality of cameras through a communication network; And
Set reference points for at least one of the transmitted multi-channel images, a rotation point and a focus point corresponding to the reference points, and rest the other images based on the set reference points, the rotation point, and the focus point. And a video server for estimating reference points, rotation points, and focus points, and transmitting the multi-channel images through the communication network according to a request of a user terminal. The method of claim 1,
The video server,
And transmitting the multichannel images including the rotation point and the estimated rotation points of the multichannel images, the focus points and the estimated focus points of the multichannel images. The method of claim 1,
The video server,
Group and store the multi-channel images by at least one criterion of time, channel, time and channel, and transmit the grouped images,
The criterion of mixing the time and channel,
Mix images of the first to Mth channels (N is a natural number of two or more) times corresponding to the first to Nth times (M is two or more natural numbers), or the M to first first to Nth times corresponding to the first to Nth times. A transmission system for multi-channel video, characterized in that for mixing the video of the channel. The method of claim 1,
The video server,
Group and store the multi-channel images by at least one criterion of time, channel, time and channel, and transmit the grouped images,
The criterion of mixing the time and channel,
Multi-channel video, characterized in that for mixing the images of the first to M channel corresponding to the N to the first time, or to the image of the M to the first channel corresponding to the N to the first time. For transmission system. The method of claim 1,
And a group of images corresponding to ± y (y is a natural number) hourly channel based on the event corresponding to any one of the predetermined time periods. The method of claim 1,
The camera control unit,
The photographing synchronization signal and the photographing driving control signal are transmitted so that the plurality of cameras simultaneously photograph the subject, and the compression processing is performed on the multi-channel images corresponding to the plurality of cameras transmitted from the plurality of cameras. The transmission system for a multi-channel image, characterized in that for determining the transmittable data transmission method according to. The method of claim 1,
The video server,
When an event corresponding to a request of the user terminal is generated in a predetermined channel among first to Mth channels, decoding is performed sequentially from the first channel with respect to the grouped video.
The video server,
And transmitting the decoded images in the direction in which the event is generated from the channel where the event is generated. Controlling photographing a subject including a plurality of reference points for a predetermined time through a plurality of cameras;
Transmitting multi-channel images corresponding to the plurality of cameras through a communication network;
Setting reference points, at least one rotation point corresponding to the reference points, and a focus point for at least one of the transmitted multichannel images;
Estimating reference points, rotation points, and focus points of the remaining images based on the set reference points, rotation points, and focus points;
The multi-channel images
Control method of a transmission system for a multi-channel image comprising the step of transmitting through the communication network at the request of a user terminal. Obtaining multichannel images by photographing a subject including a plurality of reference points for a predetermined time through a plurality of cameras;
Setting a rotation point corresponding to reference points of any one channel image of the acquired multichannel images and a focus point adjacent to the rotation point;
Estimating rotation points and focus points for the remaining channel images based on the set rotation point and focus point; And
And storing the multi-channel images including the set rotation point and the estimated rotation points and the set focus point and the estimated focus points. A recording medium on which a method for executing the method according to claim 8 or 9 is recorded on a computer.

Images