WO2017094978A1

WO2017094978A1 - Multiple camera-based image transmission method, device and system

Info

Publication number: WO2017094978A1
Application number: PCT/KR2016/004207
Authority: WO
Inventors: 이재석
Original assignee: 심플렉스 인터넷 주식회사
Priority date: 2015-12-04
Filing date: 2016-04-22
Publication date: 2017-06-08

Abstract

Disclosed are a multiple camera-based image transmission method, device and system. The image transmission method enables: acquiring multiple images by using multiple cameras; displaying a screen displayed on a monitor so as to be segmented into multiple areas corresponding to the multiple images; detecting an area at which a user is gazing; selecting the image that corresponds to the detected area; and transmitting the selected image via broadcasting. Thus, on the basis of multiple images, a broadcast image may be switched according to the gaze of a user.

Description

Image transmission method, apparatus and system based on multiple cameras

The present invention relates to a method, apparatus and system for transmitting images based on a plurality of cameras, and based on a plurality of cameras for photographing a plurality of images through a plurality of cameras and selectively transmitting the captured images according to a user's gaze. An image transmission method, apparatus, and system are provided.

In recent years, with the development of live streaming technology, the user base of personal broadcasting based on the Internet is rapidly increasing. In addition, the Internet personal broadcasting has recently evolved into a format of the public entertainment program, providing various and professional contents.

Individually operated single-person broadcasts can be compared to airwaves in many ways, one of which is the number of cameras operated. In other words, the number of cameras used is how many cameras are used for shooting a broadcast program.

In general, since a single person broadcasts one person who leads the broadcast and there is no cameraman, only one or a few web cameras fixed on the monitor can be used. Therefore, the area where the web camera captures is fixed, so there is a limit in dynamically delivering the scene or situation that the user wants to show.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problem, and a plurality of camera-based image transmission methods, apparatuses capable of capturing a plurality of images based on a plurality of cameras, and selecting and transmitting images to be transmitted according to a user's gaze and The purpose is to provide a system.

In order to achieve the above object, the present invention provides an image transmission method based on a plurality of cameras in one aspect. An image transmission method based on the plurality of cameras includes: an image transmission method performed by an image transmission apparatus, the method comprising: acquiring a plurality of images using a plurality of cameras; Dividing and displaying a screen displayed on a monitor into a plurality of areas corresponding to the plurality of images; Detecting an area gazing at a user; Selecting an image corresponding to the detected area in the sensing step; And transmitting the selected video through broadcast.

The plurality of cameras includes a main camera and at least one sub camera, and the plurality of areas may include a main area displaying an image photographed by the main camera; And at least one sub area displaying at least one image captured by the at least one sub camera.

The selecting of the image may include selecting an image corresponding to an area gazed by the user from among a plurality of areas corresponding to the plurality of images based on the detected area in the detecting step. have.

The selecting of the image may include selecting a predetermined image among the plurality of images when it is detected that the user gazes at a region other than the plurality of regions in the sensing step.

The plurality of cameras, the main camera provided in the upper center of the monitor; A first sub camera provided at an upper left end of the monitor; A second sub camera provided at an upper right end of the monitor; A third sub camera provided at a lower right end of the monitor; And a fourth sub camera provided at the lower left end of the monitor.

The plurality of areas may be displayed in the center of the screen of the monitor, the main area displaying an image acquired from the main camera; A first sub area displayed on an upper left end of a screen of the monitor and displaying an image acquired from the first sub camera; A second sub area displayed at an upper right end of a screen of the monitor and displaying an image acquired from the second sub camera; A third sub area displayed at a lower right end of a screen of the monitor and displaying an image acquired from the third sub camera; And a fourth sub area displayed on a lower left end of the screen of the monitor and displaying an image acquired from the fourth sub camera. The position of each camera may correspond to the position of each of the areas displayed on the screen of the monitor.

At least one of the plurality of cameras may be fixed by a fixing device provided at one side of the monitor or the studio. The fixing device may be fixed to one side of the monitor or studio; An extension part formed to extend from the fixing part; And a support part provided at one side of the extension part to support the camera.

On the other hand, in order to achieve the above object of the present invention the present invention provides an image transmission apparatus based on a plurality of cameras in another aspect. An image transmitting apparatus based on the plurality of cameras includes: an image receiving unit configured to receive a plurality of images acquired by several cameras; A division display unit for dividing and displaying a screen displayed on a monitor into a plurality of areas corresponding to the plurality of images; A gaze detector configured to sense an area gazed by a user; An image selecting unit which selects an image corresponding to an area detected by the line of sight detecting unit; And a transmitter for transmitting the image selected by the image selector through broadcast.

The plurality of cameras may include a main camera and at least one sub camera. The plurality of areas may include: a main area displaying an image photographed by the main camera; And at least one sub area displaying at least one image captured by the at least one sub camera.

The image selector may select an image corresponding to an area gazed by the user from among a plurality of areas corresponding to the plurality of images based on the detected area. The image selector may select a predetermined image from among the plurality of images when the gaze detector detects that the user gazes at a region other than the plurality of regions.

Meanwhile, in order to achieve the object of the present invention, the present invention provides an image transmission system based on a plurality of cameras. An image transmission system based on the plurality of cameras includes: a plurality of cameras for acquiring a plurality of images; And receiving a plurality of images acquired by the plurality of cameras, dividing a screen displayed on a monitor into a plurality of regions corresponding to the plurality of images, detecting an area gazed by a user, and detecting the detected region. And an image transmission device for selecting an image corresponding to an area and transmitting the selected image through a broadcast.

The image transmission device may select an image corresponding to an area gazed by the user from among a plurality of areas corresponding to the plurality of images based on the detected area. When it is detected that the user gazes at a region other than the plurality of regions, the image transmission apparatus may select a predetermined image among the plurality of images.

On the other hand, in order to achieve the above object of the present invention the present invention provides an image transmission system in another aspect. The image transmission system includes a plurality of cameras for acquiring a plurality of images; And receiving a plurality of images acquired by the plurality of cameras, dividing a screen displayed on a monitor into a plurality of areas corresponding to the plurality of images, and detecting a user's gaze in real time. It may include an image transmission device for changing the image to be transmitted through the broadcast in accordance with the change in the area gaze at the user.

On the other hand, in order to achieve the above object of the present invention, the present invention provides an image transmission apparatus in another aspect. The image transmission device is an image transmission device interoperating with a plurality of cameras for acquiring a plurality of images, and receives a plurality of images acquired by the plurality of cameras, and corresponds to a plurality of images that are displayed on a monitor. A first processing unit dividing the display into a plurality of regions; And a second processor configured to change an image transmitted through a broadcast according to a change in a region gazed by the user in the plurality of regions based on the detection of the gaze of the user in real time.

As described above, according to the present invention, a plurality of images may be photographed based on a plurality of cameras, and an image to be transmitted may be automatically selected and transmitted according to a user's gaze. For example, the transmission image may be automatically switched in real time according to the gaze of the user who performs the broadcast. Thus, more dynamic broadcasting is possible in personal broadcasting that is difficult to have expensive camera equipment or transmission system.

1 is a block diagram showing the configuration of a system for realizing a video transmission method using a plurality of cameras according to a preferred embodiment of the present invention.

2 is an exemplary view showing a mounting position of a plurality of cameras according to a preferred embodiment of the present invention.

3 is a perspective view exemplarily showing a fixing device for fixing a camera.

4 is an exemplary view illustrating a mounting position of a plurality of cameras according to another exemplary embodiment of the present invention.

FIG. 5 is a block diagram showing the configuration of the video transmission device shown in FIG.

FIG. 6 is a flowchart for explaining the flow of the image transmission device shown in FIG. 5 and shows the flow of the image transmission method according to the preferred embodiment of the present invention.

7 is an exemplary view illustrating a screen of a monitor displayed by being divided by a split display unit.

8 and 9 are exemplary diagrams for describing broadcast content moving in real time according to a user's gaze.

Hereinafter, with reference to the accompanying drawings, it will be described in detail a preferred embodiment of the present invention. In the following description of the present invention, the same reference numerals are used for the same elements in the drawings and redundant descriptions of the same elements will be omitted.

As illustrated in FIG. 1, the image transmission apparatus 10 of the system 1 may interwork with a plurality of cameras C1 to Cn by wire or wirelessly. The image transmission apparatus 10 may interwork with a communication network such as the Internet, and broadcast at least one of the images acquired from the plurality of cameras C1 to Cn to the viewing device through the communication network. A broadcast service server may be included between the video transmission device 10 and the viewing device.

The image transmission apparatus 10 may be linked with n cameras C1 to Cn (n is an integer greater than 2). For example, in the preferred embodiment of the present invention, the image transmission device 10 may be linked with five cameras. In this case, the five cameras may include one main camera and four sub cameras. However, this embodiment is not limited and the number and role of the camera may be variously changed. Each camera may acquire an image and transmit the acquired image to the image transmission device 10 via wire or wirelessly.

As shown in FIG. 2, the main camera CM may be provided at one side of the monitor 15, for example, an upper center. The other side of the monitor 15 may be provided with a plurality of sub cameras CS1 to CS4, for example, sub camera 1 CS1, sub camera 2 CS2, sub camera 3 CS3, and sub camera 4 CS4. have. For example, the sub camera 1 CS1 may be provided at the upper left end of the monitor 15. The sub camera 2 CS2 may be provided at the upper right end of the monitor 15. The sub camera 3 CS3 may be provided at the lower right end of the monitor 15. The sub camera 4 CS4 may be provided at the lower left end of the monitor 15.

Each camera may be integrally provided with the monitor 15 or may be fixed to the monitor 15 in a detachable form. On the other hand, each camera may be provided fixed to various positions of the monitor 15 or the studio through the fixing means.

3 is a perspective view exemplarily showing a fixing device for fixing a camera. In the example shown in FIG. 3, the camera is an example of a main camera.

As shown in FIG. 3, the fixing device includes a fixing part 22 fixed to one side of the monitor 15 or the studio, an extension part 24 extending from the fixing part 22, and an end of the extension part 24. It may include a support portion 26 provided to support the main camera (CM).

The fixing part 22 may fix the fixing device to the monitor 15. The fixing part 22 may be formed to be detachable to the monitor 15 or the studio in a compression, forceps, adhesive, or the like manner. The support part 26 may include a unit for mounting the camera module, and may fix and support the camera module.

The extension part 24 may connect them between the fixing part 22 and the support part 26 and allow the main camera CM to extend from a fixed position. The extension part 24 may be flexibly deformed in various lengths and directions to support the main camera CM having various shooting angles.

4 is an exemplary view illustrating a mounting position of a plurality of cameras according to another exemplary embodiment of the present invention, and shows an example of mounting the sub cameras to the studio instead of the monitor 15.

As shown in FIG. 4, the main camera CM may be provided at an upper center of the monitor 15. On the other hand, the four sub cameras CS1 to SS4 are installed on the front wall of the studio. For example, the sub camera 1 CS1 may be provided at the upper left end of the studio wall. The sub camera 2 CS2 may be provided at the upper right end of the wall of the studio. The sub camera 3 CS3 may be provided at the lower right end of the wall of the studio. The sub camera 4 CS4 may be provided at the lower left end of the wall of the studio. As such, as the sub-cameras CS1 to CS4 are installed on the wall of the studio as illustrated in FIG. 4, a wider range of images may be acquired.

FIG. 5 is a block diagram showing the configuration of the video transmission device 10 shown in FIG. FIG. 6 is a flowchart for explaining the flow of the image transmission device 10 shown in FIG. 5 and shows the flow of the image transmission method according to the preferred embodiment of the present invention.

As shown in FIG. 5, the image transmission device 10 includes an image receiver 101, a division display unit 102, a gaze detector 103, an image selector 104, a transmitter 105, and a monitor 15. ) May be included. The image transmission device 10 may be linked with a plurality of cameras, for example, a main camera CM and four sub cameras CS1 to CS4. On the other hand, the image transmission device 10 may provide a broadcast service based on images captured from a plurality of cameras, for example, a main camera CM and four sub cameras CS1 to CS4 through a communication network. .

5 to 6, the image receiving unit 101 may acquire a plurality of images using a plurality of cameras CM, CS1 to CS4 (step: S1). For example, the image receiving unit 101 may include a main image from a main camera CM, a sub image 1 from a sub camera 1 (CS1), a sub image 2 from a sub camera 2 (CS2), and a sub image from a sub camera 3 (CS3). 3, sub-image 4 can be acquired from sub-camera 4 (CS4).

The division display unit 102 may divide and display the screen displayed on the monitor 15 into a plurality of areas corresponding to the plurality of images (step S2). For example, the division display unit 102 may include a main area for displaying a main image acquired from the main camera CM, a sub area 1 for displaying a sub image 1 obtained from the sub camera 1 CS1, and a sub camera 2 (CS2). Sub-region 2 for displaying sub-image 2 acquired from the sub-domain 3, and the sub-image 4 for obtaining the sub-image 3 acquired from the sub-camera 3 (CS3). The screen of the monitor 15 may be divided and displayed into five areas, such as the sub area 4.

The division display unit 102 may match the arrangement on the screen of the monitor 15 of the plurality of display areas in response to the actual physical arrangement of the plurality of cameras CM, CS1 to CS4. That is, the split display unit 102 may arrange the divided display area so that the user can intuitively recognize which camera is photographed by looking at the arrangement of the divided areas on the screen of the monitor 15.

FIG. 7 is an exemplary diagram illustrating a screen of the monitor 15 divided and displayed by the division display unit 102.

As shown in FIG. 7, for example, the main area PM for displaying the main image acquired from the main camera CM is displayed at the center of the screen, and corresponds to the sub-camera 1 CS1 which is the upper left camera. Sub-area 1 (PS1) is displayed on the upper left of the screen, sub-region 2 (PS2) corresponding to sub-camera 2 (CS2), which is the upper right camera, is displayed on the upper-right side of the screen, and sub-camera 3, which is the lower-right camera Sub area 3 (PS3) for displaying sub image 3 corresponding to (CS3) is displayed on the lower right side of the screen, and the sub area for displaying sub image 4 acquired from sub camera 4 (CS4), which is a lower left camera. 4 (CS4) can be displayed on the lower left of the screen.

To this end, the split display unit 102 may provide a user interface that allows the user to set the position and layout of the display area, and automatically arrange the display area based on the arrangement of the plurality of cameras (CM, CS1 to CS4). You can also set (PM, PS1 to PS4).

For example, in the former case, when the plurality of cameras CM, CS1 to CS4 are connected, the split display unit may recognize the number and identification information of the connected cameras and display the connected camera list through the display area setting user interface. The display area setting user interface may display a real time sample image corresponding to each camera. The area setting user interface may display various arrangements of display areas corresponding to the number of cameras and display a setting window for allowing a user to select a desired arrangement. When the selection of the arrangement is input through the always setting window, the division display unit 102 may set the camera arrangement in response to the input selection and store the setting data in the database.

On the other hand, in the latter case, the image transmission device 10 may be provided with a wide-angle imaging camera. The division display unit 102 may receive an image of a wide area photographed through a wide angle photographing camera. When the plurality of cameras CM and CS1 to CS4 are connected, the division display unit 102 receives a plurality of images corresponding to the plurality of cameras CM, and compares the images acquired by the respective cameras with the images of the optical region previously received. The image acquired by the camera may be analyzed to which part of the image of the light region, respectively.

Based on the comparison and matching, the division display unit 102 may estimate the position of each camera and automatically set the position of the display area corresponding to each camera according to the estimated position. If the image of the light region and the image acquired from each camera are not properly matched, the division display unit 102 may switch to the manual setting mode to display the area setting user interface.

In a state in which a plurality of images acquired from the plurality of cameras CM and CS1 to CS4 are displayed through the screen of the monitor 15, the line of sight detector 103 tracks the line of sight of the user to detect an area gazed by the user. (Step S3). The gaze detector 103 may include, for example, an eye tracking sensor provided at one side of the monitor 15. The location of the eye tracking sensor may be variously installed according to the implementation environment. For example, at least one of the main camera CM and the sub cameras CS1 to CS4 may also function as an eye tracking sensor.

When the gaze detection unit 103 detects an area gazed by the user, the image selection unit 104 may select an image corresponding to the area detected by the gaze detection unit 103 as being stared at present ( Step: S4). For example, when it is detected that the user is currently staring at the main area PM on the screen of the monitor 15, the image selector 104 may display an image displayed at the main area PM, that is, the main camera CM. ) Can be used to select an image to be acquired. Alternatively, when it is detected that the user is staring at the sub area 1 (PS1), the image selector 104 may select an image displayed in the sub area 1 (PS1), that is, an image acquired from the sub camera 1 (CS1). have.

On the other hand, when the gaze detection unit 103 detects that the user stares at an area other than the plurality of display areas PM, PS1 to PS4, for example, the outside and the rear of the monitor 15, or when the gaze detection is impossible. The image selector 104 may select a predetermined image from the plurality of images. For example, when the main image is set as the default image and the user stares, looks to the side, or looks behind the monitor 15, the image selector 104 may select the main image. The image selector 104 may display a user interface for setting the default image in advance, and set the default image in response to a signal selected through the user interface.

The transmitter 105 may broadcast the image selected by the line of sight detector 103 through a communication network. Accordingly, the viewing devices that receive and reproduce the image serviced by the image transmission apparatus 10 may view the image acquired from the camera corresponding to the display area where the user currently gazes.

As illustrated in FIG. 8, assuming that a user gazes at the main area PM among the five display areas PM and PS1 to PS4 displayed on the monitor 15, the image transmission apparatus 10 may have a main area. The video acquired from the main camera CM corresponding to PM is broadcast.

As illustrated in FIG. 9, when the user's gaze moves from the main area PM to the sub area 1 PS1, the image transmission device 10 may select and broadcast an image acquired from the sub camera 1 CS1. Can be. Therefore, the transmission of the image acquired from the main camera CM is stopped, and the image acquired from the sub camera 1 CS1 is provided to the viewing device through broadcasting.

As described above, according to the present invention, a plurality of images may be acquired based on a plurality of cameras, and a transmission image may be automatically converted in real time according to the eyes of a user who performs a broadcast. This enables more dynamic broadcasting in personal broadcasts that are difficult to have expensive camera equipment or transmission systems.

While the present invention has been described above by way of example thereof, those skilled in the art will variously modify and change the present invention without departing from the technical matters and scope of the present invention as set forth in the claims below. It will be understood that it can be done. Accordingly, modifications to future embodiments of the present invention will not depart from the technology of the present invention.

Claims

In the image transmission method performed by the image transmission device,

Acquiring a plurality of images using a plurality of cameras;

Dividing and displaying a screen displayed on a monitor into a plurality of areas corresponding to the plurality of images;

Detecting an area gazing at a user;

Selecting an image corresponding to the detected area in the sensing step; And

And transmitting the selected image through a broadcast.
The method of claim 1, wherein the plurality of cameras comprises a main camera and at least one sub camera.

The plurality of areas,

A main area displaying an image captured by the main camera; And

And at least one sub-area displaying at least one image captured by the at least one sub-camera.
The method of claim 2, wherein the selecting of the image comprises:

Selecting an image corresponding to an area gazed by the user from among a plurality of areas corresponding to the plurality of images, based on the detected area in the detecting step. Video transmission method.
The method of claim 2, wherein the selecting of the image comprises:

And if it is detected in the detecting step that the user gazes at a region other than the plurality of regions, selecting a predetermined image from the plurality of images.
The method of claim 1, wherein the plurality of cameras,

A main camera provided at an upper center of the monitor;

A first sub camera provided at an upper left end of the monitor;

A second sub camera provided at an upper right end of the monitor;

A third sub camera provided at a lower right end of the monitor; And

And a fourth sub camera provided at a lower left end of the monitor.
The method of claim 5, wherein the plurality of areas,

A main area displayed at the center of a screen of the monitor and displaying an image acquired from the main camera;

A first sub area displayed on an upper left end of a screen of the monitor and displaying an image acquired from the first sub camera;

A second sub area displayed at an upper right end of a screen of the monitor and displaying an image acquired from the second sub camera;

A third sub area displayed at a lower right end of a screen of the monitor and displaying an image acquired from the third sub camera; And

A fourth sub area displayed on the lower left of the screen of the monitor and displaying an image acquired from the fourth sub camera;

And a position of each of the cameras and a position of each of the regions displayed on the screen of the monitor.
According to claim 1, At least one of the plurality of cameras are fixed by a fixing device provided on one side of the monitor or studio,

The fixing device

A fixing part fixed to one side of the monitor or studio;

An extension part formed to extend from the fixing part; And

And a support part provided on one side of the extension part to support the camera.
An image receiver configured to receive a plurality of images acquired by a plurality of cameras;

A division display unit for dividing and displaying a screen displayed on a monitor into a plurality of areas corresponding to the plurality of images;

A gaze detector configured to sense an area gazed by a user;

An image selecting unit which selects an image corresponding to an area detected by the line of sight detecting unit; And

And a transmitter configured to transmit an image selected by the image selector through a broadcast.
The method of claim 8, wherein the plurality of cameras comprises a main camera and at least one sub camera,

The plurality of areas,

A main area displaying an image captured by the main camera; And

And at least one sub-area displaying at least one image captured by the at least one sub-camera.
The method of claim 9, wherein the image selector,

Based on the detected area, the image transmission apparatus based on a plurality of cameras, characterized in that for selecting the image corresponding to the area gazes by the user from among a plurality of areas corresponding to the plurality of images.
The method of claim 8, wherein the image selector,

And when the user detects that the user gazes at an area other than the plurality of areas by the gaze detection unit, a predetermined image is selected from the plurality of images.
The method of claim 8, wherein the plurality of cameras,

A main camera provided at an upper center of the monitor;

A first sub camera provided at an upper left end of the monitor;

A second sub camera provided at an upper right end of the monitor;

A third sub camera provided at a lower right end of the monitor; And

And a fourth sub-camera provided at the lower left end of the monitor.
The method of claim 12, wherein the plurality of areas,

A main area displayed at the center of a screen of the monitor and displaying an image acquired from the main camera;

A first sub area displayed on an upper left end of a screen of the monitor and displaying an image acquired from the first sub camera;

A second sub area displayed at an upper right end of a screen of the monitor and displaying an image acquired from the second sub camera;

A third sub area displayed at a lower right end of a screen of the monitor and displaying an image acquired from the third sub camera; And

A fourth sub area displayed on the lower left of the screen of the monitor and displaying an image acquired from the fourth sub camera;

And a position of each of the cameras and a position of each of the regions displayed on the screen of the monitor.
The method of claim 8, wherein at least one of the plurality of cameras is fixed by a fixing device provided on one side of the monitor or studio,

The fixing device

A fixing part fixed to one side of the monitor or studio;

An extension part formed to extend from the fixing part; And

And a support part provided on one side of the extension part to support the camera.
A plurality of cameras for acquiring a plurality of images; And

Receives a plurality of images acquired by the plurality of cameras, divides and displays a screen displayed on the monitor into a plurality of regions corresponding to the plurality of images, detects an area gazed by a user, and detects the detected region. And a video transmission device for selecting a video corresponding to the video transmission and transmitting the selected video through broadcast.
The method of claim 15, wherein the plurality of cameras comprises a main camera and at least one sub camera.

The plurality of areas,

A main area displaying an image captured by the main camera; And

And at least one sub-area displaying at least one image captured by the at least one sub-camera.
The apparatus of claim 16, wherein the image transmission device is

Based on the detected area, the image transmission system based on a plurality of cameras, characterized in that for selecting the image corresponding to the area gazed by the user from among a plurality of areas corresponding to the plurality of images.
The apparatus of claim 15, wherein the image transmission device is

And if it is detected that the user gazes at an area other than the plurality of areas, the image transmission system, wherein the predetermined image is selected from the plurality of images.
The method of claim 15, wherein the plurality of cameras,

A main camera provided at an upper center of the monitor;

A first sub camera provided at an upper left end of the monitor;

A second sub camera provided at an upper right end of the monitor;

A third sub camera provided at a lower right end of the monitor; And

And a fourth sub-camera provided at the lower left end of the monitor.
An image transmission apparatus interoperating with a plurality of cameras for acquiring a plurality of images,

A first processing unit which receives a plurality of images acquired by the plurality of cameras and divides and displays a screen displayed on a monitor into a plurality of regions corresponding to the plurality of images; And

And a second processor configured to change an image transmitted through a broadcast in response to a change in an area gazed by the user in the plurality of areas based on the detection of the gaze of the user in real time. .