KR102149004B1 - Method and apparatus for generating multi channel images using mobile terminal - Google Patents

Abstract

In the method of generating a multi-channel image using a mobile terminal according to an exemplary embodiment, a multi-channel image can be conveniently generated even by general users without building a separate multi-channel image capturing system by using a camera built into the mobile terminal. .

Description

Method and apparatus for generating multi channel images using mobile terminal

The embodiments relate to a method and apparatus for generating a multi-channel image using a mobile terminal.

Recently, the public prefers to play videos using mobile devices. In line with this preference, companies are offering broadcasting platforms such as V-app, AfreecaTV, and Youtube Live. The public watching these platforms are watching videos taken from one viewpoint, that is, one camera. However, recent viewers want to watch the video shot in the desired space.

Currently, a video service that provides a multi-channel image to a user by geometrically correcting and synthesizing a plurality of images obtained by photographing a subject in various channels with a plurality of mobile terminals has been disclosed. These multi-channel images provide realistic images that go beyond the concept of high-definition, and through this, users feel more immersed in the media and can greatly enhance the effect of delivering image information in fields such as advertising, education, medical care, defense, and entertainment. have.

In the conventional multi-channel image, channel/time switching is a dimension that is simply reproduced by a predetermined merge method when producing a multi-channel image. That is, in the related art, a plurality of frames are acquired from a plurality of cameras, some of the acquired frames are selected, and these frames are merged to produce a single channel switching image. Since such a channel-switched video is a simple merge of the frames of a channel previously determined by the producer during video production, when the video file is played back, the merged frames exert a channel switching effect showing a single channel shift effect. According to such a multi-channel video of the prior art, the user is simply watching the previously produced channel switching effect, and the viewer watches the video while turning the channel to the desired point of time by manually manipulating visual switching or channel switching. It was impossible to do. In addition, a multi-channel or camera synchronization technique, and an image correction and transmission method are required to capture an image in various spaces.

Meanwhile, in order to generate the above-described multi-channel image, various transmission platforms and playback devices are required. However, in order to generate a multi-channel image, it was a burden for general users to use it due to the expensive shooting environment construction cost and time required, and there was a limit to the spread of the multi-channel platform.

Embodiments aim to provide a method and apparatus capable of generating a multi-channel image using a mobile terminal with a built-in camera.

According to an embodiment, a method for generating a multi-channel image using a mobile terminal includes an icon corresponding to the plurality of mobile terminals displayed on the display screen of the host mobile terminal using the addresses of each of the plurality of mobile terminals for photographing a subject in the host mobile terminal. Grouping a plurality of mobile terminals for photographing the subject by selecting them; Transmitting a synchronization message for synchronizing time to the plurality of grouped mobile terminals by selecting a synchronization icon corresponding to the group synchronization displayed on the display screen of the host mobile terminal; Setting photographing guide information for photographing the subject on a display screen of the host mobile terminal, and transmitting photographing information extracted from the set photographing guide information to the grouped plurality of mobile terminals; By selecting an icon corresponding to a photographing button on the display screen of the host mobile terminal, transmitting a photographing message including time information for starting photographing of the subject from all mobile terminals to the grouped plurality of mobile terminals; And photographing the subject according to the time information in the host mobile terminal and transmitting the photographed image to an external server.

In the external server, the multi-channel image is generated based on the captured image received from the host mobile terminal and the captured image received from the grouped plurality of mobile terminals.

An augmented reality image obtained by synthesizing an image augmented with augmented reality with user images of a plurality of mobile terminals captured by the host mobile terminal is displayed on the display screen.

The synchronization message is characterized in that it is generated according to the NTP (Network Time Protocol).

The photographing guide information is to display at least one of a top point, an intermediate point, and a bottom point in a straight line around the subject in the host mobile terminal, and the extracted shooting information is applied to the subject according to the shooting guide information. And at least one of tilt information and angle information of the host mobile terminal.

Tilt information and angle information of the host mobile terminal, and tilt information and angle information of the grouped plurality of mobile terminals are transmitted from the external server, and the photographing guide information and the photographing information are transmitted from the external server to all mobile terminals. It is characterized in that it additionally transmits the included metadata.

All mobile terminals are characterized by automatically photographing the subject according to each timer program according to the absolute time information.

In another embodiment, a method of generating a multi-channel image using a mobile terminal includes an icon corresponding to the plurality of mobile terminals displayed on the display screen of the host mobile terminal by using the addresses of each of the plurality of mobile terminals to be photographed by the host mobile terminal. Grouping a plurality of mobile terminals for photographing the subject by selecting them; Transmitting a synchronization message for synchronizing time to the plurality of grouped mobile terminals by selecting a synchronization icon corresponding to the group synchronization displayed on the display screen of the host mobile terminal; Setting photographing guide information for photographing the subject on a display screen of the host mobile terminal, and transmitting photographing information extracted from the set photographing guide information to the grouped plurality of mobile terminals; By selecting an icon corresponding to a photographing button on the display screen of the host mobile terminal, transmitting a photographing message including time information for starting photographing of the subject from all mobile terminals to the grouped plurality of mobile terminals; Generating a photographed image by photographing the subject according to the time information in the host mobile terminal; Receiving a plurality of photographed images photographing the subject from the plurality of grouped mobile terminals; And generating a multi-channel image using the generated photographed image and the plurality of received photographed images.

A mobile terminal according to another embodiment includes a display screen; Shooting module; And a processor, wherein the processor selects icons corresponding to the plurality of mobile terminals displayed on the display screen using the addresses of each of the plurality of mobile terminals for photographing the subject, thereby selecting a plurality of mobile terminals for photographing the subject. By grouping and selecting a synchronization icon corresponding to group synchronization displayed on the display screen, a synchronization message for synchronizing time is transmitted to the plurality of grouped mobile terminals, and photographing guide information for photographing the subject on the display screen Is set, the shooting information extracted from the set shooting guide information is transmitted to the plurality of grouped mobile terminals, and an icon corresponding to a shooting button is selected on the display screen, so that all mobile terminals start shooting a subject. A photographing message including information is transmitted to the plurality of grouped mobile terminals, the photographing module is controlled according to the time information to photograph the subject, and the photographed image is transmitted to an external server.

It includes a recording medium recording a method for executing the method according to another embodiment on a computer.

In the method of generating a multi-channel image using a mobile terminal according to an embodiment, general users can conveniently generate a multi-channel image without building a separate multi-channel image capturing system.

1 is a schematic diagram of a transmission system 100 for a multi-channel image according to an embodiment.
2 is an exemplary diagram for generating a multi-channel image.
3 is a schematic diagram of the mobile terminal 100 shown in FIG. 1.
4 is a system diagram including the mobile terminal 100 shown in FIG. 1.
5 to 7 are exemplary diagrams for explaining a method for generating a multi-channel image using the mobile terminal 100.
8 is a schematic diagram of the group management server 110 shown in FIG. 1.
9 is a schematic diagram of the image server 200 shown in FIG. 1.
10 is another schematic diagram of the image server 200 shown in FIG. 1.
11 and 12 are detailed exemplary diagrams of the transmission unit 133 shown in FIG. 10.
13 is a flowchart illustrating a method of generating a multi-channel image using a mobile terminal according to another embodiment.

The terms used in the embodiments have been selected as currently widely used general terms as possible while considering functions in the embodiments, but this may vary depending on the intention or precedent of a technician working in the art, the emergence of new technologies, etc. . In addition, in certain cases, there are terms that are arbitrarily selected, and in this case, the meaning will be described in detail in the description of the corresponding embodiment. Therefore, the terms used in the present embodiments should be defined based on the meaning of the term and the contents of the present embodiments, not a simple name of the term.

In the description of the embodiments, when a certain part is said to be connected to another part, this includes not only a case in which it is directly connected, but also a case in which it is electrically connected with another component interposed therebetween. In addition, when a part includes a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary. In addition, the term "... unit" described in the embodiments means a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software.

Terms such as “consisting of” or “comprising” used in the present embodiments should not be construed as necessarily including all of the various constituent elements or various steps described in the specification, and some constituent elements or some of them It should be construed that the steps may not be included, or may further include additional components or steps.

The description of the following embodiments should not be construed as limiting the scope of the rights, and what those skilled in the art can easily infer should be construed as belonging to the scope of the rights of the embodiments. Hereinafter, embodiments for illustration only will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a transmission system 100 for a multi-channel image according to an embodiment.

Referring to FIG. 1, the transmission system 100 includes a group management server 110 that processes and transmits multi-channel images captured from a plurality of mobile terminals 111 to 113, a plurality of mobile terminals 111 to 113, and a group It includes an image server 200 that processes and stores multi-channel images transmitted from the management server 110. The video server 200 receives a request for a multi-channel video, a request for a switching video, and a request for a specific event from the user terminal 150, and transmits the stored multi-channel video to the user terminal 150. Here, the user terminal 150 may include a plurality of mobile terminals 111 to 113 or may be different user devices. The plurality of mobile terminals 111 to 113 includes a photographing module, for example, a camera, and refers to a mobile device grouped to photograph a multi-channel image. Here, any one mobile terminal, for example, the mobile terminal 111 may be a host mobile terminal, and the remaining terminals 112 and 113 may be a plurality of mobile terminals grouped to capture a multi-channel image. Each of the mobile terminals 111 to 113 is equipped with an application for photographing a multi-channel image, and photographs the same subject according to the photographing control of the host mobile terminal 111 and transmits the photographing to the group management server 110. In addition, the plurality of mobile terminals 111 to 113 perform, for example, shooting group management, shooting control management, and shooting image transmission under the control of the group management server 110.

The image server 200 may include an image processing device 130 and an image storage unit 140.

As shown in FIG. 2, a plurality of mobile terminals 1 to N may be disposed to photograph a specific subject. A plurality of mobile terminals arranged in an array around the subject receive a plurality of images photographing the subject from various angles. A method of arranging a plurality of mobile terminals may be, for example, arranging N mobile terminals in a line on a substantially same plane based on an arbitrary mobile terminal. In this case, N cameras are sequentially arranged on a circumference spaced a certain distance from the subject, or the distance between the two mobile terminals placed on both sides and the subject is matched based on the mobile terminal placed in the center, and the remaining cameras are It is also possible to make the distance to the subject different. Also, the subject may be a fixed subject or a moving subject. Here, a plurality of mobile terminals are arranged around the subject, and the arrangement of other mobile terminals may be changed according to the shooting control of one mobile terminal, that is, the host mobile terminal, and the tilt and angle between the subject and the host mobile terminal. Such photographing information may be performed under the control of the host mobile terminal, or may be performed under the control of each mobile terminal and the group management server 110.

3 is a schematic diagram of the mobile terminal 100 shown in FIG. 1, and FIG. 4 is a system diagram including the mobile terminal 100 shown in FIG. 1.

Referring to FIG. 3, the mobile terminal 100 may be implemented as devices for various purposes. For example, the mobile terminal 100 may be implemented as a mobile phone, a smart phone, a laptop computer, a tablet device, or the like, as described above, but is not limited thereto.

Referring to FIG. 3, a display 110 may be installed on the front surface 101 of the mobile terminal 100. A speaker 120 for receiving a voice from the other party may be installed above the display 110. A microphone 130 for transmitting a user's voice of the mobile terminal 100 to a counterpart may be installed under the display 110.

According to an embodiment, components for performing various functions of the mobile terminal 100 may be arranged around the speaker 120 is installed. Components may include at least one sensor module 140. The sensor module 140 may include, for example, at least one of an illuminance sensor (eg, an optical sensor), a proximity sensor, an infrared sensor, and an ultrasonic sensor. The component may include a camera 150. According to an embodiment, the component may include an LED indicator 160 for recognizing status information of the mobile terminal 100 to a user.

The mobile terminal 100 may take a picture of a subject using the camera 150. The right side of FIG. 3 is a side view of the mobile terminal 100. Referring to FIG. 3, the mobile terminal 100 may include another camera 150 ′. However, the present invention is not limited thereto, and may include more cameras. The cameras 150 and 150 ′ of the mobile terminal 100 may form various angles of view.

4 is a diagram of a mobile terminal 201 in a network environment 200 according to various embodiments of the present disclosure. The mobile terminal 101 may include the mobile terminal 100 of FIG. 1.

Referring to FIG. 4, the mobile terminal 201 may include a bus 210, a processor 220, a memory 230, an input/output interface 250, a display 260, and a communication interface 270. In some embodiments, the mobile terminal 201 may omit at least one of the components or may additionally include other components.

The bus 210 may include, for example, a circuit that connects the components 210 to 270 to each other and transmits communication (eg, control messages and/or data) between the components.

The processor 220 may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 220 may control at least one other component of the mobile terminal 201 and/or perform an operation or data processing related to communication.

The memory 230 may include volatile and/or nonvolatile memory. The memory 230 may store commands or data related to at least one other component of the mobile terminal 201, for example. According to an embodiment, the memory 230 may store software and/or a program 240. The program 240 is, for example, a kernel 241, middleware 243, an application programming interface (API) 245, and/or an application program (or "application") 147, etc. It may include. At least a portion of the kernel 241, middleware 243, or API 245 may be referred to as an operating system (OS).

The kernel 241 is, for example, system resources used to execute an operation or function implemented in other programs (eg, middleware 243, API 245, or application program 247). : The bus 210, the processor 220, or the memory 230, etc.) can be controlled or managed. In addition, the kernel 241 can provide an interface for controlling or managing system resources by accessing individual components of the mobile terminal 201 from the middleware 243, the API 245, or the application program 247. I can.

The middleware 243 may serve as an intermediary so that, for example, the API 245 or the application program 247 communicates with the kernel 241 to exchange data.

In addition, the middleware 243 may process one or more job requests received from the application program 247 according to priority. For example, the middleware 243 can use the system resources of the mobile terminal 201 (eg, the bus 210, the processor 220, or the memory 230) for at least one of the application programs 247. Priority can be given. For example, the middleware 243 may perform scheduling or load balancing for the one or more work requests by processing the one or more work requests according to the priority assigned to the at least one.

The API 245 is, for example, an interface for controlling functions provided by the kernel 241 or the middleware 243 by the application 247. For example, file control, window control, image processing, or text It may include at least one interface or function (eg, command) for control or the like.

The input/output interface 250 may serve as an interface capable of transmitting commands or data input from a user or other external device to other component(s) of the mobile terminal 201, for example. In addition, the input/output interface 250 may output commands or data received from other component(s) of the mobile terminal 201 to a user or another external device.

The display 260 is, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, or Microelectromechanical systems (MEMS) displays, or electronic paper displays. The display 260 may display, for example, various contents (eg, text, images, videos, icons, symbols, etc.) to the user. The display 260 may include a touch screen, and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a user's body part.

The communication interface 270, for example, establishes communication between the mobile terminal 201 and an external device (eg, a first external mobile terminal 202, a second external mobile terminal 204, or a server 206). I can. For example, the communication interface 270 may be connected to the network 262 through wireless communication or wired communication to communicate with an external device (eg, the second external mobile terminal 204 or the server 206 ).

Wireless communication is, for example, a cellular communication protocol, for example, LTE (long-term evolution), LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), or GSM (global system for mobile communications), or the like may be used. Further, wireless communication may include short-range communication 264, for example. The short-range communication 264 may include at least one of, for example, wireless fidelity (WiFi), Bluetooth, near field communication (NFC), or a global navigation satellite system (GNSS). GNSS is, for example, GPS (global positioning system), Glonass (Global Navigation Satellite System), Beidou Navigation Satellite System (hereinafter referred to as “Beidou”) or Galileo, the European global satellite-based navigation system. It may include at least one. Hereinafter, in this document, “GPS” may be used interchangeably with “GNSS” (interchangeably used). Wired communication may include, for example, at least one of a universal serial bus (USB), a high definition multimedia interface (HDMI), a recommended standard 232 (RS-232), or a plain old telephone service (POTS). The network 262 is one of a telecommunications network, for example, a computer network (eg, a local area network (LAN) or a wide area network (WAN)), the Internet, or a telephone network. It may include at least one.

Each of the first and second external mobile terminals 202 and 204 may be the same or different types of devices as the mobile terminal 201. According to an embodiment, the server 206 may include a group of one or more servers. According to various embodiments, all or part of the operations executed on the mobile terminal 201 may be executed on another one or a plurality of mobile terminals (eg, the mobile terminals 202 and 204, or the server 206. In one embodiment) According to this, when the mobile terminal 201 needs to perform a certain function or service automatically or upon request, the mobile terminal 201 instead of or additionally executing the function or service itself, at least some functions associated therewith. May be requested from another device (eg, mobile terminal 202, 204, or server 206). Another mobile terminal (eg, mobile terminal 202, 204, or server 206) may perform the requested function or The additional function may be executed and the result may be transmitted to the mobile terminal 201. The mobile terminal 201 may provide the requested function or service by processing the received result as it is or additionally. For example, cloud computing, distributed computing, or client-server computing technology may be used.

According to an embodiment, the mobile terminal 101 and the first and second external mobile terminals 202 and 204 are one mobile terminal using the above-described communication methods for images photographed using a camera in each mobile terminal. It can also be transferred to and composited into a single image. For example, a multi-channel image may be generated from an image captured by the mobile terminal 101 and each captured image captured by the first and second external mobile terminals 202 and 204 at different locations. In addition, images captured by the mobile terminal 101 and the first and second external mobile terminals 202 and 204 are transmitted through the server 206, that is, the group management server 110 shown in FIG. In 200, a multi-channel image may be generated.

5 to 7 are exemplary diagrams for explaining a method for generating a multi-channel image using the mobile terminal 100.

Referring to FIG. 5, a mobile terminal 100 (hereinafter referred to as a host mobile terminal) groups mobile terminals or users 501 to 503 of mobile terminals to participate in generating multi-channel images using augmented reality. Here, the host mobile terminal 100 performs grouping by clicking on the surrounding mobile terminals 501 to 503 displayed through augmented reality on the display screen. Specifically, when clicking as a member using the mobile terminal's IP address, a member join request message is transmitted to the displayed user's terminals (501 to 503), and other mobile terminals (501 to 503) send a response message to group them. This is done. Here, an augmented reality image obtained by synthesizing an image augmented with augmented reality with user images of a plurality of mobile terminals captured by the host mobile terminal may be displayed on the display screen. Here, it has been described that there are three other mobile terminals for capturing a multi-channel image, but the present invention is not limited thereto, and a larger number of mobile terminals may participate.

In another embodiment, the mobile terminal 100 may perform grouping using a QR code. Other mobile terminals 501 to 503 may participate in generating multi-channel images through the QR code generated by the mobile terminal 100.

Referring to FIG. 6, the time of all mobile terminals is synchronized with an absolute time based on an NTP (Network Time Protocol, hereinafter referred to as NTP) protocol for image capture at the same time using a plurality of mobile terminals. NTP refers to a protocol for synchronizing the time of systems running in a networked environment. The host mobile terminal 100 becomes an NTP server and synchronizes the time of the host terminal with other mobile terminals 601 to 603 participating in the photographing. As shown in FIG. 6, when the synchronization icon 610 is selected, the host mobile terminal 100 transmits a photographing synchronization message to the other grouped mobile terminals 601 to 603, and the other mobile terminal 601 transmits an NTP protocol. Synchronization is performed in absolute time according to each.

In another embodiment, the host mobile terminal 100 may collect the relative time from the other mobile terminals 601 to 603 and synchronize the shooting time with the reference time of the host mobile terminal 100. The host mobile terminal 100 may collect a relative time, calculate a value corresponding to its own time, and calculate a time to be synchronized.

Referring to FIG. 7, the host mobile terminal 100 leading the photographing sets the highest point, the middle point, and the lowest point on a straight line around a subject to obtain information on the inclination and angle of the camera with respect to the subject. The acquired subject and camera information are transmitted to all mobile terminals 701 to 703 to induce users of other mobile terminals 701 to 703 to adjust according to the camera tilt and angle information acquired by the host mobile terminal 100. To this end, all mobile terminals may interwork with the group management server 110 shown in FIG. 1 to exchange metadata information between a subject and a camera to receive a guide for adjustment.

As shown in FIG. 7, after positioning the camera, when the shooting icon 710 is selected in the host mobile terminal 100, the absolute shooting time is transmitted to all mobile terminals 701 to 703 that participated in the shooting, and programs in each terminal Start simultaneous shooting through the timer of. Here, it has been described that shooting is performed on all mobile terminals through a timer application, but the present disclosure is not limited thereto, and various synchronization shooting methods may be performed.

8 is a schematic diagram of the group management server 110 shown in FIG. 1.

The group management server 110 may control a plurality of mobile terminals through a synchronization signal for synchronizing the plurality of mobile terminals. The group management server 110 temporarily stores images captured from a plurality of mobile terminals, reduces the size of the captured image through codec change, and enables fast transmission. A detailed configuration of the group management server 110 will be described with reference to FIG. 8.

Referring to FIG. 8, the group management server 110 may include a group management unit 111, a photographing management unit 112, and a transmission unit 113.

The group management unit 111 may manage each mobile terminal by grouping a plurality of mobile terminals participating in multi-channel image generation.

The photographing management unit 112 may provide a guide for adjustment by exchanging metadata information between a subject and a camera in conjunction with a plurality of mobile terminals participating in multi-channel image capturing. In addition, images generated from a plurality of mobile terminals are reduced in size through a codec change to facilitate fast transmission through the communication network 120. In addition, it is possible to determine a data transmission method suitable for the shooting site or for wired or wireless transmission.

The transmission unit 113 transmits the converted images to the image server 200 through the communication network 120.

The image server 200 groups and stores the multi-channel images transmitted from the group management server 110 based on at least one of a mixture of time, channel, time and channel, and stores the stored according to the request of the user terminal 150. The grouped video is transmitted through the communication network. The detailed configuration of the image server 200 will be described with reference to FIGS. 9 to 12.

9 is a schematic diagram of the image server 200 shown in FIG. 1.

Referring to FIG. 9, the image server 200 includes an image processing device 130 and an image storage unit 140. The image processing apparatus 130 includes an image processing unit 131, an image conversion unit 132, and a transmission unit 133.

The image processing unit 131 performs image correction on multi-channel images transmitted from the group management server 110, that is, images captured by a plurality of mobile terminals. For example, since images captured by a plurality of mobile terminals may be out of focus, image processing is performed so that the focus between the cameras is the same. The image processing unit 131 corrects the transmitted multi-channel images. Since the geometrical error of the array of N cameras appears as a visual shake in the process of reproducing a multi-channel image, at least one of the size or size, slope, or center position of each image may be corrected to remove this.

The image conversion unit 132 groups multi-channel images based on at least one criterion in which time, channel, time and channel are mixed. The image conversion unit 132 groups and groups several spaces into one. The grouping method can be performed according to various criteria. The transmission system according to the embodiment transmits all image data to transmit effective multi-channel images or switching images to the user terminal 150 without waste of data, and transmits grouped images to transmit only data necessary for the user. To be. The image conversion unit 132 may group and group channel images for each time ±y (y is a natural number) based on the event at time t. For example, it may be a case in which an event occurs in channel 1 and t3. Here, the event may be a predetermined case, for example, a home run scene or an out scene in a baseball game, an event at a user's request, or a user's desire.

According to various embodiments, grouping of multi-channel images may be performed in various ways. As a basic method of grouping, channel images are grouped by time. Images of channels 1 to 5 captured at t1 are grouped into one to generate a grouped image. Similarly, images of channels 1 to 5 captured at t2 are grouped into one to generate a grouped image. In this way, grouping may be performed by time, and the user terminal 150 may check images captured from all directions surrounding the subject at the same time. A method of grouping channel images by time has been described, but it is also possible to transmit each channel. The mobile terminals may be grouped in the order of the first to fifth channels, but may be grouped in the reverse order, that is, in the order of the fifth channel to the first channel. It is also possible to group by time and channel combinations. In descending order, the first channel image photographed at t1, the second channel image photographed at t2, the third channel image photographed at t3, the fourth channel image photographed at t4, and the fifth channel image photographed at t5. Grouped images are created by grouping them into one. Similarly, the first channel image photographed at t2, the second channel image photographed at t3, the third channel image photographed at t4, the fourth channel image photographed at t5, and the fifth channel image photographed at t6 are grouped into one. It is also possible to generate a grouped image. In ascending order, the fifth channel image photographed in t1, the fourth channel image photographed in t2, the third channel image photographed in t3, the second channel image photographed in t4, and the first channel image photographed in t5 are Grouped images may be created by grouping them into one. Grouping of images obtained by mixing time and channel has been described, but is not limited thereto, and grouping is possible in various ways.

Referring back to FIG. 9, the grouped images converted by the image conversion unit 132 are stored in the image storage unit 140. When there is a request from the user terminal 150, the image processing apparatus 130 extracts images stored in the image storage unit 140 and transmits the extracted images to the user terminal 150 through the transmission unit 133.

10 is another schematic diagram of the image server 200 shown in FIG. 1.

Referring to FIG. 10, the image processing apparatus 130 includes an image processing unit 131, an image conversion unit 132, and a transmission unit 133, and the transmission unit 133 is a message handler 133-1, A trimmer 133-2 and a channel manager 133-3 may be further included. The transmission unit 133 may be a streaming device and is described as being included in the image server 130, but it is of course possible to implement it as a separate device separated from the image server 130. A description of the same parts as in FIG. 9 will be omitted, and differences will be mainly described.

The transmission unit 133 transmits the processed image or the stored image in real time. For example, it may be a device for real-time streaming. The message handler 133-1 performs session management and protocol management with the user terminal. The streamer 133-2 transmits an image to the user terminal and is a set of images to be transmitted to the user terminal. The channel manager 133-3 receives a signal from the user terminal, schedules an image in units of GOP, and delivers it to the streamer 133-2.

As shown in FIG. 11, when there is no special request from the user terminal, for example, a channel change request or a channel movement request, the channel manager 133-3 schedules the streamer 133-2 in each GOP unit. Delivered video. As shown in FIG. 11, channels 5, that is, images captured by the camera 5, are scheduled in the order of GOP1 to GOPN, and then transmitted to the streamer 133-2. The streamer 133-2 collects GOP images according to the scheduling of the channel manager 133-3 and transmits them to the user terminal.

As shown in FIG. 12, when a channel movement request is received from the user terminal, the channel manager 133-3 performs channel 5, that is, GOP1 to 3 among images captured by camera 5, GOP4 of channel 4, and channel 3 GOP5 of the channel 2, GOP6 of the channel 2, and GOP7 of the channel 7 are scheduled in the order and transmitted to the streamer 133-2. The streamer 133-2 collects GOP images according to the scheduling of the channel manager 133-3 and transmits them to the user terminal.

The image server 200 according to the embodiment decodes and renders a multi-channel image according to the occurrence of an event and transmits it to the user terminal 150, or the image server 200 transmits a multi-channel image or a grouped image to the user terminal 150. ), and while playing the multi-channel or grouped video transmitted from the user terminal 150, the compressed video may be decoded and rendered according to an event occurrence according to a user request. In this case, since fast video channel conversion is required for real-time video playback, decoding and rendering are performed as follows.

For example, when there is a multi-channel image of cameras 1 to 10, an event occurs to the right in the image corresponding to camera 2. For example, in the user terminal 150, it may be a case in which the user wants to play the image captured by the camera 10 from the image captured by the camera 2. In this case, decoding of the compressed multi-channel image is performed from camera 1 to camera 2 to store the decoded image in a temporary memory or a buffer, and then the rendering starts from camera 2.

In addition, when there are multi-channel images of cameras 1 to 10, an event occurs to the left in the image corresponding to camera 7. For example, in the user terminal 150, it may be the case that the user wants to play the image captured by the camera 1 from the image captured by the camera 7. In this case, for the compressed multi-channel image, decoding is performed from camera 1 to camera 7 to store the decoded image in a temporary memory or buffer, and then, from camera 7 to camera 1, rendering is started.

Referring back to FIG. 1, according to a connection request from the user terminal 150 to the video server 200, the video server 200 accepts the request and allocates a session. When the user terminal 150 assigns a device ID according to the authentication request and responds to the authentication request, the user terminal 150 initiates an authentication or device ID protocol session.

When the user terminal 150 requests a connection for a streaming session, the video server 200 accepts the request and allocates a stream. When the user terminal 150 makes an authentication request, the video server 200 responds to the authentication request. A streaming session is initiated by the user terminal 150 using authentication or device ID.

The user terminal 150 transmits an Alive maintenance request to the video server 200 for a predetermined time, for example, once every second in order to maintain the protocol session in the Alive state. The video server 200 responds to this request and manages the session. Accordingly, a connection state or an event state is maintained between the image server 200 and the user terminal 150.

Although not shown in the drawing, when the user terminal 150 transmits a playback request, the video server 200 responds to the playback request and transmits the playback image to the user terminal 150. When the user terminal 150 transmits a stop request, the playback video is stopped while responding to the stop request.

13 is a flowchart illustrating a method of generating a multi-channel image using a mobile terminal according to another embodiment.

Referring to FIG. 13, in step 1300, icons corresponding to a plurality of mobile terminals displayed on the display screen of the host mobile terminal are selected using the addresses of each of the plurality of mobile terminals to be photographed by the host mobile terminal. Group a plurality of mobile terminals to be photographed.

In step 1302, by selecting a synchronization icon corresponding to the group synchronization displayed on the display screen of the host mobile terminal, a synchronization message for synchronizing in absolute time is transmitted to a plurality of grouped mobile terminals. Here, the NTP protocol can be used for time synchronization.

In step 1304, photographing guide information for photographing a subject is set on the display screen of the host mobile terminal, and information on the subject extracted from the set photographing guide information is transmitted to a plurality of grouped mobile terminals.

In step 1306, by selecting an icon corresponding to the photographing button on the display screen of the host mobile terminal, all mobile terminals transmit a photographing message including absolute time information for starting photographing of a subject to a plurality of grouped mobile terminals.

In step 1308, the host mobile terminal photographs a subject according to the absolute time information, and transmits the photographed image to an external server. Here, it has been described that images captured by all mobile terminals participating in multi-channel image capturing are transmitted to an external server, that is, the group management server and image server shown in FIG. 1, and the image server generates multi-channel images. It is not limited, and of course, the host mobile terminal may receive images captured by another mobile terminal, and a multi-channel image may be generated by the host mobile terminal.

An embodiment may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transmission mechanism, and includes any information delivery medium.

The above description is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily transformed into other specific forms without changing the technical spirit or essential features of the present invention. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims (10)

By selecting icons corresponding to the plurality of mobile terminals displayed on the display screen of the host mobile terminal using the addresses of each of the plurality of mobile terminals for photographing the subject in the host mobile terminal, grouping the plurality of mobile terminals for photographing the subject Step-The host mobile terminal performs grouping by clicking on a plurality of mobile terminals in the vicinity displayed through augmented reality on the display screen, but if selected as a member using the IP address of the mobile terminal, the member on the displayed plurality of mobile terminals Transmitting a join request message and receiving response messages from a plurality of mobile terminals that have transmitted the member join request message;
Transmitting a synchronization message for synchronizing time to the plurality of grouped mobile terminals by selecting a synchronization icon corresponding to the group synchronization displayed on the display screen of the host mobile terminal;
Setting photographing guide information for photographing the subject on a display screen of the host mobile terminal, and transmitting photographing information extracted from the set photographing guide information to the grouped plurality of mobile terminals;
By selecting an icon corresponding to a photographing button on the display screen of the host mobile terminal, transmitting a photographing message including time information for starting photographing of the subject from all mobile terminals to the grouped plurality of mobile terminals; And
The host mobile terminal photographs the subject according to the time information, and transmits the photographed image to an external server. The method of claim 1,
In the external server,
And generating a multi-channel image based on the photographed image received from the host mobile terminal and the photographed image received from the grouped plurality of mobile terminals. delete The method of claim 1,
The synchronization message is a multi-channel video generation method using a mobile terminal, characterized in that generated according to the NTP (Network Time Protocol). The method of claim 1,
The above shooting guide information,
The host mobile terminal displays at least one of a top point, an intermediate point, and a bottom point on a straight line around the subject,
The extracted photographing information,
A method for generating a multi-channel image using a mobile terminal, comprising at least one of tilt information and angle information of the host mobile terminal for the subject according to the photographing guide information. The method of claim 1,
Tilt information and angle information of the host mobile terminal, and tilt information and angle information of the grouped plurality of mobile terminals are transmitted from the external server, and the photographing guide information and the photographing information are transmitted from the external server to all mobile terminals. A method of generating a multi-channel image using a mobile terminal, characterized in that transmitting additional metadata to be included. The method of claim 1,
All mobile terminals automatically photograph the subject according to each timer program according to absolute time information. By selecting icons corresponding to the plurality of mobile terminals displayed on the display screen of the host mobile terminal using the addresses of each of the plurality of mobile terminals for photographing the subject in the host mobile terminal, grouping the plurality of mobile terminals for photographing the subject Step-The host mobile terminal performs grouping by clicking on a plurality of mobile terminals in the vicinity displayed through augmented reality on the display screen, but if selected as a member using the IP address of the mobile terminal, the member on the displayed plurality of mobile terminals Transmitting a join request message and receiving response messages from a plurality of mobile terminals that have transmitted the member join request message;
Transmitting a synchronization message for synchronizing time to the plurality of grouped mobile terminals by selecting a synchronization icon corresponding to the group synchronization displayed on the display screen of the host mobile terminal;
Setting photographing guide information for photographing the subject on a display screen of the host mobile terminal, and transmitting photographing information extracted from the set photographing guide information to the grouped plurality of mobile terminals;
By selecting an icon corresponding to a photographing button on the display screen of the host mobile terminal, transmitting a photographing message including time information for starting photographing of the subject from all mobile terminals to the grouped plurality of mobile terminals;
Generating a photographed image by photographing the subject according to the time information in the host mobile terminal;
Receiving a plurality of photographed images photographing the subject from the plurality of grouped mobile terminals; And
And generating a multi-channel image using the generated photographed image and the plurality of received photographed images. A recording medium on which a program for executing the method according to any one of claims 1, 2, 4 to 8 is recorded. Display screen;
Shooting module; And
Including a processor,
The processor,
By selecting icons corresponding to the plurality of mobile terminals displayed on the display screen using the addresses of each of the plurality of mobile terminals to photograph the subject, a plurality of mobile terminals to be photographed are grouped-and augmented reality is displayed on the display screen. Grouping is performed by clicking on a plurality of mobile terminals in the vicinity displayed through, but when a member is selected using the IP address of the mobile terminal, a member join request message is transmitted to the displayed plurality of mobile terminals, and the member join request message is transmitted. Receiving response messages from a plurality of mobile terminals -,
By selecting a synchronization icon corresponding to the group synchronization displayed on the display screen, a synchronization message for synchronizing time is transmitted to the plurality of grouped mobile terminals,
Shooting guide information for shooting the subject is set on the display screen, and shooting information extracted from the set shooting guide information is transmitted to the grouped plurality of mobile terminals,
By selecting an icon corresponding to a photographing button on the display screen, all mobile terminals transmit a photographing message including time information to start photographing a subject to the grouped plurality of mobile terminals,
A mobile terminal that controls the photographing module according to the time information to photograph the subject and transmits the photographed image to an external server.

Images