KR20090131891A - System and method for providing video stream of multiple personal roi - Google Patents

Abstract

PURPOSE: A system and a method for providing video stream of multiple personal ROI are provided to monitor each store by setting up as an ROI(Region of Interest) by the users. CONSTITUTION: A camera unit(310) processes video frame, and a single merged frame is offered on a real time basis. An encoding unit(314) receives the merged frame. The encoding unit detects the images about the region of interests which client terminals requested. An interface server(312) creates encoding modules(314a-314c) corresponding to the client terminals.

Description

System and method for providing video stream of multiple personal ROI}

The present invention relates to a system and a method for providing a personalized real-time video for each individual, and particularly, in a surveillance and broadcasting environment having a bidirectional transmission path, to obtain a panoramic video of a specific region or space in real time, and to set each user in the obtained real-time video. The present invention relates to a real-time video providing system and method for extracting a region of interest (ROI) and providing the same to a corresponding user in real time.

In general, an omni-directional video camera system refers to a camera system capable of capturing 360-degree omnidirectional views from a fixed point of view. The omni-directional video camera system may be equipped with a special type of special lens or may be omni-directional using a plurality of cameras. Shoot.

In addition, the remote control camera system is composed of a camera for photographing the subject and a controller for controlling the camera remotely, and adjusts the controller to zoom-in or zoom-out the subject to focus the subject closer. The camera shoots after focusing so that the surroundings of the subject are comprehensive.

1 is a view showing a typical 1: 1 remote control camera system. As shown in FIG. 1, the remote control camera system includes a camera 110 disposed at a remote location, and a controller 120 that controls the camera to control photographing of a subject. An image 130 for a specific portion of the image is obtained. Since such a general remote control camera system can photograph or monitor a subject through only one controller, it can provide an image of only one region of interest. Thus, there is a disadvantage in that it cannot provide images for a plurality of ROIs.

In addition, the conventional remote control camera system is to directly zoom in or zoom out the camera, or to move or rotate the camera to obtain the image of the desired position, so that when a plurality of users have different areas of interest, each user to each other There is a problem that it is impossible to provide images of other regions of interest in real time.

Therefore, the present invention is to propose a method that can be provided to a plurality of users by photographing the omnidirectional using a plurality of cameras, a plurality of users for each region of interest of the image of the omnidirectional photographed This paper proposes a method to efficiently provide images.

An object of the present invention for solving the problems of the prior art is to obtain a real-time video in all directions for a specific area or space in a surveillance and broadcast environment having a bidirectional transmission path, and from the obtained real-time video The present invention provides a real-time video providing system and method for extracting regions of interest set by users and providing a video of a corresponding region of interest for each user in real time.

Another object of the present invention is to provide a real-time video providing system and method that can provide a plurality of users a video for each region of interest in real time, more efficiently to provide a higher quality image.

A feature of the present invention for achieving the above-mentioned aspect relates to a real-time video providing system for providing a plurality of real-time images of different regions of interest to a plurality of client terminals, respectively,

A camera unit having a camera set unit consisting of a plurality of cameras and a PC for image processing, for processing a video frame input from the camera set unit and providing a single merged frame in real time for each time stamp; A plurality of sessions are connected to each client terminal in a 1: 1 session and receive a merge frame from the camera unit, and detect an image of a region of interest requested by the session connected client terminal from the received merge frame and transmit the image to the client terminal. An encoding unit configured of an encoding module; And an interface server configured to generate an encoding module corresponding to the client terminal in response to a video providing request input from the client terminal, and provides images of different regions of interest to each client terminal in real time.

PC for image processing of the camera unit of the system having the above-mentioned features,

A camera input PC connected to the cameras of the camera set unit and processing an image frame output from each camera; A merging PC that receives image frames for each time stamp from a camera input PC, performs grouping according to each time stamp, generates a merge frame, and packetizes the merge frame to multicast to each encoding module of an encoding unit. To send merge frames according to each time stamp to all encoding modules,

In particular, the camera unit includes a plurality of camera input PCs, divides the cameras of the camera assembly unit and connects the camera input PCs, and the camera input PCs include a time stamp including image frames and current time information input from the cameras. It is preferable to combine to provide to the merging PC.

The merging PC of the system having the above-mentioned features includes a plurality of synchronization buffers provided to correspond to the respective cameras, and an image for sequentially storing image frames input from the camera input PC in the synchronization buffers corresponding to the cameras. Receiving unit; A merge frame generation unit for generating a merge frame by grouping image frames stored in the synchronization buffers for each time stamp; And a transmission unit for packetizing the merge frame and transmitting the packet to all the encoding modules.

The encoding module of the system having the above-described feature receives and stores location information about the region of interest and QoS parameters from the session-connected client terminal, and receives a merge frame from the camera unit. It is preferable to extract an image of the ROI from the merged frame according to the location information of the merged frame, and to compress the image of the extracted ROI based on the stored QoS parameter to transmit the image of the ROI to the session-connected client terminal.

According to another aspect of the present invention, in order to provide real-time images of different regions of interest to a plurality of client terminals, it is connected to each camera constituting the camera aggregation unit and processes an image frame output from each camera for each time stamp. An image processing method for outputting a merge frame. The image processing method includes:

(a) receiving image frames from each camera; (b) combining the received image frame with a time stamp including current time information; (c) storing image frames associated with the time stamp in a synchronization buffer set to correspond to each camera; (d) generating a merge frame by grouping image frames having the same time stamp among the image frames stored in the synchronization buffer; (e) receiving and storing location information on a region of interest from a session-connected client terminal; (f) extracting an image of the ROI from the merged frame according to the stored position information of the ROI; (g) transmitting the extracted image of the ROI to the session-connected client terminal.

According to the system and method according to the present invention, a plurality of users having different regions of interest can provide a video of the region of interest in real time.

In addition, when the system according to the present invention is applied to a densely packed shopping district, the camera obtains an omnidirectional image of the corresponding area using a plurality of cameras, and provides each user with a region of interest. Only the image of the communication area can be provided. Therefore, according to the present invention, each user sets a region corresponding to his or her store as a region of interest, so that a plurality of users can individually monitor their stores using a single system.

In addition, in the case of relaying sports by applying the system and method according to the present invention to an IP TV environment supporting two-way communication, an entire view of the stadium is provided to the user as a real-time video, and each user is interested in a specific portion of the stadium. In this case, a higher quality image can be provided in real time for the ROI set by the user.

In addition, in the system according to the present invention, each user may move, zoom in, or zoom out of his / her region of interest, and may receive a video of the changed region of interest in real time. Thus, the system according to the present invention can achieve the same effect as a large number of users remotely controlling their cameras even when using a single camera system.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In describing the present invention, when it is determined that the detailed description of the related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

FIG. 2 is a conceptual diagram illustrating that a real time video for different ROIs is provided to a plurality of users by a real time video providing system according to an exemplary embodiment of the present invention. As illustrated in FIG. 2, a plurality of images 203, 204, and 205 obtained by using a plurality of cameras 202 are merged to generate a single merged image, and the merged image is displayed on a full screen ( 210, and a plurality of users may set a region of interest of one full screen 210 through the controllers 211, 213, and 215, respectively, and zoom in, zoom out, or move it. . That is, the first user may zoom in or zoom out of the ROI 212 to be viewed on the full screen using the first controller 211, and the second user may view the full screen using the second controller 213. The region of interest 214 to be viewed can be zoomed in or zoomed out. In addition, the third user may zoom in, zoom out, or move the ROI 216 to be viewed on the entire screen using the third controller 215. As described above, the real-time video providing system according to the present invention can not only service a plurality of users at the same time, but each user can interactively change the region of interest while providing the service, and the video of the changed region of interest is displayed in real time. You will be provided.

3 is a block diagram showing the overall configuration of a real-time video providing system according to a preferred embodiment of the present invention. Hereinafter, the configuration and operation of the system according to the present invention will be described in detail with reference to FIG. 3.

Referring to FIG. 3, the real-time video providing system according to an exemplary embodiment of the present invention receives a video frame from a plurality of cameras, merges the image frames, and generates and provides a real-time merged frame from a user through a client terminal. Interface server 312 that receives and processes video service requests, and is assigned to each client terminal, consisting of encoding modules (314a, 314b, 314c) for transmitting a real-time video for the region of interest set by the user to the corresponding client terminal The encoder 314 includes a plurality of client terminals 316a, 316b, and 316c for receiving and displaying a real-time video of the ROI from the encoding module. Hereinafter, the components of the real-time video providing system having the above-described configuration will be described in detail.

Each user accesses the interface server 312 through his or her client terminal, sets a region of interest among all videos provided by the interface server, sets a desired Quality of Service (QoS) parameter, and then provides a video providing service. Ask. Here, the client terminal may be a computer capable of accessing a network such as the Internet, a mobile communication terminal, a personal digital assistant (PDA), and the like, and the client terminal may be connected to the interface server through the Internet or may be connected to a session module. The video of the region of interest can be streamed in real time.

The interface server 312 includes a user web page that users can access, and each user accesses the user web page of the interface server to set a region of interest of the entire screen and request a video providing service. When the interface server requests a video providing service from each client terminal by each user, the interface server generates and allocates a new encoding module for each client terminal, and then connects a session between each client terminal and the allocated encoding module. do.

Accordingly, by the session connection between each client terminal and the assigned encoding module, the client terminal and the encoding module can transmit and receive data in a 1: 1 manner, and as a result, the user can use the client terminal to transmit the ROI or QoS parameters. Changeable in real time, the changed information is transmitted to the encoding module, the encoding module provides a video in real time according to the changed information.

In addition, after the encoding module for each client terminal is generated, the interface server transmits the position information and the quality of service (QoS) parameter of the ROI set by the user to each encoding module.

Each encoding module receives and stores address information, region of interest, and QoS parameters for the client terminal from the interface server. Each encoding module receives a merge frame from a camera unit in real time, extracts an image of the ROI from the received merge frame, compresses the image according to a requested QoS parameter, and transmits the image to the session-connected client terminal. In this case, when the merged frame received from the camera unit is compressed, the encoding module may reduce the amount of computation by decoding only the portion corresponding to the ROI. In addition, each encoding module receives in real time information on the region of interest or information on the QoS parameters from the client terminal connected to the session, extracts an image of the region of interest from the merged frame according to the received information, and compresses the session. Send to the connected client terminal.

Meanwhile, the camera unit acquires an omni-directional or panoramic image of a region or region to be provided using a plurality of cameras and image processing computers, generates a merged frame constituting the image, and transmits the merged frame to all encoding modules of the encoding unit. do. As described above, the system according to the present invention transmits a merge frame to all encoding modules and allows each encoding module to extract and transmit an image for one region of interest, thereby preventing bottlenecks in one encoding module. The configuration and operation of the camera unit will be described later.

Hereinafter, the configuration and operation of the camera unit 310 used in the system according to the preferred embodiment of the present invention will be described in detail with reference to FIG. 4.

4 is a block diagram schematically illustrating a configuration of a camera unit of a real-time video providing system according to an exemplary embodiment of the present invention. Referring to FIG. 4, the camera unit 310 of the present invention includes a camera assembly unit 510 composed of a plurality of cameras and a plurality of cameras connected to each camera of the camera assembly unit for distributing an image output from each camera, respectively. Camera input PCs 512a to 512c, which receives image frames from each camera input PC and performs grouping to generate a merge frame for each time stamp and multicast it to each encoding module of the encoding unit. Equipped. Hereinafter, each component of the camera unit will be described in detail.

5 is an exemplary view illustrating a camera assembly unit 510 of a camera unit according to an embodiment of the present invention. As shown in FIG. 5, a plurality of cameras may be constructed in a single housing, and in this case, it is preferable to arrange the cameras in different directions so as to photograph all directions or photograph in a panorama form. In particular, by arranging the plurality of cameras as shown in (a) of FIG. 5 so that a predetermined interval is spaced in the spherical housing, an image of 360 ° can be obtained, and the plurality of cameras are cylindrical as shown in (b) of FIG. Or by placing in a polygonal housing it is possible to obtain a panoramic image. As a result, the system according to the present invention can provide the user with a panoramic image or a 360 ° image of the region.

Also, by increasing the number of cameras of the camera assembly unit or improving the performance of each camera, a wide range of high resolution images can be obtained. Therefore, the number of cameras of the camera assembly unit and the performance of each camera may be appropriately determined according to the resolution requirements of the installed system.

The camera input PCs 512a-512c are connected to respective cameras of the camera assembly unit, respectively, and receive images output from each camera. The camera input PCs process the received images, correct the spatial misalignment of each image, perform lossless compression, and provide the image to the merging PC.

6 is a flowchart for sequentially explaining the operation of each camera input PC. Hereinafter, an operation of the camera input PC described above will be described in detail with reference to FIG. 6. First, the camera input PC synchronizes the clock with the merging PC (S610). Next, an image frame photographed from a camera is received (S614), current time information is read out, and the current time information is set in a time stamp (S616). Next, distortion of the received image frame is corrected and lossless compression is performed (S618 and S620). When lossless compression is performed, the time stamp is combined with the image frame and transmitted to the merging PC (S622).

On the other hand, in another embodiment of the present invention, the correction for the spatial deviation of the respective images can be performed in the merging PC, not the camera input PC. For example, when the correction for the spatial misalignment should refer to the input image of the adjacent camera, it is preferable to be performed in the merging PC.

As shown in FIG. 5, the merging PC 514 includes an image receiving unit 521, a merging frame generating unit 522, and a transmission unit having a plurality of synchronization buffers 520 provided to correspond to each camera 1: 1. The unit 523 is provided.

The image receiver 521 includes a plurality of synchronization buffers 520, and each synchronization buffer corresponds to each camera of the camera assembly unit 1: 1.

The image receiver 521 receives image frames combined with a time stamp from camera input PCs connected to each camera, and sequentially stores the received image frames in a synchronization buffer corresponding to the corresponding camera according to the time stamp. 7 is an exemplary diagram illustrating a state in which image frames for three time stamps are sequentially stored in three synchronization buffers.

The merge frame generator 522 generates a single merge frame by grouping image frames stored in the synchronization buffer for the same time period. The merge frame generated by the merge frame generator 522 is provided to the transmitter 523. The transmitter packetizes the merge frames and multicasts them to all encoding modules.

In the system according to the present invention, since each camera input PC has a different clock, a clock synchronization process is required, and even if the clocks are synchronized, image frames may be captured at different times depending on the state of each camera PC. An effective grouping method is needed. The merging PC 514 provides this effective grouping method. Table 1 below shows examples of such grouping methods.

Hereinafter, operations of the image receiver, the merge frame generator, and the transmitter of the merging PC of the system according to the present invention will be described in detail with reference to FIGS. 8 and 9. FIG. 8 is a flowchart sequentially illustrating operations of an image receiver of a merging PC of a camera unit according to the present invention, and FIG. 9 is a flowchart sequentially illustrating operations of a merge frame generator and a transmitter.

Referring to FIG. 8, an image receiving unit of a merging PC first performs clock synchronization with all camera input PCs (S710), and then receives image frames combined with a time stamp from each camera input PC (S714). When an image frame is received from each camera input PC, the received image frame is sequentially stored in a corresponding synchronization buffer (S716). The above processes are repeated until the system is shut down.

Referring to FIG. 9, the merge frame generation unit and the transfer unit of the merging PC group the image frames stored in the synchronization buffer by the same time stamp to generate a merge frame (S820), and packetize the merge frame (S824). , Multicasting in all encoders (S826).

FIG. 10 is a transmission protocol illustrating a merge frame and a transport packet generated by the merge frame generator and the transmitter of the merging PC using image frames. Referring to FIG. 10, a merge frame 'b' is generated by grouping image frames 'a' transmitted from respective camera input PCs by time stamps, and then, the merge frame is transmitted to an encoding module. Convert to UDP packets 'c'. The UDP packet includes a header in which a sequence number SeqNum is described and a payload representing streaming data. Here, the sequence number is for counting the generated stream.

The merge frame generated by the merging PC according to the present invention is converted into UDP packets and multicasted to all encoding modules. As such, the reason for multicasting the entire image equally for all encoding modules is as follows.

First, if only the image of the ROI is transmitted to each encoding module, since the image frame received from the camera input PC is compressed, decoding is required to extract the ROI from the merging PC. This multi-casts the entire image equally, because a large number of users can cause a bottleneck on the merging PC.

Second, if the sum of areas of interest requested by all users is larger than the size of the entire image, the amount of transmission increases. Thus, multicasting the entire image equally to each encoding module saves transmission and avoids bottlenecks in the merging PC.

Hereinafter, the configuration and operation of each encoding module of the encoder unit 314 used in the real-time video providing service system according to an exemplary embodiment of the present invention will be described in detail with reference to FIG. 11.

11 is a flowchart sequentially illustrating the operation of the encoding module used in the system according to the present invention. Referring to FIG. 11, when an encoding parameter is initially received from an interface server, each encoding module initializes an encoding module (S910). Next, the grouped merged frames are received from the merging PC in the form of packets (S914). After the merge frame is received, it is checked whether there is a request for changing the region of interest (ROI) or QoS parameter from the client terminal (S916). If the region of interest has been changed or the QoS parameters have been changed, the configuration information for them is updated (S918). Next, after decoding packets including an area corresponding to the set ROI in the merged frame (S920), the ROI is extracted from the decoded packets (S922). The extracted region of interest is encoded according to a predetermined QoS parameter (S924) and then transmitted to the client terminal (S926). The above process is repeated until the system is terminated.

The system according to the present invention having the above-described configuration provides a region of interest image to a plurality of users from a high resolution image having a wide viewing angle. FIG. 12 is an exemplary diagram for explaining providing images of different regions of interest to a plurality of users in real time according to the present invention. As illustrated in FIG. 12, in the entire screen including the entire merged frame 10-1, when each user sets an ROI, the screens 10-2, 10-3, and 10-4 of FIG. As described above, an image of the ROI is provided to the client terminal of each user.

1 is a view showing a typical 1: 1 remote control camera system.

FIG. 2 is a conceptual diagram illustrating that a plurality of users are provided with a real time video for a different ROI by a real time video providing system according to an exemplary embodiment of the present invention.

3 is a block diagram showing the overall configuration of the real-time video providing system according to an embodiment of the present invention.

4 is a block diagram schematically illustrating a configuration of a camera unit of a real-time video providing system according to an exemplary embodiment of the present invention.

5 is an exemplary view illustrating a camera assembly unit 510 of a camera unit according to an embodiment of the present invention.

6 is a flowchart for sequentially explaining the operation of each camera input PC.

FIG. 7 is an exemplary diagram illustrating a state in which image frames for three time stamps are sequentially stored in three synchronization buffers in a merging PC of the system according to an exemplary embodiment of the present invention.

FIG. 8 is a flowchart sequentially illustrating operations of an image receiver of a merging PC of a camera unit according to the present invention, and FIG. 9 is a flowchart sequentially illustrating operations of a merge frame generator and a transmitter.

FIG. 10 is a transmission protocol illustrating a merge frame and a transport packet generated by the merge frame generation unit and the transmission unit of the merging PC.

11 is a flowchart sequentially illustrating the operation of the encoding module used in the system according to the present invention.

FIG. 12 is an exemplary diagram for explaining providing images of different regions of interest to a plurality of users in real time according to the present invention.

Claims (9)

In the real-time video providing system for providing a plurality of client terminals each of the real-time video for different regions of interest, A camera unit having a camera set unit consisting of a plurality of cameras and a PC for image processing, for processing a video frame input from the camera set unit and providing a single merged frame in real time for each time stamp; A plurality of sessions are connected to each client terminal in a 1: 1 session and receive a merge frame from the camera unit, and detect an image of a region of interest requested by the session connected client terminal from the received merge frame and transmit the image to the client terminal. An encoding unit configured of an encoding module; An interface server configured to generate an encoding module corresponding to the client terminal in the encoding unit according to a video providing request input from the client terminal; The system includes a real-time video service providing system comprising providing images of different regions of interest to each client terminal in real time. The image processing PC of claim 1, wherein the image processing PC of the camera unit comprises: A camera input PC connected to the cameras of the camera set unit and processing an image frame output from each camera; A merging PC that receives image frames for each time stamp from a camera input PC, performs grouping according to each time stamp to generate a merge frame, and packetizes the merge frame to multicast to each encoding module of an encoding unit; And providing a merged frame according to each time stamp to all encoding modules. The method of claim 2, wherein the camera unit is provided with a plurality of camera input PC, characterized in that to divide the cameras of the camera assembly unit to connect to the camera input PC, The camera input PCs provide a real-time video service providing system by combining the image frames input from the camera and a time stamp including current time information to the merging PC. The method of claim 2, wherein the merging PC An image receiver including a plurality of synchronization buffers corresponding to each camera, and sequentially storing image frames input from the camera input PC in a synchronization buffer corresponding to the camera; A merge frame generation unit for generating a merge frame by grouping image frames stored in the synchronization buffers for each time stamp; A transmitter for packetizing the merged frame and transmitting the packetized packet to all encoding modules; And providing a merged frame to all encoding modules. The apparatus of claim 1, wherein the encoding module receives and stores location information of the ROI from the session-connected client terminal in real time. Receiving a merge frame from the camera unit, according to the position information of the stored region of interest extracts the image of the region of interest from the merge frame, characterized in that for transmitting to the session-connected client terminal. The method of claim 5, wherein the encoding module receives and stores information about QoS parameters from the session-connected client terminal in real time. And compressing the image of the extracted region of interest according to the information on the stored QoS parameter and transmitting the compressed image to the session-connected client terminal. In order to provide real-time images of different regions of interest to a plurality of client terminals, an image processing process is performed to output a merge frame for each time stamp by processing an image frame output from each camera connected to each camera constituting the camera aggregation unit. In the method, (a) receiving image frames from each camera; (b) combining the received image frame with a time stamp including current time information; (c) storing image frames associated with the time stamp in a synchronization buffer set to correspond to each camera; (d) generating a merge frame by grouping image frames having the same time stamp among the image frames stored in the synchronization buffer; Image processing method comprising a. The method of claim 7, wherein the image processing method (e) receiving and storing location information on a region of interest from a session-connected client terminal; (f) extracting an image of the ROI from the merged frame according to the stored position information of the ROI; (g) transmitting an image of the extracted region of interest to the session-connected client terminal; Image processing method further comprising. The method of claim 8, further comprising: (h) receiving and storing information on QoS parameters from a session-connected client terminal, In the step (g), after compressing the image of the extracted region of interest according to the information on the stored QoS parameter, the image processing method comprises transmitting the compressed image to the session-connected client terminal.

Images