JP2022134548A - Distribution control device and control method - Google Patents

Abstract

To allow more suitable stream distribution.SOLUTION: A distribution control device comprises: storage means that stores a plurality of profiles corresponding to the distribution settings of a plurality of imaging units; creation means that integrates the plurality of profiles to create a virtual profile; and distribution means that, based on the virtual profile, distributes a plurality of videos acquired from the plurality of imaging units as a single stream. When the virtual profile is changed to an invalid configuration while the distribution is performed by the distribution means, the creation means updates the virtual profile to a valid configuration while maintaining coding settings included in the plurality of profiles, and the distribution means continues the distribution of the stream based on the updated virtual profile.SELECTED DRAWING: Figure 6

Description

The present invention relates to a technology for synthesizing and distributing a plurality of videos.

A surveillance camera having a plurality of imaging units provides a function of synthesizing images acquired from the plurality of imaging units into one video stream and distributing the combined video stream. Further, Japanese Patent Application Laid-Open No. 2002-200000 discloses a technique for changing settings so that image quality becomes uniform when a plurality of images are synthesized and displayed.

ONVIF (Open Network Video Interface Forum) is a common standard for connection between a camera and a video receiving client device. Non-Patent Document 1 published by ONVIF describes a function called a virtual profile that enables video from a plurality of imaging units to be distributed as one stream. A virtual profile is a collection of profiles (video settings and encoding settings) of individual imaging units as one virtual setting value group.

Japanese Patent Application Laid-Open No. 2001-320616

ONVIF, ONVIF Media2 Service Specification Version 20.12, December 2020

When settings related to each image pickup unit are changed or video settings are deleted, the video stream generated based on images from multiple image pickup units becomes invalid due to inconsistent setting values, etc. may be lost. At this time, in order to validate the video stream again, it is conceivable to reconfigure the virtual profile based on the changed configuration and settings of the imaging unit. However, if the post-reconfiguration settings differ from the pre-reconfiguration settings, the client device receiving the video stream may not be able to decode/display the video stream.

The present invention has been made in view of such problems, and an object of the present invention is to provide a technology that enables more suitable stream distribution.

In order to solve the above problems, the distribution control device according to the present invention has the following configuration. That is, the distribution control device
a storage means for storing a plurality of profiles corresponding to distribution settings of each of the plurality of imaging units;
generating means for generating a virtual profile by integrating the plurality of profiles;
distribution means for distributing a plurality of videos acquired from the plurality of imaging units as a single stream based on the virtual profile;
with
If the virtual profile becomes an invalid configuration while the distribution is being performed by the distribution means, the generating means converts the virtual profile to a valid configuration while maintaining the encoding settings included in the plurality of profiles. and the distributing means continues distributing the stream based on the updated virtual profile.

According to the present invention, it is possible to provide a technology that enables more suitable stream distribution.

It is a figure which shows the system configuration|structure in 1st Embodiment. 3 is a block diagram showing the internal configuration of the camera; FIG. 3 is a block diagram showing the internal configuration of the client device; FIG. FIG. 4 is an exemplary diagram illustrating generation of a virtual profile; FIG. 10 is a sequence diagram relating to virtual profile generation and video distribution processing; FIG. 7 is a sequence diagram relating to virtual profile update processing in the first embodiment; It is a figure which shows the system configuration|structure in 2nd Embodiment. FIG. 11 is a sequence diagram relating to virtual profile update processing in the second embodiment; FIG. 11 is a sequence diagram relating to session management processing in the second embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

(First embodiment)
As a first embodiment of the control apparatus according to the present invention, distribution control in a camera having a plurality of imaging units and distributing images to a client device, which is a terminal, will be described below as an example.

<Configuration of each device in the system>
FIG. 1 is a diagram showing the system configuration in the first embodiment. The system includes a camera 1000 and a client device 2000 communicably connected to each other via a network 3000 . The camera 1000 includes a plurality of imaging units (imaging mechanisms). The client device 2000 transmits to the camera 1000 a command requesting change of video settings and encoding information related to each imaging mechanism, and a command requesting video distribution. The camera 1000 processes the received command and transmits a response to the client device 2000 .

FIG. 2 is a block diagram showing the internal configuration of the camera 1000. As shown in FIG. A control unit 1001 controls the entire camera 1000 . The control unit 1001 can be implemented, for example, by the CPU executing various programs.

The storage unit 1002 stores programs executed by the control unit 1001, and is also used as a work area during program execution. The storage unit 1002 is also used as a storage area for various data. For example, it stores video settings and encoding information related to the imaging mechanisms 1004A to 1004D (hereinafter referred to as the imaging mechanism 1004) and setting values related to image data synthesized by the video synthesizing unit 1007. FIG. Furthermore, image data generated by the imaging unit 1003 and image data synthesized by the video synthesizing unit 1007 are stored.

The imaging unit 1003 converts an analog signal of the subject image obtained by the imaging mechanism 1004 into a captured image that is digital data. Then, the captured image obtained is output to the storage unit 1002 or the image synthesizing unit 1007 .

The imaging mechanism 1004 is composed of an imaging optical system including a lens, an imaging element, and the like, and pan, tilt, and zoom mechanisms for controlling the imaging direction and angle of view. The imaging mechanism control unit 1005 controls the pan, tilt, and zoom mechanisms of the imaging mechanism 1004 . After controlling the pan, tilt, and zoom mechanisms, values such as the position and range of the imaging region changed by the control are output to the storage unit 1002 .

The communication unit 1006 receives each setting value change and control command from the client device 2000 via the network 3000 . It also transmits various data such as responses to commands and image data stored in the storage unit 1002 to the client device 2000 .

The image synthesizing unit 1007 synthesizes the captured images captured by the imaging mechanisms 1004 A to 1004 D and obtained by the imaging unit 1003 based on the setting values stored in the storage unit 1002 . The image obtained by combining is distributed to the client device 2000 via the communication unit 1006 .

Note that the processing block configuration shown in FIG. 2 is an example, and is not limited to the configuration in FIG. For example, it may be configured to further include an audio input section and an audio output section.

FIG. 3 is a block diagram showing the internal configuration of the client device 2000. As shown in FIG. A control unit 2001 controls the entire client device 2000 . The control unit 2001 can be implemented, for example, by the CPU executing various programs.

The storage unit 2002 stores programs executed by the control unit 2001 and is also used as a work area during program execution. The storage unit 2002 is also used as a storage area for various data. For example, information of connectable cameras existing on the network 3000 is stored.

The display unit 2003 is composed of, for example, an LCD display, and provides the user of the client device 2000 with various information such as setting screens, data acquisition/display screens, and various messages.

The input unit 2004 is composed of, for example, buttons, a touch panel, a mouse, etc., receives operations from the user, and notifies the control unit 2001 of the received contents.

The communication unit 2005 transmits each change command including change of the imaging area to the camera 1000 via the network 3000 . Also, a response to each change command is received from the camera 1000 .

Note that the processing block configuration shown in FIG. 3 is an example, and is not limited to the configuration in FIG. For example, it may be configured to further include a received video display unit, an image analysis processing unit, and a video storage unit.

<Operation of each device in the system>
FIG. 4 is a diagram exemplifying generation of a virtual profile. More specifically, the storage unit 1002 of the camera 1000 stores profiles corresponding to distribution settings (video settings, encoding information settings, etc.) of the imaging mechanisms 1004A to 1004D. The client device 2000 designates two or more profiles and transmits a CreateMultiTrackConfiguration command to the camera 1000 . The CreateMultiTrackConfiguration command is a command that requests creation of a Virtual Profile defined by ONVIF. The control unit 1001 of the camera 1000 integrates the profiles specified in the CreateMultiTrackConfiguration command to generate one virtual profile.

Note that FIG. 4 shows an example in which one virtual profile is generated by designating all four profiles corresponding to the four imaging mechanisms 1004A to 1004D. However, another number of profiles (for example, two or three) may be specified to generate one virtual profile.

FIG. 5 is a sequence diagram of virtual profile generation and video distribution processing. Specifically, it shows the exchange of data between the camera 1000 and the client device 2000 at the start of video distribution.

In S<b>1001 , the client device 2000 transmits a CreateMultiTrackConfiguration command to the camera 1000 . As described above, the client device 2000 transmits a CreateMultiTrackConfiguration command specifying two or more profiles to be collectively distributed as a single stream among a plurality of profiles stored in the camera.

Camera 1000 creates a virtual profile including the profile specified in the CreateMultiTrackConfiguration command. Then, as a response to the command, the Token, which is an identifier indicating the virtual profile, is transmitted to the client device 2000 .

In S1002, the client device 2000 transmits to the camera 1000 a DESCRIBE method defined by RTSP (Real Time Streaming Protocol). The DESCRIBE method is a method for acquiring information on video/audio distributed by the camera. More specifically, the client device 2000 transmits a DESCRIBE method designating a URI (Uniform Resource Identifier) associated with the virtual profile.

In S1003, upon receiving the DESCRIBE method, the camera 1000 determines settings for distribution. Camera 1000 determines the virtual profile to be used for distribution from the URI included in the DESCRIBE method. Then, based on information of each profile included in the virtual profile, settings for distribution are determined. For example, video settings (resolution of video to be distributed, frame rate, etc.) and coding information (coding method, tier, level, etc.) are determined. After that, the determined setting is transmitted to the client device 2000 as a response to the DESCRIBE method.

In S<b>1004 , the client device 2000 transmits an RTSP SETUP method to the camera 1000 . The SETUP method is a method for requesting establishment of an RTP (Real-time Transport Protocol) session for distribution. More specifically, the client device 2000 transmits to the camera 1000, based on the information acquired in S1003, the URI to which the information of the stream for which the session is to be established is added.

Upon receiving the SETUP method, the camera 1000 establishes a session for multicast distribution and transmits session information to the client device 2000 .

In S<b>1005 , the client device 2000 transmits the RTSP PLAY method to the camera 1000 . The PLAY method is a method for requesting the camera to start video distribution.

Upon receiving the PLAY method, the camera 1000 synthesizes the images associated with each profile into a single image based on the settings determined in S1003, and starts distribution to the client device 2000. FIG.

FIG. 6 is a sequence diagram relating to virtual profile update processing in the first embodiment. Specifically, it shows the exchange of data between the camera 1000 and the client device 2000 when the virtual profile is disabled (unusable) by a profile deletion request from the client device 2000 .

In S<b>1011 , the client device 2000 transmits a DeleteProfile command to the camera 1000 . A DeleteProfile command is a command that requests the camera to delete a profile. That is, the client device 2000 can request deletion of the profile included in the virtual profile.

Upon receiving the DeleteProfile command, camera 1000 deletes the specified profile. However, when a profile included in a virtual profile is deleted (or its settings are changed), the configuration of the virtual profile may become invalid.

In S1012, the camera 1000 reconstructs the invalid virtual profile. That is, in accordance with the deletion (or setting change) of the profile in S1011, the video setting and encoding setting used for distribution are determined and updated to a valid configuration. However, here, the encoding settings (encoding method, level, tier, etc.) are maintained without being changed from the settings before reconstruction. This is to prevent the client device 2000 from being unable to reproduce (decode). After determining the settings again, a response to the command is transmitted to the client device 2000, and video distribution is resumed with the updated settings.

For example, H.264 as an encoding setting. 264 and H. Consider the case where H.265 is delivering using mixed virtual profiles. H.D. during delivery. When the H.265 profile is deleted, a virtual profile is generated (reconfigured) in which only the video settings are changed while the encoding settings are maintained (that is, the H.265 settings are left). An example of changing the video settings is changing the video corresponding to the deleted profile to a blank image (eg, a solid black image).

In S<b>1013 , the client device 2000 transmits an RTSP TEARDOWN method to the camera 1000 . The TEARDOWN method is a method for requesting stoppage of video/audio distribution. Specifically, the client device 2000 transmits a TEARDOWN method specifying the session to be terminated. When the camera 1000 receives the TEARDOWN method, it stops video/audio distribution.

In S<b>1014 , the client device 2000 transmits an RTSP DESCRIBE method to the camera 1000 .

In S1015, upon receiving the DESCRIBE method, the camera 1000 determines distribution settings. That is, when the camera 1000 receives the TEARDOWN method and the DESCRIBE method after resuming distribution, it newly determines settings for distribution. In S1012, the settings for distribution are determined while maintaining the encoding settings, but in S1015, the encoding settings are also changed as necessary.

As an example of the case where it is necessary to change the setting, there is a case where it is desired to change the encoding method according to the combination of the encoding methods of the profiles forming the virtual profile. There are also cases where it is desired to change level and tier information due to changes in resolution, frame rate, video bit rate, and the like.

For example, previously using H.264 as an encoding setting. 264 and H. H.265 used to be distributed using a mixed virtual profile, but now H.265 is used. Consider the case where the H.265 profile is deleted. At this time, the encoding setting is set to H.264. H.264 and regenerate (reconfigure) the virtual profile in which the video settings are changed as necessary.

Camera 1000 transmits the changed settings to client device 2000 as a response to the DESCRIBE method. After that, the camera 1000 distributes video in response to a request from the client device 2000 (for example, reception of the PLAY method).

As described above, according to the first embodiment, when profiles included in a virtual profile are deleted or settings are changed during video distribution, the virtual profile is reconfigured while maintaining the encoding settings of each profile. do. As a result, distribution can be restarted in a format that can be received by the client device.

Further, when a distribution start request is received again after receiving a distribution stop request from the client device, the distribution settings including the encoding settings are re-determined. As a result, distribution can be performed with optimum settings according to a request from the client device.

(Second embodiment)
In the second embodiment, a mode in which a camera having a plurality of imaging mechanisms distributes images to a plurality of client devices will be described.

<Configuration of each device in the system>
FIG. 7 is a diagram showing the system configuration in the second embodiment. The system includes a camera 1000 and client devices 2000A and 2000B (hereinafter referred to as client devices 2000) that are communicably connected to each other via a network 3000. FIG. Camera 1000 includes a plurality of imaging mechanisms. The client device 2000 transmits to the camera 1000 a command requesting change of video settings and encoding information related to each imaging mechanism, and a command requesting video distribution. The camera 1000 processes the received command and transmits a response to the corresponding client device 2000 .

Note that the internal configurations of the camera 1000 and the client device 2000 are the same as those of the first embodiment (FIGS. 1 and 2), so description thereof will be omitted.

<Operation of each device in the system>
The virtual profile generation and video distribution processing are also the same as those of the first embodiment (FIGS. 3 and 4), so description thereof will be omitted.

FIG. 8 is a sequence diagram relating to virtual profile update processing in the second embodiment. Specifically, it shows the exchange of data between the camera 1000 and the client devices 2000A and 2000B when the virtual profile is invalidated by a profile deletion request from the client device 2000B during video distribution to the client device 2000A. there is

In S2001, the camera 1000 starts video distribution to the client device 2000A. Specifically, the camera 1000 designates the virtual profile created in S1001, receives the video distribution request in S1002 to S1005, and starts video distribution to the client device 2000A.

In S<b>2002 , the client device 2000</b>B transmits a DeleteProfile command to the camera 1000 . A DeleteProfile command is a command that requests the camera to delete a profile. That is, as in the first embodiment, the client device 2000B can request deletion of the profile included in the virtual profile.

Upon receiving the DeleteProfile command, camera 1000 deletes the specified profile. However, when a profile included in a virtual profile is deleted (or its settings are changed), the configuration of the virtual profile may become invalid.

In S2003, the camera 1000 reconstructs the invalid virtual profile. That is, the video settings and encoding settings used for distribution are determined again in accordance with the profile deletion (or setting change) in S2002. However, here, the encoding settings (encoding method, level, tier, etc.) are maintained without being changed from the settings before reconstruction. After determining the settings again, a response to the command is transmitted to the client device 2000B, and video distribution to the client device 2000A is resumed with the updated settings.

In S<b>2004 , the client device 2000</b>B transmits an RTSP DESCRIBE method to the camera 1000 . The DESCRIBE method is a method for acquiring information on video/audio distributed by the camera. Here, the client device 2000B transmits the DESCRIBE method specifying the URI associated with the virtual profile.

In S2005, the camera 1000 determines distribution settings upon receiving the DESCRIBE method. That is, when the camera 1000 receives the DESCRIBE method from the client device 2000B, which is the new distribution destination, the camera 1000 newly determines the settings for distribution. In S2003, the settings for distribution are determined while maintaining the encoding settings, but in S2005, the encoding settings are also changed as necessary.

As an example of the case where it is necessary to change the setting, there is a case where it is desired to change the encoding method according to the combination of the encoding methods of the profiles forming the virtual profile. There are also cases where it is desired to change level and tier information due to changes in resolution, frame rate, video bit rate, and the like.

Camera 1000 transmits the changed settings to client apparatus 2000B as a response to the DESCRIBE method.

In S<b>2006 , the client device 2000</b>B transmits an RTSP SETUP method to the camera 1000 . The SETUP method is a method for requesting establishment of an RTP session for distribution. More specifically, based on the information acquired in S2004, the client apparatus 2000B transmits to the camera 1000 the URI to which the information of the stream for which the session is to be established is added.

Upon receiving the SETUP method, the camera 1000 establishes a session for multicast distribution, and transmits session information to the client device 2000B.

In S<b>2007 , the client device 2000</b>B transmits the RTSP PLAY method to the camera 1000 . The PLAY method is a method for requesting the camera to start video distribution.

Upon receiving the PLAY method, the camera 1000 synthesizes the images associated with each profile into a single image based on the settings determined in S2005, and starts distribution to the client device 2000B. That is, apart from the stream based on the virtual profile reconstructed in S2003 being distributed to the client apparatus 2000A, a stream based on the new virtual profile determined in S2005 is further generated and distribution is started.

FIG. 9 is a sequence diagram relating to session management processing in the second embodiment. Session management processing is performed during video distribution, and is processing for confirming whether or not a session is continuing. Here, the camera 1000 and the client device 2000A periodically repeat transmission and reception of session management packets to confirm that the session is being continued.

In S2011, the camera 1000 starts video distribution to the client device 2000A. Since it is the same as S2001, the explanation is omitted.

In S2012, the camera 1000 transmits a session management packet to the client device 2000A. An example of a session management packet that the camera 1000 transmits to the client device 2000A is an RTCP SenderReport.

In S<b>2013 , the client device 2000</b>A transmits a session management packet to the camera 1000 . An example of a session management packet that the client apparatus 2000A transmits to the camera 1000 is an RTCP ReceiverReport.

In S2014, the client device 2000B transmits a DeleteProfile command. Since it is the same as S2002, the explanation is omitted.

In S2015, the camera 1000 reconstructs the invalid virtual profile. Since it is the same as S2003, the explanation is omitted. As described above, after the settings are determined again (that is, reconfigured), a response to the command is sent to client device 2000B, and video delivery to client device 2000A is resumed with the updated settings. Thereafter, transmission of session management packets from the camera 1000 to the client device 2000A is stopped.

In S<b>2016 , the client device 2000</b>A transmits a session management packet to the camera 1000 . That is, the client device 2000A periodically transmits session management packets even after resuming video distribution. However, since the client device 2000A does not receive the session management packet from the camera 1000, it determines that the continuation of the session has been terminated. Therefore, the client device 2000A transmits the TEARDOWN method (S2017) and the DESCRIBE method (S2018).

In S2019, when the camera 1000 receives the DESCRIBE method designating the same virtual profile as the virtual profile used in the previous distribution from the client device 2000A, the camera 1000 determines distribution settings. That is, similarly to S1015 in the first embodiment, when the camera 1000 receives the TEARDOWN method and the DESCRIBE method after resuming distribution, the camera 1000 newly determines settings for distribution. In S2019, the encoding setting is also changed as necessary.

Camera 1000 transmits the changed settings to client device 2000A as a response to the DESCRIBE method. After that, the camera 1000 distributes video in response to a request from the client device 2000A (for example, reception of the PLAY method).

In the above description, the camera 1000 stops transmitting session management packets after S2015, but it is also possible to continue generating and distributing a plurality of streams for one virtual profile without stopping transmission. .

As described above, according to the second embodiment, when a virtual profile becomes invalid due to an operation by another client device during video distribution to a client device, the encoding settings of each profile are maintained. Reconfigure the virtual profile in . As a result, distribution can be restarted in a format that can be received by the client device.

After that, when a request to start distribution using the same virtual profile is received from another client device, settings for distribution including encoding settings for the other client device are newly determined. On the other hand, transmission of the session management packet is stopped for distribution based on the virtual profile with the encoding setting maintained. This makes it possible to perform distribution with more suitable settings for each client device.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

1000 cameras; 2000 client devices; 3000 networks

Claims (12)

a storage means for storing a plurality of profiles corresponding to distribution settings of each of the plurality of imaging units;
generating means for generating a virtual profile by integrating the plurality of profiles;
distribution means for distributing a plurality of videos acquired from the plurality of imaging units as a single stream based on the virtual profile;
with
If the virtual profile becomes an invalid configuration while the distribution is being performed by the distribution means, the generating means converts the virtual profile to a valid configuration while maintaining the encoding settings included in the plurality of profiles. , and the distribution means continues distribution of the stream based on the updated virtual profile. 2. The distribution control device according to claim 1, wherein the change to an invalid configuration of said virtual profile is caused by deletion or modification of one or more profiles included in said plurality of profiles included in said virtual profile. . After the generation means updates the virtual profile, receiving a request to stop distribution of the stream from the first terminal to which the stream is distributed by the distribution means and a request to start distribution from the first terminal. 3. The distribution control according to claim 2, wherein, if received, said generating means regenerates a virtual profile by integrating a plurality of profiles after said one or more profiles have been deleted or changed. Device. After the generation means has updated the virtual profile, if a request to start distribution of the stream is received from a second terminal that is not the distribution destination of the stream by the distribution means, the generation means receives the one or more 3. The distribution control device according to claim 2, wherein the virtual profile is regenerated by integrating a plurality of profiles after the profile has been deleted or changed. 5. The distribution control device according to any one of claims 1 to 4, wherein the encoding setting includes at least one of encoding method, tier, and level. 6. The distribution control device according to any one of claims 1 to 5, wherein the plurality of profiles further includes video settings including at least one of video resolution and frame rate. 7. The delivery control apparatus according to claim 1, wherein said delivery means transmits and receives a session management packet to and from a terminal to which said stream is delivered during delivery of said stream. . 8. The distribution control apparatus according to claim 7, wherein when the virtual profile is updated during distribution of the stream, the distribution means stops transmission of the session management packet to the terminal to which the stream is distributed. . 9. The delivery control apparatus according to claim 1, wherein said virtual profile is a virtual profile generated by a CreateMultiTrackConfiguration command defined by ONVIF (Open Network Video Interface Forum). 4. The distribution control apparatus according to claim 3, wherein said request to stop distribution and said request to start distribution correspond to a TEARDOWN method and a DESCRIBE method defined by RTSP (Real Time Streaming Protocol). A distribution control method in a distribution device,
a generation step of generating a virtual profile by integrating a plurality of profiles corresponding to distribution settings of each of a plurality of imaging units;
a distribution step of distributing a plurality of videos acquired from the plurality of imaging units as a single stream based on the virtual profile;
updating the virtual profile to a valid configuration while maintaining the encoding settings included in the plurality of profiles when the virtual profile becomes an invalid configuration while the delivery is being performed by the delivery step; a control step for continuing delivery of said stream based on an updated virtual profile;
A delivery control method, comprising: A program for causing a computer to function as the distribution control device according to any one of claims 1 to 10.

Images