JP2011234233A - Video distribution device and video distribution method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide more clients with a video with less missing data by efficiently using a memory.SOLUTION: When a distribution processing to a specific partner device is detected to be slower than the generation of video frame, a video distribution device reduces the number of video frame in one transmission unit held by distribution control means. Thereafter as the device detects that the delay in the distribution processing has been resolved for a certain period of time, the video distribution device increases the number of video frame in one transmission unit held by the distribution control means. In this way, the video distribution device can prevent the occurrence of an inconvenience such as an interruption of continuous distribution of image data due to a temporal inability to store generated image data or a limitation on the number of clients to be distributed.

Description

  The present invention relates to a video distribution apparatus and a video distribution method, and more particularly to a technique suitable for using a memory efficiently when an embedded device with a small installed memory distributes video to a large number of people.

  In recent years, distribution systems using IP (Internet Protocol) networks such as the Internet have been increasing. For example, it is used for Internet sites that distribute the status of ski resorts and zoos, and is also used for store / building monitoring. Since the above-described distribution system is also used for monitoring purposes, it is necessary to suppress the delay of the distribution image, and high real-time performance is required. In addition, it is required to be distributed to many clients at the same time.

  Conventionally, when distributing to a large number of people, distribution processing is performed according to the distribution speed of each client while temporarily storing distribution data of all the clients connected at that time in the memory on the server.

For example, Patent Literature 1 discloses a method of always discarding a past video frame group that could not be transmitted and always transmitting the latest video frame group when transmission delay or interruption occurs due to a network failure or the like. Has been.
Patent Document 2 discloses a method of dynamically changing the fragment size of multimedia data in a network that transmits voice and multimedia data.

JP 2005-086362 A JP 2000-022749 A

In the prior art disclosed in Patent Document 1 described above, when video is distributed to a large number of clients, if there are a plurality of clients whose distribution processing is not in time due to a narrow network bandwidth or the like, it must be temporarily retained. The amount of data that must be increased. In such a case, the memory capacity that can be used by the distribution server is insufficient. For this reason, the generated image data cannot be temporarily stored and the image data cannot be sent continuously, or the number of clients to be distributed must be limited.
The present invention has been made in view of the above-described problems, and an object of the present invention is to provide video distribution with less data loss to more clients by efficiently using a memory.

  The video distribution apparatus of the present invention includes an imaging processing unit that generates and encodes a video frame, a temporary storage unit that temporarily stores the video frame encoded by the imaging processing unit, and a single unit that is stored in the temporary storage unit. Alternatively, a distribution control unit that distributes a plurality of video frames as one transmission unit, a communication processing unit that actually transmits the video frames collected as one transmission unit to a partner device, and a distribution process performed by the distribution control unit Delay monitoring means for monitoring the delay situation, the delay monitoring means, when detecting that the delivery processing to the specific partner device being performed by the delivery control means is delayed from the generation of the video frame, Decrease the number of video frames in one transmission unit held by the delivery control means, and then detect that the delivery processing delay has been resolved for a certain period Characterized by increasing the number of video frames for one transmission unit, wherein the holding distribution control means and.

  According to the present invention, since the number of image frames included in a client fragment whose distribution processing is not in time is reduced, data transmitted by the client can be concentrated on a new frame. As a result, the amount of data that must be temporarily stored in the memory can be reduced, and video distribution with less data loss can be provided to more clients.

It is a block diagram explaining the hardware constitutions of a delivery camera server. It is a block diagram which shows the outline of a software function structure of a delivery camera server. It is a flowchart explaining the operation | movement procedure of an imaging processing program. It is a flowchart explaining the operation | movement procedure of a delivery control program. It is a flowchart explaining the operation | movement procedure of a delay monitoring program. It is a figure which shows the example of the client information in which the delivery delay occurred. It is a figure explaining the memory usage when fragment size 5 is fixed. It is a figure explaining the memory usage at the time of applying 1st Embodiment. It is a flowchart explaining the 2nd example of a delivery control program. It is a flowchart explaining the operation | movement procedure of a memory monitoring program. It is a figure explaining the usage-amount of the temporary memory part at the time of applying 2nd Embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
(First embodiment)
Hereinafter, a first configuration example and operation of a distribution system suitable for implementing the present invention will be described with reference to the drawings. In addition, since the distribution of audio data can be handled in the same line except for the image data distribution process and the data contents, this embodiment will be described focusing on the distribution of the image data.

FIG. 1 is a block diagram illustrating a hardware configuration of a distribution camera server representing the first embodiment of the present invention.
As shown in FIG. 1, the distribution camera server of this embodiment includes a CPU 100, a primary storage device 110, a secondary storage device 120, an image capture I / F 130, an audio capture I / F 140, and a network I / F 160 via the internal bus 101. Are connected to each other.

  Here, the primary storage device 110 is a high-speed writable storage device represented by a RAM, and is loaded with an OS, various programs, and various data, and is also used as a work area for the OS and various programs. The secondary storage device 120 is a non-volatile storage device represented by FDD, HDD, flash memory, CD-ROM drive, etc. In addition to being used as a permanent storage area for the OS, various programs, and various data. It is also used as a storage area for various short-term data. Details of various programs and the like placed in the primary storage device 110 and the secondary storage device 120 of the camera server will be described later.

  A camera 170 is connected to the image capture I / F 130, and image data captured by the camera 170 is converted and compressed into a predetermined format and transferred to the primary storage device 110. A microphone 180 is connected to the audio capture I / F 140, and sound data collected by the microphone 180 is converted and compressed into a predetermined format and transferred to the primary storage device 110. The network I / F 160 is an I / F for connecting to a communication medium 190 such as Ethernet (registered trademark), and is responsible for communication with clients and the like via the communication medium 190. The network between the camera server and the client is composed of a plurality of routers, switches, cables, etc. that satisfy the communication standard. In the present embodiment, any communication standard, scale, and configuration may be used as long as communication between each server and client can be performed without any problem. Therefore, it is applicable from the Internet to a LAN (Local Area Network).

FIG. 2A is a block diagram illustrating an outline of a software function configuration of the distribution camera server.
As shown in FIG. 2A, the primary storage device 110 is loaded with an OS, an imaging processing program 210, a distribution control program 211, a communication processing program 212, a temporary storage unit 213, and a delay monitoring program 214. The OS is a basic program for controlling the entire camera server, and the main configuration requirements of the video distribution apparatus 200 of this embodiment are configured by the OS.

  The imaging processing program 210 acquires and encodes an image frame (video frame) generated by the camera 170 via the image capture I / F 130, and temporarily stores it in the temporary storage unit 213. Then, the unnecessary image frame is deleted from the temporary storage unit 213. The distribution control program 211 distributes one or a plurality of image frames stored in the temporary storage unit 213 through the communication processing program 212 as one transmission unit in response to requests from various clients. In this embodiment, for easy understanding, it is assumed that 5 frames are set as 1 GOP (Group Of Pictures) and 5 frames are distributed as one transmission unit (hereinafter referred to as 1 fragment) unless otherwise specified.

  The communication processing program 212 controls the network I / F 160 and sends delivery data to various clients that are communication counterpart devices via the communication medium 190. The delay monitoring program 214 determines whether or not delivery processing to various clients is delayed when the image frame is stored in the temporary storage unit 213, and monitors the delay status of the delivery processing. In this embodiment, the distribution process performs the delay determination timing at the timing when the image frame is stored in the temporary storage unit 213. However, the timing at which transmission of the image frame for the fragment is completed for each client (each partner device), or You may carry out at a regular timing. The cooperation between the programs uses functions provided by the OS as necessary.

FIG. 3 is a flowchart for explaining the operation procedure of the imaging processing program 210 of the distribution camera server. The imaging processing program 210 is activated when the apparatus is turned on.
When activated, the image capture I / F 130 is initialized in S300. Next, it waits for an image frame to be supplied from the camera 170 in S301.

  When an image frame is generated in the standby state in S301, the process proceeds to S302 to perform encoding. Thereafter, the process proceeds to S303, and the free capacity of the temporary storage unit 213 is checked. As a result of this check, if there is no space in the temporary storage unit 213, the process returns to S301. In this case, the image frame is not stored and the next image frame is awaited. If the result of the check in S303 is that there is an empty buffer, the process proceeds to S304, where the image frame is saved and the latest frame number is updated.

  Next, in S305, it is determined whether there is a specific client currently distributed. If the result of this determination is that there is no specific client, it enters a standby state. If there is a distribution client, the process advances to step S306 to register an image frame in the client information existing in the temporary storage unit 213. Next, in S307, it is determined whether the number of untransmitted frames is larger than the next fragment size. The next fragment size is a value indicating how many image frames are delivered to the client as one transmission unit at the next transmission.

  If the number of untransmitted frames is larger as a result of the determination in S307, the process proceeds to S308, where the old data of the untransmitted frame is deregistered until it becomes equal to or smaller than the next fragment size. Then, it progresses to S309. If the number of untransmitted frames is not greater as a result of the determination in S307, the process proceeds directly to S309.

In S309, it is determined whether or not the processing in S306 to S308 has been performed for all the distribution clients. As a result of this determination, if it has not been performed, the process returns to S306 and the above-described processing is executed. If so, the process proceeds to S310. In S310, an image generation notification message is transmitted to the delay monitoring program 214.
In step S311, the transmission processing to the client is completed, the deregistered image frame that is not registered in any client is released from the temporary storage unit 213, and the process returns to the image generation waiting process in step S301.

FIG. 4 is a flowchart illustrating the operation procedure of the distribution control program 211 of the distribution camera server. The distribution control program 211 is activated when a distribution request is received from a client.
When activated, client information is generated and stored in the temporary storage unit 213 in S400. In the present embodiment, at this time, the next fragment size indicating how many image frames are distributed to the client as one transmission unit is set to “5”.

  In step S401, a message reception process from the delay monitoring program 214 is performed. Thereafter, in S402, distribution is performed for each received message. If a delivery delay notification has been received from the delay monitoring program 214, the next fragment size is reduced (S403), and if a delivery recovery notification has been received, the next fragment size is increased (S404). In this embodiment, when a delivery delay notification is received, the next fragment size is decreased from a predetermined value (for example, “5”) to a predetermined value (for example, “1”), and the number of video frames in one transmission unit is decreased. When the distribution restoration notification is received, the next fragment size is increased from “1” to “5”, and the number of video frames in one transmission unit is increased. It is also possible to notify the delivery delay notification or the delivery recovery notification in stages according to the delay time, and to change the maximum number of one transmission unit in a plurality of stages. If there is no message in the process of S402, the process proceeds to S406 without performing the process of S403 or S404.

  In the present embodiment, the image processing program 210 appropriately writes the generated image frame information to the client information, so in S406, it is determined whether there are untransmitted frames for the next fragment size. If the untransmitted frame is less than the next fragment size, a predetermined time is waited (S405), and the process returns to the message reception process (S401).

  If the untransmitted frame is larger than the next fragment size, the continuity of the top frame is checked (S407). Specifically, it is checked whether the head of the untransmitted frame is the next frame of the previously transmitted data, or whether the head is an intra-frame encoded image frame (hereinafter referred to as I frame).

  If the result of the check in S407 is that there is no continuity (both are not satisfied), the first image frame cannot be displayed even if it is received on the client side, so a waiting period of time is held (S405), and message reception processing (S401) Return to. By waiting for a predetermined time, the imaging processing program 210 newly generates an image frame and rewrites untransmitted frame information.

  On the other hand, if the result of the check in S407 is continuity (whichever is satisfied), the client information stored in the temporary storage unit 213 is updated (S408). Next, an image frame for the next fragment size and its header information are created from the beginning of the untransmitted frame and transmitted to the client through the communication processing program 212 (S409).

  When the transmission process of S409 is completed, it is confirmed whether or not the next data needs to be transmitted in S410. If the next data is transmitted as a result of the confirmation, the process returns to the message reception process (S401). If communication disconnection is detected in the transmission process to the client, or if the delivery of the image frame for the client request is completed, the process proceeds to S411, where the client information stored in the temporary storage unit 213 is released, and the process Exit.

FIG. 5 is a flowchart for explaining the operation procedure of the delay monitoring program 214 of the distribution camera server.
The delay monitoring program 214 is activated when the apparatus is turned on.
When activated, the system waits until a message is received from another program in S500. When a message is received, the process proceeds to S501, where it is confirmed whether the message is a delay check message. In the present embodiment, the image generation notification from the imaging processing program 210 is a message for delay check. In addition to the image generation timing, it is also possible to carry out at the timing of the completion of delivery of the client, the elapse timing of a certain time, or the timing of a combination thereof.

  If the result of the confirmation in S501 is not an image generation notification, the process returns to S500 and again waits for message reception. When the image generation notification is received, the following processing is repeated for all the distributing clients. First, in S502, the client state is determined, and the process is distributed according to the state of the distribution client.

  That is, as a result of the determination in S502, when there is no distribution delay in the client (no distribution delay), the process proceeds to S503 to determine whether the last frame of the transmission frame and the first frame of the untransmitted frame are continuous. If the result of this determination is continuous, nothing is done. In the case of discontinuity, that is, when unregistered frames are canceled in S308 of the imaging processing program 210, a delivery delay notification is sent to the delivery control program 211 (S504), and the client status is changed to delivery delay. (S505).

  If a delivery delay has occurred in the client (during delivery delay), it is first checked whether data can be transmitted without being discarded for a certain period of time (S506). Specifically, the determination is made based on whether or not the number of continuously transmitted frames is equal to or greater than a certain number of frames indicating restoration of delivery delay. If the result of this determination does not reach a certain time, nothing is done. If the predetermined time has been reached, a delivery restoration notice is sent to the delivery control program 211 (S507), and the client state is changed without delivery delay (S508). When the processing of S505 or S508 is completed, the process proceeds to S509, and it is determined whether or not the distribution status has been determined for all clients. If the result of this determination is that it has not been performed, processing returns to S502 and the above-described processing is performed. Moreover, it returns to S500 which performed.

FIG. 6 is a diagram illustrating an example of client information in which a delivery delay has occurred.
The manner in which information held in the client information changes in the course of frame generation and distribution processing will be described with reference to FIG. However, FIG. 6 is merely an example, and the variables in the figure are not necessarily used, and the processing is not executed in the order shown in the figure.

The transmission frame number is the leading number of the image frame being transmitted that is changed by the distribution control program 211, and -1 is set in the untransmitted state.
The transmission fragment size is the number of image frames being transmitted, which is changed by the distribution control program 211, and 0 is set in the untransmitted state.
The untransmitted frame number is the head number of the transmission-waiting image frame that is changed by the imaging processing program 210, and -1 is set as the untransmitted state.
The number of untransmitted frames is the number of transmission-waiting image frames that the imaging processing program 210 changes.
The next fragment size is the number of image frames of a fragment to be transmitted next, and increases or decreases when the distribution control program 211 receives a distribution delay notification or a distribution recovery notification. In this embodiment, the next fragment size is changed to “1” when a delivery delay occurs, and the next fragment size is changed to “5” when delivery recovery occurs.
The number of continuous transmission frames is the number of image frames transmitted continuously, and is set to 0 if no image frame is transmitted.

  When frame number 0 (initially an I frame) is generated, 0 is set to the untransmitted frame, and the number of untransmitted frames is incremented and set to 1. When frame numbers 1, 2, and 3 are generated, the number of untransmitted frames sequentially increases to 2, 3, and 4.

  Then, when frame number 4 is generated, the frame size is exactly 5 frames, which is the next fragment size, so the distribution control program 211 starts transmission processing. Then, the transmission frame number is changed to 0, the transmission fragment size is changed to 5, the number of continuous transmission frames is added to 5, and the untransmitted frame number and the number of untransmitted frames are changed to the initial state.

  When frame number 5 is generated, 5 is registered in the untransmitted frame number, and the number of untransmitted frames is incremented and changed to 1. Then, if frame numbers 6, 7, 8, and 9 are generated before transmission of frame numbers 0 to 5 is completed, the number of untransmitted frames increases in order to 2, 3, 4, and 5. When the frame number 9 is generated, 10 frames from 0 to 9 are used by the client.

  When the frame number 10 is generated, the untransmitted frame number is changed to 10 by the imaging processing program 210, and the registration of the frame numbers 5 to 9 is canceled. Then, the delay monitoring program 214 detects the delivery delay from the last frame number 4 of the transmission frame and the first frame number 10 of the untransmitted frame, and the delivery control program 211 changes the next fragment size to “1”.

  When the frame number 11 is generated, since the next fragment size is “1”, the untransmitted frame number is updated to 1, and the number of untransmitted frames is set to 1. Similarly, when the frame number 12 is generated, the untransmitted frame number is updated. In the frame numbers 10, 11, and 12, the frames used by the client are a total of 6 frames including the frame numbers 0 to 4 and the frame 10 (or 11 or 12).

Next, assuming that the transmission process is completed, transmission of the untransmitted frame number 12 is attempted. However, since it is determined in S407 that there is no continuity (because the head is not an I frame), the transmission process is not executed.
When the frame number 15 that is the next I frame is generated, the transmission frame number is changed to 15, the transmission fragment size is set to “1”, the number of continuous transmission frames is changed to 1, and the untransmitted frame number and the number of untransmitted frames are initialized. Change to

  When the frame number 16 is generated, 16 is stored in the untransmitted frame number, and the number of untransmitted frames is incremented and changed to 1. After frame number 16, the number of frames used by the client is a total of two frames, frame number 15 and one frame that has not been transmitted.

  As described above, during the period when the fragment size was “fixed to 5”, the number of frames used by one client was 5 frames at the minimum and 10 frames at the maximum, but after the fragment size was changed to 1, The minimum is 1 frame and the maximum is 2 frames.

  FIG. 6 is an explanation of client information when the distribution process is delayed. A delivery delay is not detected for a client having sufficient delivery capability such that delivery processing is completed while frame numbers 5 to 9 are generated. For this reason, since the fragment size remains “5”, it can be delivered as before.

  In this embodiment, in order to improve the real-time property, the image frame to be temporarily held is described as only the next one fragment, but a plurality of fragments may be temporarily held. Also, when the next image frame is generated beyond the next fragment, the delivery delay is detected. However, in actuality, the data is calculated with the intra-frame encoded I frame and the inter-frame encoded P frame / B frame. The size is different. For this reason, it is effective to calculate the data size in units of 1 GOP, calculate the time corresponding to the data amount of the next fragment to be distributed from there, and detect the delay when the time is exceeded.

  FIG. 7 is a diagram for explaining the state of the temporary storage unit 213 when the fragment size is “5 fixed” and distributed to a plurality of clients. FIG. 7 shows an example of a client 701 whose distribution capability is about half that of image generation and a client 702 whose distribution capability is sufficiently higher than that of image generation.

In FIG. 7, while the client 701 distributes the first fragment (frame numbers 0 to 4), the frame number 12 is generated, and the frame numbers 5 to 9 are deregistered.
The client 702 indicates that frame numbers 0 to 4 have been transmitted and deregistered, frame numbers 5 to 9 are being transmitted, and frame numbers 10 to 12 are waiting to be transmitted.

  The image memory area of the temporary storage unit 213 of the distribution camera server is shared by all clients, and assuming that a maximum of 15 frames can be stored, the amount of image memory used is 86 percent (13/15 frames). .

  FIG. 8 is a diagram for explaining a state of the temporary storage unit 213 when the present embodiment is applied and distributed to a plurality of clients. This figure shows an example of a client 801 whose distribution capability is about half that of image generation and a client 802 whose distribution capability is sufficiently higher than that of image generation.

The client 801 detects the delay and the fragment size is 1, and is in a state where 2 out of 5 frames are transmitted every time. At this time, two frames are used.
Since the client 802 has sufficient distribution capability as in FIG. 7, it indicates that all frames are sent in units of 5 frames. At this time, 8 frames are used.

  The image memory area in the temporary storage unit 213 of the distribution camera server is shared by all clients, and assuming that a maximum of 15 frames can be stored, the amount of image memory used is 53 percent (8/15 frames). .

  In this way, by applying the present embodiment and reducing the fragment size of the delayed client, it is possible to concentrate the reference of the image memory on recent data and improve the memory usage efficiency. Further, by improving the use efficiency of the memory, it is possible to reduce the possibility that the image frame cannot be stored in the temporary storage unit 213 at the time of generation, and it is possible to perform video distribution with less image frame dropping. At the same time, even if the data is distributed to a plurality of clients, the memory usage does not increase greatly, so that it can be distributed to more clients.

(Second Embodiment)
The second configuration and operation of the distribution system suitable for implementing the present invention will be described below with reference to the drawings. In addition, since the distribution of audio data can be handled in the same line except for the image data distribution process and the data contents, this embodiment will be described focusing on the distribution of the image data. The hardware configuration of the distribution camera server in this embodiment is the same as that in FIG.

FIG. 2B is a block diagram illustrating a software functional configuration of the distribution camera server.
The primary storage device 110 is loaded with an OS, an imaging processing program 910, a distribution control program 911, a communication processing program 912, a temporary storage unit 913, and a memory monitoring program 914.

The OS is a basic program for controlling the entire camera server, and the main configuration requirements of the video distribution apparatus 900 of this embodiment are configured by the OS.
The imaging processing program 910 acquires and encodes an image frame generated by the camera 170 via the image capture I / F 130, and saves it in the temporary storage unit 913. Then, the unnecessary image frame is deleted from the temporary storage unit 913.
The distribution control program 911 distributes one or more image frames stored in the temporary storage unit 913 through the communication processing program 912 as one transmission unit in response to requests from various clients.

  In this embodiment, for easy understanding, it is assumed that 5 frames are set as 1 GOP (Group Of Pictures) and 5 frames are distributed as one transmission unit (hereinafter referred to as 1 fragment) unless otherwise specified.

The communication processing program 912 controls the network I / F 160 and sends delivery data to various clients via the communication medium 190.
The memory monitoring program 914 calculates the amount of memory used when an image frame is stored in the temporary storage unit 913, and determines whether the free space is sufficient.
Further, in this embodiment, the distribution process performs the delay determination timing at the timing when the image frame is stored in the temporary storage unit 913, but at the timing when transmission of the image frame for the fragment is completed for each client or at a regular timing. You may implement.
The cooperation between the programs uses functions provided by the OS as necessary.
Each program performs the same processing as the corresponding program in FIG. 2 unless otherwise specified.

The description of the flowchart of the imaging processing program 910 in this embodiment is the same as FIG.
FIG. 9 is a flowchart for explaining the operation procedure of the distribution control program 911 of the distribution camera server. The distribution control program 911 is activated when a distribution request is received from a client. In addition, in the process of each step in the flowchart of FIG. 9, steps that perform the same processes as those described in the flowchart of FIG. 4 are denoted by the same step numbers as those in FIG.

  As a result of the sorting process for each received message in S402, if a memory compression notification is received from the memory monitoring program 914, it is determined whether or not the next fragment size exceeds the maximum fragment size at the time of compression (S903). If the result of this determination is that it has exceeded, the next fragment size is changed to the maximum fragment size at the time of compression (S904). If the memory recovery notification has been received from the memory monitoring program 914, it is determined whether or not the next fragment size is the same as the maximum fragment size at the time of tightness and can be transmitted continuously for a certain period of time (S905). As a result of this determination, if the condition is satisfied, the delay is eliminated and the next fragment size is increased (S906). If there is no message in the distribution process of S402, the process proceeds to S406.

In this embodiment, when the memory tightness notification is received, the next fragment size is reduced from a predetermined value (for example, “5”) to a predetermined value (for example, “2”) (S904), and when the memory recovery notification is received, the next fragment size is decreased. The fragment size is increased from “2” to “5” (S906). However, it is also possible to notify the memory tightness notification or the memory restoration notification in stages according to the memory usage, and change the next fragment size in stages.
Since the processing after the processing of S904 or S906 is the same as the processing of S406 to S411 of FIG. 4 described above, detailed description thereof is omitted.

  FIG. 10 is a flowchart for explaining the operation procedure of the memory monitoring program 914 of the distribution camera server.

The memory monitoring program 914 starts when the apparatus is turned on.
When activated, the system waits until a message is received from another program (S1000). When a message is received, it is confirmed whether the message is a memory check message (S1001). In this embodiment, the image generation notification from the imaging processing program 910 is a message for delay check. In addition to the image generation timing, it is also possible to carry out at the timing of the completion of delivery of the client, the elapse timing of a certain time, or the timing of a combination thereof. If it is not an image generation notification, the process returns to message reception waiting (S1000) again. If an image generation notification is received, the process proceeds to S1002.

  In S1002, the state of the image memory is determined. If the result of this determination is that the image memory is normal (initial state), processing proceeds to S1003. In step S1003, the current usage amount of the image memory is measured, and it is determined whether the usage amount is equal to or greater than the upper limit threshold value.

  If the result of determination in S1003 is less than the upper threshold, nothing is done. If the upper limit threshold is exceeded, a memory tight notification is sent to the distribution control program 911 in S1004, and then the state of the image memory is changed to tight in S1005. In this embodiment, the upper limit threshold of the memory usage is set to 80% of the maximum number of image frames that can be held in the temporary storage unit 913, but it can be simply set to 80% of the image memory capacity in the temporary storage unit 913. is there.

  On the other hand, if the result of determination in S1002 is that the state of the image memory is tight, the process proceeds to S1006 to check whether the usage amount of the image memory has fallen below the restoration threshold (lower limit threshold). If the result of this check does not fall below the recovery threshold, do nothing. If it is below the recovery threshold, the process advances to S1007 to send a memory recovery notification to the distribution control program 911. Thereafter, the process proceeds to S1008, and the state of the image memory is changed to normal.

When the memory monitoring program 914 detects that the memory usage of the temporary storage unit 913 exceeds the upper limit threshold, one transmission unit (for example, “5”) to the partner apparatus in which one transmission unit of the video frame exceeds a predetermined upper limit value. ) Is reduced to a predetermined value (less than the upper limit value, for example, “2”).
Further, when it is detected that the memory usage of the temporary storage unit 913 has fallen below the restoration threshold, the transmission unit of the video frame is increased. One transmission unit to the partner apparatus set so that one transmission unit is decreased to a predetermined value (for example, “2”) is increased to a value before the decrease (for example, “5”). That is, in this embodiment, the lower limit threshold has a plurality of thresholds, and the maximum value of the number of video frames included in one transmission unit is increased or decreased.

  In this embodiment, the recovery threshold of the memory usage is set to 30% of the maximum number of image frames that can be held in the temporary storage unit 913. However, it can be simply set to 30% of the image memory capacity in the temporary storage unit 913. It is. It is also possible to set multiple lower thresholds in stages and retain multiple thresholds so that the next fragment size can be changed in stages or the next fragment size can be changed abruptly many times. it can. Therefore, when the next fragment size is reduced, it is also effective to set the fragment size to a value for a certain period.

FIG. 11 shows a state of the temporary storage unit 913 when the present embodiment is applied and distributed to a plurality of clients.
FIG. 11 shows an example of a client 1101 whose distribution capability is about half that of image generation and a client 1102 whose distribution capability is sufficiently higher than that of image generation.
When distribution is started to the client 1101 and the client 1102, since the fragment size is “5”, the memory usage exceeds 80% as shown in FIG. For this reason, the fragment size at the time of compression is applied, and the fragments of the client 1101 and the client 1102 are both 2 frames.

  The image memory area of the temporary storage unit 913 of the distribution camera server is shared by all clients, and assuming that a maximum of 15 frames can be stored, the amount of image memory used is 33 percent (5/15 frames). . As described above, by applying the present embodiment and reducing the fragment size of the delayed client, it is possible to concentrate the reference of the image memory on recent data, and to improve the use efficiency of the memory.

Further, by improving the use efficiency of the memory, it is possible to reduce the possibility that the image frame cannot be stored in the temporary storage unit 913 at the time of generation, and it is possible to perform video distribution with less image frame dropping. At the same time, even if the data is distributed to a plurality of clients, the memory usage does not increase greatly, so that it can be distributed to more clients. In addition, by applying together with the first embodiment, the memory can be used more efficiently.
As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (computer program) that implements the functions of the above-described embodiments is supplied to a system or apparatus via a network or various computer-readable storage media. Then, the computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program.

200 Video Distribution Device, 210, Imaging Processing Program, 211 Distribution Control Program, 212 Communication Processing Program, 213 Time Storage Unit, 214 Delay Monitoring Program, 914 Memory Monitoring Program

Claims (12)

Imaging processing means for generating and encoding a video frame;
A temporary storage unit that temporarily stores the video frame encoded by the imaging processing unit;
Distribution control means for distributing one or more video frames stored in the temporary storage unit as one transmission unit;
Communication processing means for actually sending video frames collected as one transmission unit to a partner device;
Delay monitoring means for monitoring the delay status of the distribution processing performed by the distribution control means;
Have
When the delay monitoring unit detects that the distribution process to the specific partner device being performed by the distribution control unit is delayed from the generation of the video frame, the delay monitoring unit stores one transmission unit held by the distribution control unit. A video distribution characterized in that the number of video frames is decreased, and then the number of video frames in one transmission unit held by the distribution control means is increased when it is detected that the delay in the distribution process has been eliminated for a certain period of time. apparatus.   The video distribution apparatus according to claim 1, wherein the delay time detected by the delay monitoring unit has a plurality of threshold values, and the maximum number of one transmission unit is changed in a plurality of stages. Imaging processing means for generating and encoding a video frame;
A temporary storage unit that temporarily stores the video frame encoded by the imaging processing unit;
Distribution control means for distributing one or more video frames stored in the temporary storage unit as one transmission unit;
Communication processing means for actually sending video frames collected as one transmission unit to a partner device;
Memory monitoring means for monitoring the memory usage of the temporary storage unit with an upper limit threshold and a recovery threshold;
Have
When the memory monitoring unit detects that the memory usage of the temporary storage unit exceeds the upper limit threshold, the distribution control unit decreases the transmission unit of the video frame to be distributed,
The video distribution apparatus, wherein when the memory usage of the temporary storage unit is detected to be less than the restoration threshold, the transmission control unit increases a transmission unit of a video frame to be distributed.   4. The video distribution apparatus according to claim 3, wherein the lower limit threshold detected by the memory monitoring unit has a plurality of thresholds, and increases or decreases the maximum value of the number of video frames included in one transmission unit. Imaging processing means for generating and encoding a video frame;
A temporary storage unit that temporarily stores the video frame encoded by the imaging processing unit;
Distribution control means for distributing one or more video frames stored in the temporary storage unit as one transmission unit;
Communication processing means for actually sending video frames collected as one transmission unit to a partner device;
Delay monitoring means for monitoring the delay status of the distribution processing performed by the distribution control means;
Memory monitoring means for monitoring the memory usage of the temporary storage unit with an upper limit threshold and a recovery threshold;
Have
A video distribution apparatus, wherein the delay monitoring means increases or decreases the number of video frames included in one transmission unit for each partner apparatus, and the memory monitoring means increases or decreases the number of video frames included in one transmission unit. An imaging process that generates and encodes video frames; and
A temporary storage step of temporarily storing the video frame encoded in the imaging processing step in a temporary storage unit;
A delivery control step of delivering one or more video frames stored in the temporary storage unit as one transmission unit;
A communication processing step of actually sending video frames collected as one transmission unit to a partner device;
A delay monitoring step of monitoring a delay state of the distribution processing performed in the distribution control step;
Have
In the delay monitoring step, when it is detected that the distribution process to the specific partner apparatus performed in the distribution control step is delayed from the generation of the video frame, one transmission unit held in the distribution control step And then increasing the number of video frames in one transmission unit held in the distribution control step when it is detected that the delay in the distribution process has been eliminated for a certain period of time. Delivery method. An imaging process that generates and encodes video frames; and
A temporary storage step of temporarily storing the video frame encoded in the imaging processing step in a temporary storage unit;
A delivery control step of delivering one or more video frames stored in the temporary storage unit as one transmission unit;
A communication processing step of actually sending video frames collected as one transmission unit to a partner device;
A memory monitoring step of monitoring the memory usage of the temporary storage unit with an upper limit threshold and a recovery threshold;
Have
In the memory monitoring step, when it is detected that the memory usage of the temporary storage unit exceeds the upper limit threshold, the transmission unit of video frames to be distributed in the distribution control step is decreased,
A video distribution method comprising: increasing a transmission unit of a video frame to be distributed in the distribution control step when detecting that a memory usage amount of the temporary storage unit is less than the restoration threshold. An imaging process that generates and encodes video frames; and
A temporary storage step of temporarily storing the video frame encoded in the imaging processing step in a temporary storage unit;
A delivery control step of delivering one or more video frames stored in the temporary storage unit as one transmission unit;
A communication processing step of actually sending video frames collected as one transmission unit to a partner device;
A delay monitoring step of monitoring a delay state of the distribution processing performed in the distribution control step;
A memory monitoring step of monitoring the memory usage of the temporary storage unit with an upper limit threshold and a recovery threshold;
Have
A video distribution method comprising: increasing or decreasing the number of video frames included in one transmission unit for each partner device in the delay monitoring step; and increasing or decreasing the number of video frames included in one transmission unit in the memory monitoring step. An imaging process that generates and encodes video frames; and
A temporary storage step of temporarily storing the video frame encoded in the imaging processing step in a temporary storage unit;
A delivery control step of delivering one or more video frames stored in the temporary storage unit as one transmission unit;
A communication processing step of actually sending video frames collected as one transmission unit to a partner device;
Causing the computer to execute a delay monitoring step of monitoring a delay state of the distribution processing performed in the distribution control step;
In the delay monitoring step, when it is detected that the distribution process to the specific partner apparatus performed in the distribution control step is delayed from the generation of the video frame, one transmission unit held in the distribution control step And then controlling the computer to increase the number of video frames in one transmission unit held in the distribution control step when it is detected that the delay in the distribution process has been eliminated for a certain period of time. A computer program characterized by the above. An imaging process that generates and encodes video frames; and
A temporary storage step of temporarily storing the video frame encoded in the imaging processing step in a temporary storage unit;
A delivery control step of delivering one or more video frames stored in the temporary storage unit as one transmission unit;
A communication processing step of actually sending video frames collected as one transmission unit to a partner device;
Causing the computer to execute a memory monitoring step of monitoring the memory usage of the temporary storage unit with an upper limit threshold and a recovery threshold;
In the memory monitoring step, when it is detected that the memory usage of the temporary storage unit exceeds the upper limit threshold, the transmission unit of video frames to be distributed in the distribution control step is decreased,
A computer program for controlling a computer to increase a transmission unit of a video frame to be distributed in the distribution control step when it is detected that a memory usage amount of the temporary storage unit is less than the recovery threshold. An imaging process that generates and encodes video frames; and
A temporary storage step of temporarily storing the video frame encoded in the imaging processing step in a temporary storage unit;
A delivery control step of delivering one or more video frames stored in the temporary storage unit as one transmission unit;
A communication processing step of actually sending video frames collected as one transmission unit to a partner device;
A delay monitoring step of monitoring a delay state of the distribution processing performed in the distribution control step;
Causing the computer to execute a memory monitoring step of monitoring the memory usage of the temporary storage unit with an upper limit threshold and a recovery threshold;
The computer is controlled to increase or decrease the number of video frames included in one transmission unit for each counterpart device in the delay monitoring step, and to increase or decrease the number of video frames included in one transmission unit in the memory monitoring step. Computer program.   A computer-readable storage medium storing the computer program according to any one of claims 9 to 11.

Images