JP2010278968A - Apparatus and method for encoding, transmitting and displaying of image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus and method for encoding, transmitting and displaying of images, in which subjective image quality is prevented from being visually deteriorated or discrimination of an overall image of motion is prevented from becoming impossible, while reducing a code amount. <P>SOLUTION: An input image is encoded by an ROI (Region Of Interest) encoding section 40, and the encoded image is transmitted via a network 110 by a network interface 60. When encoding an image, an ROI encoding control unit 50 controls a code amount in accordance with a band of the network 110, and the ROI encoding section 40 separates a foreground image and a background image from the input image, encodes the foreground image by increasing frame rate of the foreground image higher than that of the background image, and encodes the background image using a combination background image obtained by combining background images in a plurality of frame terms of the foreground image. Thus, while reducing the code amount, subjective image quality is prevented from being visually deteriorated or discrimination of an overall image of motion is prevented from becoming impossible. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image encoding transmission display apparatus and an image encoding transmission display method for encoding and transmitting an image obtained by photographing with a camera.

  When an image obtained by photographing with a camera is encoded and transmitted, as shown in FIG. 23, the code amount increases at a normal frame rate and resolution, so that the code amount is reduced. As methods for reducing the code amount, methods disclosed in Patent Document 1 and Patent Document 2 are known. That is, Patent Document 1 discloses a method of changing the resolution inside and outside the ROI (Region Of Interest) region (for example, making the inside high and making the outside low). Here, ROI refers to a portion of interest in the image.

  Patent Document 2 discloses a method for lowering the resolution while lowering the frame rate outside the ROI area. In the method disclosed in Patent Document 2, for example, as shown in FIG. 24, the background region 500 of the image has a low resolution and a low frame rate, and the region of interest 501 has a high resolution and a high frame rate.

JP 2009-005238 A JP 2006-080832 A

  However, if the code amount is greatly reduced due to network congestion at the time of information transmission, the apparent subjective image quality will be greatly reduced, and the overall picture of the motion will not be understood as the frame rate and resolution decrease. There is.

  The present invention has been made in view of such circumstances, and even when the code amount is greatly reduced due to the congestion state of the network, the deterioration of the apparent subjective image quality is minimized, and the overall image of the motion is as much as possible. An object of the present invention is to provide an image encoding transmission display device and an image encoding transmission display method for encoding an input image in a state where it is easy to grasp.

  An image encoding / transmission display apparatus according to the present invention includes an encoding unit that encodes an input image, a first network interface unit that transmits an image encoded by the encoding unit via a network, and a network bandwidth. Encoding control means for controlling a code amount in accordance with the input image, the encoding means separates a foreground image and a background image from an input image, and sets the frame rate for the foreground image to the background image. The frame rate is higher than the frame rate, and the background image is encoded using a synthesized background image obtained by synthesizing the background images of the foreground images of a plurality of frame periods.

  According to the above configuration, the foreground image with motion is encoded with a frame rate higher than the frame rate of the background image, and the background image with less motion is encoded with the background of a plurality of frame periods of the foreground image as described later. Since the image is encoded using a composite background image that is synthesized as several images so that people can see and intuitively understand it, the transmitted and displayed image is subject to visual perception while reducing the amount of code. It is possible to prevent a large deterioration in image quality and to make the whole image of movement as easy to understand as possible.

  In the above configuration, the encoding control unit estimates a bandwidth of the network and changes a frame rate of the background image according to the estimated bandwidth.

  According to the above configuration, since the network bandwidth is estimated and the frame rate of the background image is changed according to the estimated network bandwidth, efficient coding according to the network bandwidth is possible. In addition, it is possible to synthesize a background image in which the length of the frame rate is intuitively determined according to the length of the frame rate, and to easily understand the entire image of the motion.

  In the above configuration, the decoding means for decoding the encoded image transmitted over the network, the second network interface means for estimating the state of the network, and the image synthesis corresponding to the state of the network Image synthesizing means.

  According to the above configuration, since the state of the network is estimated and the image is synthesized in accordance with the network state, the image with the code amount controlled can be accurately reproduced, and the transmission synthesized image Can reduce the subjective image quality and can easily grasp the whole image.

  In the above configuration, display control means is provided for performing control to display the frame rate of the background image. With this configuration, the background frame rate can be easily confirmed.

  The said structure WHEREIN: The said display control means performs control which displays the elapsed time after background update. With this configuration, the elapsed time after the background update can be easily confirmed.

  In the above configuration, the encoding unit obtains a composite background image by combining an image within a predetermined frame rate and an average of a part of images within the frame rate. With this configuration, a background image with improved S / N can be obtained.

  In the above configuration, the encoding means averages an image within a predetermined frame rate or an average of a part of images within the frame rate, and an average of the most recent image in a portion where the image changes between frames. A composite background image is obtained by overwriting the image and compositing. With this configuration, a background image having an improved S / N and a background image without blurring of the changed portion can be obtained.

  In the above configuration, the encoding means averages images within a predetermined frame rate or averages some images within the frame rate, and a portion where the image changes between frames is an image of a changed portion. A composite background image is obtained by overwriting and synthesizing the average image at a predetermined time interval. With this configuration, a background image in which the S / N is improved and a background image in which the time change of the changed portion is easily understood can be obtained.

  In the above configuration, the encoding unit causes the overwriting synthesis of the image of the changed portion to correspond to the frame rate of the background image. With this configuration, a background image with improved S / N and a background image that is easy to understand even when the frame rate is variable can be obtained.

  In the above configuration, the encoding means changes the density of the image at the time of overwriting in accordance with the frame rate of the background image in the overwriting synthesis of the image of the changed portion. With this configuration, a background image with an improved S / N and a background image that can be easily understood over time can be obtained.

  In the above configuration, the encoding unit cuts out the ROI image from the input image, and sends the coordinate value of the image after the background transmission of the background change portion to be cut out and combined together with the ROI image. With this configuration, the position of the ROI image cut out after the background transmission can be known.

  In the above configuration, the encoding unit cuts out the ROI image from the input image, and sends the coordinate value when the background change portion of the background change portion cut out and combined together with the ROI image. The ROI image cut out by this configuration and the background image can be synthesized.

  In the above configuration, the encoding unit upgrades the image of the background change portion to the ROI image in a situation where the network bandwidth is estimated to be wide. With this configuration, a background portion having a large change can be used as a new ROI image.

  In the above configuration, the encoding unit does not perform background update when a transmission error occurs due to network conditions. With this configuration, the background image is not disturbed even when a transmission error occurs.

  In the above configuration, the encoding means displays a background unupdated if the background cannot be updated when a transmission error occurs due to network conditions. With this configuration, it is understood that the background image has not been updated.

  In the configuration described above, the encoding unit divides and transmits the background area into a plurality of parts. With this configuration, background transmission according to the network bandwidth can be performed.

  In the above configuration, the encoding unit does not perform background update of the background divided into a plurality when a transmission error occurs due to network conditions. With this configuration, the background image is not disturbed at the time of transmission error even when the background is in the split display state. Even when some data is discarded due to network congestion in this way, it is possible to grasp the overall picture including the network congestion status by compositing images that can intuitively understand data discard as described later. It can be easy to do.

  In the above configuration, the display control unit performs control to display the frame rate of the background image as a numerical value or an image. With this configuration, the frame rate can be easily understood and intuitive in various situations.

  The said structure WHEREIN: The said display control means performs control which displays the elapsed time after updating a background image with a numerical value or an image. With this configuration, the elapsed time after updating the background image is easy to understand.

  In the above configuration, the display control unit performs control to reduce the density of the background change partial image added to the background image according to the elapsed time after the background image is updated. With this configuration, the elapsed time after updating the background image is easy to understand and intuitive.

  In the above configuration, the display control unit performs control to paste the transmitted foreground image on the background image at regular time intervals. With this configuration, the time change of the background image is easily understood.

  According to the image encoded transmission display method of the present invention, an encoding step for encoding an input image, an image transmission step for transmitting the image encoded in the encoding step via a network, and a bandwidth of the network An encoding control step for controlling a code amount, wherein in the encoding step, a foreground image and a background image are separated from an input image, and a frame rate of the background image is set for the foreground image. Encoding is performed at a rate higher than the rate, and the background image is encoded using a synthesized background image obtained by synthesizing background images of a plurality of frame periods of the foreground image.

  According to the above method, the foreground image with motion is encoded with the frame rate higher than the frame rate of the background image, and the background image with a plurality of frame periods of the foreground image is synthesized for the background image with less motion. Since encoding is performed using the composite background image, the subjective image quality of appearance does not deteriorate and the entire image of the motion cannot be determined while reducing the code amount.

  In the above method, a decoding step for decoding an encoded image transmitted over a network, a network state estimation step for estimating the state of the network, and an image corresponding to the state of the network are synthesized. An image composition step.

  According to the above method, since the state of the network is estimated and the images are synthesized in accordance with the state of the network, an image whose code amount is controlled can be accurately reproduced. In addition, even when data is discarded due to network congestion, it is possible to reproduce the entire image in an easily understandable manner.

  According to the present invention, when an image obtained by photographing with a camera is encoded, the subjective image quality of the appearance is not deteriorated, and the entire image of the motion is not understood, and the code amount can be reduced. it can.

The block diagram which shows schematic structure of the image coding transmission display apparatus which concerns on one embodiment of this invention The block diagram which shows the detailed structure of the ROI encoding part of the image coding transmission display apparatus of FIG. The block diagram which shows the detailed structure of the ROI decoding part of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating operation | movement of the image coding transmission display apparatus of FIG. The figure for demonstrating the conventional code amount reduction method in the case of encoding and transmitting the image obtained by image | photographing with a camera The figure for demonstrating the conventional method of lowering the resolution while lowering the frame rate outside the ROI area

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments for carrying out the invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a schematic configuration of an image encoded transmission display device according to an embodiment of the present invention. In FIG. 1, the image coding / transmission display device 1 of the present embodiment includes an imaging device 10, a display device 20, an image coding / transmission device 30, and an image reception decoding device 70. The imaging device 10 converts the subject into an image signal and outputs it. The display device 20 displays the image signal decoded by the image reception decoding device 70. The image encoding / transmission apparatus 30 encodes the image signal input from the imaging apparatus 10 and transmits the encoded image signal to the network (NW) 110.

  The image encoding transmission apparatus 30 includes an ROI encoding unit 40, an ROI encoding control unit 50, and a network interface (NWIF) 60. The ROI encoder 40 divides the image signal input from the imaging device 10 in accordance with a set condition, synthesizes and encodes the image signal. The ROI encoding control unit 50 estimates the state of the network 110 that transmits the encoded image and controls the ROI encoding unit 40. The network interface 60 converts the encoded data encoded by the ROI encoding unit 40 into a network transmission format, adds communication information necessary for network transmission, and transmits it, and confirms the congestion state of the network 110. The congestion state of the network is detected based on, for example, the discard rate of packets from the receiving side (the transmission unit of encoded data transmitted to the network) or the number of retransmissions when packet retransmission control is performed. When a congestion state is detected, the ROI encoding control unit 50 controls the ROI encoding unit 40 to control the frame rate to decrease and the code amount to decrease.

  The image receiving / decoding device 70 decodes encoded data transmitted via the network 110 and converts it into an image signal. The image receiving / decoding device 70 includes an ROI decoding unit 80, an ROI display control unit 90, a network interface (NWIF). ) 100. The ROI decoding unit 80 decodes the encoded image signal corresponding to the control signal. The ROI display control unit 90 generates a control signal for controlling the display content of the image in accordance with the image signal and the congestion state of the network 110, and provides it to the ROI decoding unit 80. The network interface 100 converts the encoded image signal transmitted in the network transmission format into an encoded image signal that can be decoded, and confirms the congestion state of the network 110 and the reception state of the image signal. From the received information, the sequence No. When it is confirmed that the transmission information is discarded, the ROI display control unit 90 controls the ROI decoding unit 80 to minimize the influence of the discarded information, and the communication status is displayed in an easy-to-understand manner. .

  FIG. 2 is a block diagram illustrating a schematic configuration of the ROI encoding unit 40 of the image encoding / transmission apparatus 30. In the figure, an ROI encoding unit 40 includes an image input unit 41, an ROI unit cutout unit 42, an image averaging unit 43, a background change cutout unit 44, a background image synthesis unit 45, and FR control timing control. A unit 46 and an image encoding unit 47 are provided. The image input unit 41 captures and temporarily accumulates images from the imaging device 10 as needed. The ROI cutout unit 42 cuts out the ROI foreground image from the input image. Foreground segmentation takes a difference between an average image obtained by averaging a plurality of frames over a certain period and an input image, and there is a difference of a certain level or more. The average image of a plurality of frames for obtaining the foreground part and the background change part described below may be an image synthesized using a median value.

  The image averaging unit 43 averages the input images according to conditions. The images in the image input unit during the period corresponding to the transmission timing of the background image are averaged. A new average is started at the timing when the background image is transmitted, and a new image is calculated. The average calculation may be a median calculation. The background change cutout unit 44 cuts out a change part of the background portion of the input image. The change part of the background part is a small area that exists far away and is above the image, and a group of parts that differ from the average image by a certain amount or more is defined as a change part of the background part (background change part). The background image synthesis unit 45 synthesizes a background image from the input image, the average image, and the background change image. The average image is used as the most basic background image. In addition, a background changing portion may be added to this average image at regular time intervals. Further, the addition of the background changing portion may be performed only once just before the background image is transmitted. When an afterimage of a portion that has moved as a background image is unnecessary, an input image that is not averaged immediately before the background image is sent may be used. In addition, as a countermeasure for discarding data due to network congestion, a still image at the time immediately before the background transmission is independently encoded with an image cut out from the background change unit. The frame rate control timing control unit 46 performs control for obtaining an optimal frame rate and an optimal image composition method for the network 110 situation. When the network is free of congestion and a wide bandwidth can be secured, the frame rate is set high, and when congestion occurs and the bandwidth becomes narrow, the frame rate is set low. The setting of the frame rate is determined by the ROI encoding control unit 50. The image encoding unit 47 encodes the foreground image and the background image in accordance with a standard such as MPEG or JPEG. Also, in order to cope with the extremely low bandwidth of the network, the coordinate values of the ROI foreground image and the coordinate values of the background changing portion are also transmitted as data separately from the image information.

  FIG. 3 is a block diagram showing a schematic configuration of the ROI decoding unit 80 of the image receiving / decoding apparatus 70. In the figure, an ROI decoding unit 80 includes an image decoding unit 81, an ROI unit synthesizing unit 82, a background image synthesizing unit 83, an image synthesizing unit 84, and a frame rate / network adaptive control unit 85. Composed. The image decoding unit 81 decodes the encoded image signal transmitted via the network 110. The ROI foreground image and background image are obtained by decoding. The ROI unit synthesizing unit 82 synthesizes the foreground image that displays the decoded foreground according to the frame rate and the state of the network 110. When the background frame rate is low, the foreground image of the ROI may be left at a constant time interval to make it easy to understand how the past ROI foreground image has moved. The foreground image of the ROI left with the passage of time may be a foreground image in which the density is reduced and the passage of time is easily understood.

  The background image synthesis unit 83 synthesizes a background image that displays the decoded background according to the frame rate and the state of the network 110. When the frame rate is lowered, the background change portion (received in the past) is overwritten and combined with the background image corresponding to the received coordinate value of the background change portion. In particular, when the background image is discarded due to network congestion and a transmission error occurs, it is observed that the background change part is overwritten and combined with the old background image together with the transmission error, and the overall situation is displayed in an easy-to-understand manner. It is effective for a person's intuitive understanding. The image synthesis unit 84 synthesizes an image to be displayed according to the frame rate and the state of the network 110 using the foreground of the synthesized ROI unit synthesis unit 82 and the background of the background image synthesis unit 83. In particular, in a transmission error state in which the background image is discarded due to network congestion, an image that is easily understood to be discarded, which will be described later, is synthesized. The frame rate / network adaptive control unit 85 uses the ROI unit synthesis unit 82, the background image synthesis unit 83, and the image synthesis unit 84 to perform image synthesis based on information indicating the congestion state of the network 110, the frame rate at the time of encoding, and the reception status. The part is optimally controlled as indicated above. When the background image data is discarded due to network congestion, control such as pasting of the foreground image at a fixed time interval and composition by coordinates of the background changing portion is performed.

  Next, the synthesis of the background image at the time of ROI encoding in the image encoding transmission apparatus 30 will be described in detail.

(Synthesis method 1)
FIG. 4 is a diagram showing an average image obtained by averaging images within the frame rate interval. For example, when the imaging device (camera) 10 outputs 60 images per second, the image encoding transmission device 30 transmits images at a frame rate (FR) of 1/8 images / second (ie, 8 images). The number of images between frames is 60 × 8 = 480. These 480 images are averaged. By averaging the images within the frame rate interval, a moving part such as a person is blurred, but the S / N (signal to noise ratio) can be improved. Further, the current frame rate (FR = 1/8 frames / second) is displayed on the average image so that the sending state can be confirmed at a glance. When the FR has to be dropped, the moving part is blurred, but the whole movement is somewhat easy according to the observer's assumption.

  FIG. 5 is a diagram showing a background composite image obtained by overwriting and synthesizing the latest image of the changed portion with respect to the average image of FIG. For example, among the 480 images described above, the 479th image is taken as the most recent image, and a portion that is different from the most recent image and the average image (only the human portion in FIG. 4) is overwritten on the average image in FIG. In this way, by overwriting the change part of the latest image on the average image, the motion of the image remains as an afterimage, an overall image of the motion can be assumed, and the blur of the moving person's image is eliminated, and the overall image is more It becomes easier to assume. In addition, the current frame rate (FR = 1/8 frames / second) is displayed on the image synthesized in this way so that the sending state can be confirmed at a glance.

The synthesis method 1 has the following characteristics.
・ You can check (imagine, assume) the whole picture of the movement.
-Even if the frame rate is greatly reduced, there is little deterioration in the subjective subjective image quality. Even if the frame rate is controlled to be longer or shorter based on the degree of congestion of the network 110, there is little deterioration in subjective image quality. -Frame rate can be confirmed, and network congestion and bandwidth can be assumed.

(Synthesis method 2)
6 and 7 are diagrams illustrating another example of a background image synthesis method at the time of ROI encoding.
FIG. 6 is a diagram showing a background composite image obtained by cutting out the moved part in the average image of FIG. 4 and using it as a motion difference image, overwriting the latest image at a fixed time interval (for example, once per second). is there. In addition, the current frame rate (FR = 1/8 frames / second) is displayed on the background composite image.
FIG. 7 is a diagram showing a background composite image obtained by cutting out a moving part in the average image of FIG. 4 and overwriting the average image at a certain time interval (for example, once per second) as a motion difference image. is there. In this case, since the image to be overwritten is an average image, an afterimage appears. However, since it is pasted once per second, for example, the shape of a person can be clearly seen. In addition, the current frame rate (FR = 1/8 frames / second) is displayed on the background composite image.

This synthesis method 2 has the following characteristics.
・ Easy confirmation (imagination, assumption) of the whole movement.
-Even if the frame rate is greatly reduced, there is little deterioration in the subjective subjective image quality. Even when the frame rate is controlled to be low based on the degree of congestion of the network 110, there is little deterioration in subjective image quality.
6 and 7 may be encoded as a moving image between the background images. Further, the cut-out image (motion difference image) may be transmitted with time, coordinates, and shape without performing background synthesis.

(Synthesis method 3)
8 and 9 are diagrams illustrating another example of a background image synthesis method at the time of ROI encoding.
FIG. 8 is a diagram showing a background composite image in which the average image is overwritten with the latest image of the changed portion, and the motion difference image 210 is combined at regular time intervals. FIG. 9 is a diagram illustrating a background composite image in which the motion difference image 210 is combined with the latest image at regular time intervals. In either case of FIG. 8 or FIG. 9, the motion difference image 210 that has passed over time is thinned and combined. In other words, the motion difference image 210 is synthesized so that the density of the motion difference image 210 increases as it approaches the current time. Further, in both cases of FIGS. 8 and 9, the current frame rate (FR) is displayed to notify the frame rate, and the time after the background image is updated (6 seconds after the update) is displayed to indicate the passage of time. Inform. In addition, by sending the coordinates of the clipped portion of the motion difference image 210 to the receiving side, it is possible to display lightly corresponding to the elapsed time after the update on the receiving side, making it easier to intuitively understand the time after the background image update. .

This background image synthesis method 3 has the following characteristics.
-The frame rate is easy to understand intuitively (if the frame rate is lowered and the image update interval is extended, the portion that is thinly combined increases).
・ If the network bandwidth cannot be secured, the frame rate is lowered (the image update interval becomes longer), but the change in the frame rate is easily understood.

Next, the following points of the method for synthesizing the ROI portion and the background image will be described.
1) The ROI part is synthesized on the background image.
2) When the ROI moves, the ROI portion is also pasted and displayed on the background at regular time intervals.
3) The ROI pasting composite part is updated and displayed in the user specified mode as follows:-It is erased after a certain time.
・ Thinner with time.
・ Reset all at background update.
-Does not reset even when the background is updated.

  The ROI image is synthesized with any one of the background synthesized images shown in FIGS. For example, FIG. 10 is a diagram illustrating a background composite image obtained by combining the background image of FIG. 6 and the ROI image 300. Only the ROI image 300 is a moving image. In addition, the current frame rate (FR = 1/8 frames / second) is displayed on the background composite image. FIG. 11 is a diagram showing a background composite image in which the background image of FIG. 8 and the ROI image 300 are combined. In addition to the current frame rate (FR = 1/8 frames / second), the background composite image displays the time after updating the background image (elapsed 6 seconds after the update).

This background image synthesis method has the following characteristics.
ROI is a moving image, and the background portion displays a still image in which the changed portion is synthesized.
Even when the background frame rate is low and the image update period is long, intuitive understanding (imagination, assumption) of the movement of the background is possible, and there is little deterioration in subjective image quality.
-Even when the bandwidth of the network 110 fluctuates and the frame rate and the elapsed time after the background update fluctuate, it is easy to understand by the change in the number and the expression of the image.

  FIG. 12 is a diagram showing a background composite image synthesized by leaving ROI images at regular intervals on the background image of FIG. That is, it is a diagram showing a background composite image in which ROI images are pasted as a still image 310 at regular intervals. The ROI image has moved in the direction of arrow A. Each ROI image 310 as a still image is displayed with the same darkness, and is erased sequentially after a predetermined time has elapsed. By periodically pasting the ROI image in this way, it becomes easier to grasp the passage of time within the background update period. In addition, it is intuitively understood that the background has been updated by deleting the still image 310 of the ROI image when the background is updated, and that the time after the background is updated. In addition, the current frame rate (FR = 1/8 frames / second) is displayed on the background composite image.

  FIG. 13 is a diagram showing a background composite image displayed by combining the background image of FIG. 9 while leaving ROI images at regular intervals and reducing the density after a predetermined time has elapsed. By diluting the density corresponding to the time after the background update, the passage of time after the background update can be understood more intuitively. In addition to the current frame rate (FR = 1/8 frames / second), the background composite image displays the time after the background image update (6 seconds have elapsed since the update).

This background image synthesis method has the following characteristics.
-By using the same display method as that for the background image, it is possible to intuitively understand (imagine, assume) the entire time passage and the time passage after the background update. Further, even when the frame rate is low and the background update period is long, the uncomfortable appearance can be reduced.
The network bandwidth (congestion state) can be intuitively understood when the frame rate is controlled in accordance with the network bandwidth, or by erasing old ROI images and changing the density (thinning) at the frame rate and the elapsed time after background update.

  FIG. 14 is a background composite image that is displayed by combining the image of FIG. 5 (an image obtained by overwriting and synthesizing the latest image of the changed portion on the average image) while leaving the ROI images at regular intervals, and reducing the density after a certain period of time. FIG. By displaying the ROI as a still image and a moving image, the flow of time can be understood. In the afterimage portion of the background image, the background image update period can be assumed, the number of still images of the ROI image and the way in which the density decreases, the time after the background update can be assumed, and even if the frame rate is not displayed numerically It is possible to obtain an image expression that can be understood only by looking at it.

  FIG. 15 shows a still image ROI image 310 left at a constant interval in the image of FIG. 8 (an image obtained by overwriting and synthesizing the latest image of the changed portion on the average image and further synthesizing the motion difference image 210 at a constant time interval). It is a figure which shows the background synthetic | combination image displayed by diminishing the density | concentration after progress for a fixed time while combining. By displaying the ROI as a still image and a moving image, the flow of time can be understood. The change part of the background image is cut out and the still image is added, and the update period of the background image can be assumed in the part where the afterimage is expressed, and the passage of time after the background update can be assumed due to the number of still images and the density of the ROI image becoming lighter. That is, even if the frame rate is not displayed numerically, it is possible to obtain an image expression in which the flow of time can be understood only by looking more intuitively.

  The composition method of the background image change cutout unit controls the display method at the frame rate. When the frame rate is low and the update interval is long, the afterimage portion becomes long and the number of portions where the changed portion is cut out and pasted increases. The ROI image composition method is controlled by the elapsed time after the background update. As the elapsed time after background update becomes longer, the number of ROI images cut out and pasted increases. By performing the above display, it is possible to intuitively understand the network status. This is a display method that can be understood intuitively without using numerical display, and is effective when monitoring many camera images at once.

Next, a display method when a transmission error occurs will be described.
When the network 110 is in an abnormal state where congestion has occurred, the transmitted data is discarded and a transmission error occurs. A countermeasure for the case where a transmission error occurs due to such an abnormality of the network 110 will be described below.
FIG. 16 is a diagram illustrating a display example at the time of transmission error in the background composite image obtained by combining the background image and the ROI moving image. The following processing is performed when a transmission error occurs.

(1) The background image 350 is retained without being updated without using the distorted background image data at the time of error. That is, since it is a transmission error, the correct data is not received, so the current state is maintained without updating.
(2) The ROI image 300 is updated according to the received information. Since the ROI image 300 is important and the number of transmissions is large, the ROI image 300 is updated as it is with the received information.
(3) If the background cannot be updated due to a transmission error, the post-update elapsed time 351 is changed to a red character (such as a conspicuous color).
(4) If the background cannot be updated due to a transmission error, the number of unupdated times 352 is displayed. Also in this case, it is displayed in red (such as a conspicuous color).

  FIG. 17 is a diagram illustrating a display example at the time of a transmission error in a background composite image obtained by combining a background image, a moving image ROI image, and a still image ROI image. The following processing is performed when a transmission error occurs.

(1) The unupdated background image 353 is combined with the coordinates corresponding to the transmitted background change portion with the newest change cutout portion in the background portion. Since the coordinate value has a small data amount, it is transmitted with a higher priority together with the ROI data. The coordinate information of the background change portion is received from the video transmission side including the camera via the network, and the newest change cutout portion in the background portion is synthesized with the unupdated background image 353. That is, when the background image has been updated, the image 353 that has changed in the background image on the unupdated background image is synthesized to notify that it is around this.
(2) The ROI image 300 is updated according to the received information.
(3) The stop motion unit 354 of the ROI image 300 is the same as the previous background. By changing the density after the passage of time (decreasing the density with the passage of time), the passage of time is represented by an image. This adaptation may be performed or not performed as necessary.
(4) If the background cannot be updated due to a transmission error, the post-update elapsed time 351 is changed to red (such as a conspicuous color). FIG. 17 shows an image when 12 seconds have elapsed since the update, and “12 seconds after the update” is displayed in red.
(5) If the background cannot be updated due to a transmission error, the number of unupdated times 352 is displayed. In this case, it is displayed in red.

  The background synthesis method may be any of the synthesis methods described above. The ROI synthesis method may be any of the above-described synthesis methods.

  FIG. 18 summarizes the display method at the time of transmission error, and is an image when 3 seconds have elapsed from the state shown in FIG. 17, and “15 seconds after update” is displayed in red. The background image 350 is a previous background image when the background cannot be updated due to a transmission error, and is synthesized on the transmission side. The ROI moving image 300 is the latest ROI moving image. The ROI still image 355 is a still image obtained from the ROI moving image immediately after the transmission error. The ROI cut-and-paste image 354 is an ROI image that could not be updated due to a transmission error. The ROI cut-and-paste image 354 can have several variations such as thinning or erasing the image corresponding to the post-update elapsed time. In FIG. 18, the image is displayed corresponding to the elapsed time. It is thin. The background change portion 356 is a portion that is additionally combined after the transmission error has occurred with the transmitted coordinate value, and is combined on the receiving side based on the coordinate information sent on the transmitting side.

Next, a display method when a transmission error occurs during background image division will be described.
19 and 20 are diagrams illustrating display examples when an image is divided into four.
Region (1) has not been updated once (10% lighter, 10% sepia)
Region (2) has not been updated twice (20% lighter, 20% sepia)
Areas (3) and (4) have no background update (do not display lightly, nor change color)

In the area (4) of FIG. 19, the correct value is displayed as a character (numerical value), and is displayed in a color different from the normal display when there is an abnormality (such as when the screen cannot be updated due to a transmission error).
In the area (4) of FIG. 20, the rectangle in the lower right is an indicator 400 indicating that the background has not been updated. In this indicator 400,
Area (1) is the background not updated once (displayed in yellow)
In area (2), the background has not been updated twice (displayed in orange)
Areas (3) and (4) have no background update (displayed in green)
Indicates.
The background division need not be limited to four. Also, if there are many divisions, the area of the area where the background cannot be updated at the time of transmission error becomes small.

  FIG. 21 is a diagram illustrating the transmission order of image data. In FIG. 21, 101 is background image data to be transmitted first, 102 is ROI foreground image data to be transmitted after data transmission of the background image data 101, and 103 to 105 are ROI foreground image data to be transmitted after data transmission of the ROI foreground image data 102. The group 106 is background image data transmitted for the second time. The number of ROI foreground image data in the ROI foreground image data groups 102 to 105 increases or decreases in accordance with the interval of the background frame rate, and the background FR = 1/8 sheets / When the ROI foreground is FR = 30 frames / second, 240 image data are obtained. Reference numerals 107 to 110 denote 240 pieces of ROI foreground image data, 111 denotes background image data to be transmitted for the third time, and 112 denotes ROI foreground image data to be transmitted thereafter.

  FIG. 22 is a diagram illustrating a transmission transmission order when background transmission image data is divided into four and transmitted. In FIG. 22, 120 is background image data divided into four parts to be transmitted first, 121 and 122 are ROI foreground image data groups transmitted after data transmission of the background image data 120, and 123 is the next of the four parts divided into four parts to be transmitted for the second time. The number of ROI foreground image data between the background image data 120 and 123 and the ROI foreground image data groups 121 and 122 increases or decreases in accordance with the background frame rate interval, and the background FR = 1/8. When the background is divided into four at FR / 30 sheets / second and ROI foreground FR = 30 sheets / second, the image data is 60 sheets.

  124-125 is a part of 60 image data groups of ROI foreground image data, 126 is a third divided fourth background image data transmitted for the third time, and 127 to 128 are ROI foreground image data groups transmitted thereafter, Reference numeral 129 denotes the last remaining background image data divided into four, and transmission of the background image data divided into four is completed. Thereafter, the process is repeated. 132, 135, 138, 141, 144, 147, 150, and 153 are divided background image data, and 130 to 155 other than the divided background are ROI foreground image data.

  As described above, according to the image encoding / transmission display apparatus 1 of the present embodiment, the input image is encoded by the ROI encoding unit 40 and the encoded image is transmitted by the network interface unit 60 via the network 110. To do. At the same time, the network interface unit 60 estimates the transmittable bandwidth of the network 110 and sends information to the ROI encoding control unit 50. When an image is encoded, the ROI encoding control unit 50 controls the code amount according to the bandwidth of the network 110, and the ROI encoding unit 40 separates the foreground image and the background image from the input image, and the foreground image On the other hand, encoding is performed with a frame rate higher than the frame rate of the background image, and the background image is encoded using a synthesized background image obtained by synthesizing background images of a plurality of frame periods of the foreground image. That is, the foreground image with motion is encoded with a frame rate higher than the frame rate of the background image, and the background image with less motion is encoded with a background image of a plurality of frame periods of the foreground image as shown in the embodiment. Therefore, while reducing the amount of code, the subjective image quality of appearance does not deteriorate, and the whole image of the motion is not understood. In addition, the ROI encoding control unit 50 and the network interface unit 60 estimate the bandwidth of the network 110 and change the frame rate of the background image and the frame rate of the ROI foreground image according to the estimated bandwidth. Can be realized. Also, the encoded image transmitted through the network 110 is received by the network interface 100, and the ROI decoding unit 80 decodes it. In addition, the network interface 100 estimates the state of the network 110, and the ROI decoding unit 80 synthesizes an image corresponding to the estimated network state, so that an image whose code amount is controlled is accurately reproduced. be able to. Even when data is discarded due to network congestion and transmission errors occur, the status of transmission errors is displayed on the screen in an easy-to-understand manner, and the display screen is not disturbed, enabling transmission and display of very high-quality images. be able to. As described above, according to the present invention, even when the network bandwidth is greatly limited, the ROI image and the background image are synthesized, and the image that makes the best use of the network bandwidth is synthesized, encoded, and transmitted. . The receiving side also detects a transmission error due to network congestion, displays the transmission error in an easy-to-understand manner, and displays an image without disturbing the image. By such a system, it is possible to realize transmission display of an image that is easy to understand (high subjective image quality) even in a situation where the bandwidth of the network varies greatly.

  The present invention has an effect that the subjective image quality of appearance does not deteriorate and the whole image of motion is not understood while reducing the code amount, and can be applied to a surveillance camera or the like. .

DESCRIPTION OF SYMBOLS 1 Image encoding transmission display apparatus 10 Imaging apparatus 20 Display apparatus 30 Image encoding transmission apparatus 40 ROI encoding part 41 Image input part 42 ROI part extraction part 43 Image average part 44 Background change extraction part 45 Background image synthetic | combination part 46 Frame rate control timing control unit 47 Image encoding unit 50 ROI encoding control unit 60 Network interface unit 70 Image reception decoding device 80 ROI decoding unit 81 Image decoding unit 82 ROI unit synthesizing unit 83 Background image synthesizing unit 84 Image synthesis 85 Frame rate / network adaptive control unit 90 ROI display control unit 100 Network interface unit 110 Network

Claims (23)

Encoding means for encoding an input image;
First network interface means for transmitting the image encoded by the encoding means via a network;
Coding control means for controlling the amount of code according to the bandwidth of the network,
The encoding means separates a foreground image and a background image from an input image, encodes the foreground image with a frame rate higher than the frame rate of the background image, and for the background image, An image encoding transmission display apparatus for encoding using a synthesized background image obtained by synthesizing background images of a plurality of frame periods of the foreground image.   The image coding transmission display apparatus according to claim 1, wherein the coding control means estimates a band of the network and changes a frame rate of a background image according to the estimated band. Decoding means for decoding the encoded image transmitted over the network;
Second network interface means for estimating the state of the network;
Image synthesizing means for synthesizing images corresponding to the state of the network;
The image encoding transmission display apparatus of Claim 1 or Claim 2 provided with these.   4. The image encoded transmission display apparatus according to claim 1, further comprising display control means for performing control to display a frame rate of the background image.   The image encoded transmission display apparatus according to claim 4, wherein the display control unit performs control to display an elapsed time after background update.   The said encoding means obtains a synthetic | combination background image by carrying out the synthesis | combination by averaging the image within the defined frame rate, or averaging some images within the frame rate. Image encoding transmission display device.   The encoding means averages images within a predetermined frame rate or averages some images within the frame rate, and overwrites the newest image on the average image when the image changes between frames. 6. The image encoded transmission display apparatus according to claim 1, wherein a synthesized background image is obtained by combining the images.   The encoding means averages images within a predetermined frame rate or averages a part of images within the frame rate, and a portion where the image changes between frames indicates an image of a changed portion at a predetermined time. 6. The image encoded transmission display apparatus according to claim 1, wherein a synthesized background image is obtained by overwriting and synthesizing the average image at intervals.   The image encoding transmission display apparatus according to claim 8, wherein the encoding unit associates overwriting synthesis of the image of the changed portion with a frame rate of a background image.   The image encoding transmission display apparatus according to claim 9, wherein the encoding means changes the density of the overwritten image corresponding to the frame rate of the background image in the overwriting combination of the image of the changed portion.   The image according to any one of claims 1 to 10, wherein the encoding means cuts out the ROI image from the input image, and sends the coordinate value of the image after background transmission of the background change portion to be cut out and combined together with the ROI image. Encoded transmission display device.   11. The image code according to claim 1, wherein the encoding unit cuts out the ROI image from the input image, and sends the coordinate value of the background change portion that has been cut out and combined together with the ROI image. Transmission display device.   The image encoding transmission display apparatus according to claim 11 or 12, wherein the encoding means upgrades an image of a background change portion to an ROI image in a situation where it is estimated that a network bandwidth is wide.   The image encoding transmission display apparatus according to any one of claims 2 to 13, wherein the encoding means does not perform background update when a transmission error occurs due to network conditions.   The image encoding transmission according to any one of claims 2 to 13, wherein the encoding means displays a background unupdated when a background error is not performed when a transmission error occurs due to a network condition. Display device.   The image encoding transmission display apparatus according to any one of claims 1 to 15, wherein the encoding means divides and transmits a background area into a plurality of parts.   The image encoding transmission display apparatus according to claim 16, wherein the encoding means does not perform background update of a background divided into a plurality when a transmission error occurs due to network conditions.   18. The image encoded transmission display apparatus according to claim 4, wherein the display control unit performs control to display a frame rate of a background image as a numerical value or an image.   The image encoded transmission display apparatus according to any one of claims 4 to 17, wherein the display control unit performs control to display an elapsed time after updating the background image as a numerical value or an image.   20. The image code according to claim 4, wherein the display control unit performs control to reduce a density of the background change partial image added to the background image according to an elapsed time after the background image is updated. Transmission display device.   21. The image encoded transmission display apparatus according to claim 4, wherein the display control unit performs control to paste the transmitted foreground image on a background image at a constant time interval. An encoding step for encoding the input image;
An image transmission step of transmitting the image encoded in the encoding step via a network;
An encoding control step for controlling a code amount in accordance with a network bandwidth,
In the encoding step, a foreground image and a background image are separated from the input image, and the foreground image is encoded with a frame rate higher than the frame rate of the background image. An image encoding transmission display method for encoding using a synthesized background image obtained by synthesizing background images of a plurality of frame periods of the foreground image. A decoding step of decoding an encoded image transmitted over the network;
A network state estimating step for estimating the state of the network;
An image compositing step for compositing an image corresponding to the state of the network;
The image encoded transmission display method according to claim 22, further comprising:

Images