JPH07288806A - Moving image communication system - Google Patents

Abstract

PURPOSE:To designate the part to be observed in details and its resolution by deciding automatically the resolution of each part of an image depending on the allowable transmission quantity when the image data are sent while being compressed. CONSTITUTION:A transmission system 100 or a reception system 120 is provided with area designation means 123, 125 to designate an optional area in a moving image to designate a part that is desired to be observed in details by the user. Accordingly, image processing means 103,104 of the transmission system set the resolution of the designated area higher than that of the other areas to reduce the image data quantity. Or the resolution may be set by the user. The traffic of the communication network is measured and the resolution other than the designated area is decided to satisfy automatically the allowable transmission quantity.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image communication system including a transmitting system and a receiving system for transmitting and receiving moving images (hereinafter referred to as moving images) via a communication network.

[0002]

2. Description of the Related Art Generally, moving image information has a huge amount of data. Therefore, in order to communicate a moving image through a network with a low transmission capacity, an image compression method for reducing the amount of moving image data transmitted (hereinafter referred to as traffic amount) has been studied. For example, "AnImage Knowledge Based Vi
deo CODEC For Low Bitrates: RHJM Plompenetc.
SPIE p379-p384 Vol. 804 (1987) ", etc.

As an example of a conventional image compression method, a transmission side compresses a part of image data of a moving image, and a reception side decompresses the compressed image data according to a compression protocol, whereby a communication network is provided. There is a known difference method for reducing the traffic volume of the.

For example, as a moving image compression method used in an electronic conference system, there is a method of compressing image data on the basis of a change amount (difference) between successive screens. That is, in view of the fact that a moving image includes a moving portion and a stationary portion, only the stationary portion is transmitted once, and then the moving portion image data is transmitted. . As a result, the traffic volume can be reduced without substantially lowering the image quality such as the resolution of the moving image.

Even if the image data to be transmitted is compressed by such a difference method, in the case of a network having a remarkably low transmission capacity (for example, Ethernet, ISDN, etc.), problems such as transmission delay may remain.

In such a case, in order to further reduce the transmission amount of image data, it is conceivable to uniformly reduce the resolution of the moving portion to reduce the information amount.

Further, it is conceivable to perform higher compression by a known moving image compression algorithm such as MPEG method.

Further, in the case of the electronic conference system, since the object of communication is mainly the movement of a person, when transmitting image data on the transmitting side, the face portion and the shoulder portion of the person are recognized and the face portion is recognized. Can be detailed and the shoulders can be coarse. As a result, it is possible to delete the image data amount of the shoulder portion even in the moving portion.

[0009]

However, the conventional technique has a problem because the following points are not taken into consideration.

(1) The user cannot specify the portion he wants to see in detail. For example, in the electronic conference system, the face part of the attendee is sent in a detailed image,
Since the resolution of other parts is dropped and sent,
Even if the user tries to look at a part other than the face in detail, it is not possible to specify to increase the resolution of the part and send it.

(2) The user cannot specify the resolution of the portion that he or she wants to see in detail. (3) It cannot cope with fluctuations in traffic volume of the communication network.

That is, in the conventional electronic conferencing system, the traffic volume does not fluctuate because it is assumed that the communication network is a dedicated telephone line whose communication permission is predetermined.

However, in the case of an ordinary computer network, since a plurality of computers share one communication network, even if one computer tries to communicate with another computer, other computers are communicating with each other, so that the traffic volume of the communication network is increased. May increase and the allowable transmission capacity may be insufficient.

In such a case, according to the conventional image compression method, since the compressed image data amount is constant, a transmission delay occurs due to the transmission waiting of the image data, and the moving image can be transmitted in real time. Disappear.

(4) The camera movement is not taken into consideration. According to the conventional technology, when the camera is panned or zoomed, the part that is transmitted in high resolution is changed according to the movement of the camera, even though the part to be viewed in detail moves on the screen. I haven't. Therefore, when the camera moves, there is a disadvantage that the transmitting side transmits in detail the image of a completely different part, not the part that the user wants to see.

An object of the present invention is to provide a moving image communication system in which a user can specify a portion he or she wants to see in detail when reducing the amount of image data transmitted.

Another object of the present invention is to provide a moving image communication system in which the user can specify the quality of the image of the portion that the user wants to see in detail.

Another object of the present invention is to provide a moving picture communication system capable of transmitting a moving picture with a required quality according to a change in traffic volume of a communication network.

Another object of the present invention is to provide a moving image communication system capable of automatically adjusting the resolution of each part of an image according to the state of the camera.

[0020]

In order to achieve the above object, a moving image communication system according to the present invention includes a transmission system for transmitting image data of a moving image to a communication network, and the image data via the communication network. And a receiving system that displays the moving image on a display, and one of the transmitting system and the receiving system is provided with a region designating unit that designates an arbitrary region in the moving image, The transmission system is provided with image processing means for reducing the image data amount for each of the designated area designated by the area designation means and the non-designated area not designated.

In this case, resolution setting means for setting at least one of the resolution of the designated area and the resolution of the non-designated area is provided, and the image processing means
It is preferable to reduce the image data amount of the designated area and the non-designated area based on the resolution set by the resolution setting unit.

In addition to the above-mentioned invention, another invention of the present invention is provided with a resolution setting means for setting the resolution of the designated area designated by the area designating means, and the transmission system is set by the resolution setting means. Reducing the image data amount of the designated area based on the resolution, and calculating the resolution of the non-designated area not designated by the area designating means so as to satisfy the allowable transmission capacity of the communication network,
It is preferable to include image processing means for reducing the image data amount of the non-designated area based on the calculated resolution.

In this case, it is preferable that the area designating means is formed so that a user can interactively set the designated area.

In another aspect of the present invention, means for measuring the current traffic volume of the communication network is provided, and the resolution of the non-designated area is calculated according to the current traffic volume measured by this means. It is preferable.

In the above invention, the resolution may be spatial resolution, temporal resolution, or a combination thereof.

Here, the spatial resolution means the amount of image information per unit area of the screen. For example, the number of pixels per unit area, the number of colors of a color image, the resolution of the amplitude of a frequency signal obtained by frequency-converting image data, the resolution of the amplitude of pixel data, and the quantization step size when quantizing pixel data. Means at least one of

The time resolution means the amount of image information per unit time viewed on the time axis of the image. For example, the time resolution can be changed by changing the frame rate, which is the number of screens per unit time, or the thinning-out interval of fields (called the number of interlaces) in interlacing in which scanning lines are skipped.

In addition to the above-mentioned present invention, a camera detection means for detecting the movement of the camera such as the focus position and zoom ratio of the TV camera is provided, and the designated area is determined according to the movement of the camera detected by the camera detection means. It is preferable to provide designated area correction means for correcting the position and size of the. Similarly, instead of the movement of the camera, the movement of the designated image in the designated area can be detected to correct the position and size of the designated area.

[0029]

According to the above solving means, the object of the present invention is achieved by the following operations. Since the area designating means for designating an arbitrary area in the moving image is provided, the user can designate the portion to be viewed in detail.

The transmission system reduces the image data amount for each designated area and non-designated area. Therefore, by lowering the resolution of the image of the portion that the user wants to see in detail and reducing the resolution of other portions, the image of the required portion can be transmitted in detail even if the transmission capacity of the communication network is low.

By the resolution setting means of the designated area, detailed image information according to the user's preference is provided.

Since the resolution of the non-designated area is calculated so as to satisfy the allowable transmission capacity of the communication network, the transmission delay of image data can be prevented.

In particular, if the resolution of the non-designated area is calculated according to the measured current traffic amount, it is possible to surely prevent the transmission delay of the image data. That is, when an increase in the traffic amount is detected, the resolution of the non-designated area is calculated again and the image data is further reduced. In this case, if it is necessary to further reduce the transmission amount, the resolution of the designated area can be corrected to be lower than the designated resolution.

When the position of the camera or the zoom ratio changes, the position and size of the designated area are corrected in association with the movement of the camera, so that the user can be provided with a moving image that does not cause discomfort. Similarly, by designating a video in the designated area, detecting its movement, and moving the designated area in accordance with the movement of the designated video, it is possible to provide a moving image that does not give a feeling of strangeness to the user.

[0035]

【Example】

(First Embodiment) FIG. 1 shows the overall configuration of a moving image communication system according to the first embodiment of the present invention.

This embodiment is a moving image communication system for transmitting a moving image having an enormous amount of data via a communication network having a limited data transmission capacity. The feature of the present embodiment is that the user can interactively designate a designated region (hereinafter referred to as a region of interest) designated by the user and a non-designated region other than that (hereinafter referred to as a background region), and those regions are designated. Each time the spatial resolution of the image is changed. As a result, the amount of data in the entire image is reduced, while the image in the region of high user interest is sharpened. In this example, the spatial resolution is changed by thinning out the pixels of the image. The spatial resolution means the resolution per unit area of the screen. Further, in the present invention, the term “resolution” can be rephrased as “detailed degree” or “fineness degree” indicating the detail or fineness of an image in the present invention.

As shown in FIG. 1, the system includes an image transmitting unit 100, an image receiving unit 120, and a network 1 for performing data transmission between the image transmitting unit 100 and the image receiving unit 120.
Divided into 10 parts.

The image transmitting section 100 includes a TV camera 101,
The moving image capturing unit 102, the background image processing unit 103, the area image clipping unit 104, the transmission side control unit 105, and the transmission side communication unit 106 are included.

The moving image capturing section 102 is the TV camera 1
The moving image captured with 01 is digital RGB (Red, Green, B
lue) Performs the process of loading into memory as a signal. The area image cutout unit 104 cuts out an image of the region of interest from the image data captured in the memory by the moving image capturing unit 102, and reduces the amount of image data of the region of interest at the reduction ratio specified by the user. The background image processing unit 103 calculates the reduction rate of the image in the background area and reduces the image data amount of the background area by the reduction rate. The transmission-side communication unit 106 transmits the reduced image data of the region of interest and the image data of the background region to the image receiving unit 120, or the image receiving unit 12 respectively.
It receives the coordinate data of the region of interest and the reduction ratio transmitted from 0, and measures the load of the network 110. The transmission side control unit 105 controls the entire image transmission unit 100.
For example, the transmission side control unit 105 uses the image capturing process,
The control of the calculation process of the reduction ratio of the background region image, the reduction process of the background region image, the region of interest image cutout process, the reduction process of the region of interest image, the network communication process, the measurement process of the network traffic volume, and the ending process are performed.

The image receiving section 120 includes a receiving side control section 12
1, a receiving side communication unit 122, a region determination unit 123, an image composition unit 124, a mouse (input unit) 125, and a display (image display unit) 126. The receiving-side communication unit 122 receives the image data of the region of interest and the image data of the background region transmitted from the transmitting-side communication unit 106, and transmits the coordinate data of the region of interest and the reduction rate to the transmitting-side communication unit 106. Mouse 125
Is an input unit for the user to interactively specify a region of interest and a reduction ratio thereof. The area determination unit 123
The coordinates and the reduction rate in the screen of the area designated by the mouse 125 are stored and managed. The image synthesizing unit 124 enlarges the image data of the region of interest and the image data of the background region received by the communication unit 122 on the receiving side, synthesizes the image data, and displays the synthesized image on the display 126. Then, the receiving-side control unit 121 controls the entire image receiving unit 120, for example,
It controls network communication processing, ROI specification processing, image composition processing, and termination processing.

According to the basic operation of this embodiment, the image sent via the network 110 is displayed on the display 126 of the image receiving section 120. The user looks at the image displayed on the display 126 and uses the mouse 125.
Is operated to specify a region of interest to be viewed in detail and a parameter (reduction rate) for determining the spatial resolution of the region. This reduction ratio is set to be large when the interest in the region of interest is large and small when the interest is small. The reduction rate of the background region excluding the region of interest is calculated by the image transmitting unit 100. The image transmission unit 100 reduces the image data corresponding to each area based on these reduction rates. In this way, the image data reduced in accordance with the user's preference flows through the network 110, and the image receiving unit 1
20 and is displayed on the display 126.

Next, the detailed configuration of the image communication system of this embodiment will be described together with the operation. The image transmission unit 1 is shown in FIG.
00 shows a flowchart of processing operation, and FIG. 3 shows a flowchart of processing operation in the image receiving unit 120.

(Processing Operation of Image Transmission Unit 1: Step 20
0 to 205) Step 200: When the image transmitting unit 100 is started, one screen (frame) of a moving image captured by the TV camera 101 is captured by the moving image capturing unit 102 as a digital RGB signal in the memory.

Step 201: It is judged whether or not a region of interest is designated. If the user does not specify the region of interest, the process proceeds to step 205,
The transmission side communication unit 106 receives the image data captured in the memory.
To the receiving side communication unit 122 via the network 110. Even if the user does not specify the ROI, the entire area of the screen can be treated as the ROI. On the other hand, if the region of interest is designated, the process proceeds to step 202.

Step 202: Area image cutout unit 10
4, the array of image data of the region of interest is read from the image data captured in the memory, and the image cutting process of the region of interest is performed.

Steps 203 and 204: A region of interest image reduction process and a background region image reduction process are performed.

Step 205: The processed image data is transmitted from the transmitting side communication section 106 to the image receiving section 120 via the network 110.

(Processing Operation of Image Receiving Unit) Step 220: When the image receiving unit 120 is started, the receiving side communication unit 122 receives the image data sent from the image transmitting unit 100.

Step 221: It is judged whether or not a region of interest is designated. Then, when the user does not specify the region of interest, the process proceeds to step 223, and the image data received by the receiving side communication unit 122 is displayed on the display 126. On the other hand, if the region of interest is designated, the process proceeds to step 222.

Step 222: The image synthesizing unit 124 performs an image synthesizing process for synthesizing the image data of the region of interest and the image data of the background region.

Step 223: The image data is displayed on the display 126.

Step 224: The user's reception end instruction is received by the mouse 125. For example, mouse 12
5 has three button switches, a left button, a middle button, and a right button. Therefore, the reception end instruction can be specified by double-clicking the middle button.

If there is an end instruction here, step 2
Proceed to 27, end processing is performed. On the other hand, if there is no end instruction, the process proceeds to step 225.

Step 225: A region of interest is designated. In other words, the user looks at the image displayed on the display 126 and determines the region of interest and its reduction ratio with the mouse 125.
Specify using.

Here, an example of a method of designating a region of interest will be described with reference to FIG. 3 shown in bold in FIG.
A case in which one region of interest 301, 302, 303 is designated will be described. First, the user displays the display 126.
The mouse 125 is operated while observing the window 300 inside, and the region of interest is designated by a rectangular region. FIG. 5 (A),
FIG. 5B shows a state of designating a region of interest. When specifying a region of interest, first one vertex 4 of the region of interest is specified.
Specify 00. To specify one vertex 400 of this region of interest, move the cursor 304 over the vertex 400 and press the left button of the mouse 125. Next, when the cursor is moved while the left button is kept pressed, the rectangle 401 is displayed. The rectangle 401 indicates the region of interest, and the point where the left button of the mouse 125 is released becomes the diagonal point of the vertex 400, and the region of interest 401 is determined. The designated region of interest 401 is recognized by the coordinates of the vertex 400 and its diagonal points.

After designating the region of interest, the user designates the reduction ratio in window 310. That is, when the region of interest is designated, the reduction ratio setting device 311 for designating the reduction ratio of the region of interest is displayed in the window 310. The user operates the reduction rate setting device 311 with the mouse 125. The right end of the black portion 312 of the reduction rate setting unit 311 indicates the reduction rate, and for example, the reduction rate in the window 310 indicates 1/2. The reduction rate can be specified by a real value from 1 to 1/4. If the cursor 304 is moved to the left or right while pressing the left button on the reduction rate setting device 311 in the window 310, the black portion moves to the left or right. When the user releases the left button of the mouse 125 when the scale of the reduction rate setting unit 311 points to the desired reduction rate, the reduction rate can be determined to that value.

Similarly, with respect to the regions of interest 301 and 303, the range and the reduction ratio are designated by the left button of the mouse 125. If the user wants to erase the designation of the region of interest, the designation of the region can be erased by clicking the right button of the mouse 125 once in the region of interest.

In this way, the regions of interest 301, 30
When the designations of 2, 303 are completed, the middle button of the mouse 125 is clicked once to instruct that there are no more regions of interest. Information on the region of interest designated in this way and the reduction ratio thereof is stored and managed in the region determination unit 123.

Here, the coordinate system for recognizing the region of interest will be described. The coordinate system set in the window 300 is duplicated in FIG. In this example, window 30
The size of 0 is 320 pixels horizontally and 240 pixels vertically. The coordinates of the region of interest are represented by the lower left vertex coordinates and the upper right vertex coordinates. For example, the coordinates of the region of interest 301 are A
(120,68): B (238,176), region of interest 30
The coordinates of 2 are C (36,188): D (120,232),
The coordinates of the region of interest 303 are E (4,38): F (28,9)
2).

In this way, the process of designating the region of interest is performed in step 225. If the ROI is not specified, the process returns to step 220 and the network communication process is performed. If specified, go to step 226.

Step 226: The coordinate data and the reduction ratio of the region of interest designated in step 225 are used as the receiving side communication section 122.
Send from

(Processing Operation 2 of Image Transmission Unit: Step 20)
6 to 211) Step 206: Determine whether there is a communication end instruction from the image receiving unit 120. If there is a communication end instruction, the communication is ended in step 212. If not, step 2
The processing after 07 is executed.

Step 207: It is judged whether or not there is received data in the transmitting side communication section 106. If there is received data, the process proceeds to step 208, and if there is no received data, the process proceeds to step 209.

Step 208: The coordinate data of the region of interest and the reduction rate are received by the receiving side communication section 122 and are taken into the inside.

Step 209: It is judged whether or not a region of interest is designated. If the region of interest is not designated, the process returns to step 200 and the above process is repeated. If the region of interest has been designated, the process proceeds to step 210.

Step 210: The transmission side communication section 106 measures the traffic amount (network load amount or data transmission amount) of the network 110.

Here, an example of a method of measuring the traffic volume of the network 110 will be described with reference to FIGS. When performing network communication between the terminals 1 to 3, a communication buffer is prepared in the terminal, and data transmitted from the terminal is temporarily buffered for communication 601, 611, 621 before being sent to the network. Stay in. That is, although not shown, the transmission side communication unit 106 and the reception side network reception unit 122 of FIG. 1 are provided with a communication buffer. Then, when the image data is transmitted from the image transmission unit 100, the network 11 is transmitted through the communication buffer prepared in the transmission side communication unit 106.
Data flows to 0.

FIGS. 6 and 7 show a large amount of data in the communication buffer 600 in the transmission communication unit 106, and FIG. 6 shows a case in which the amount of network traffic is small.
Indicates that the network traffic is heavy. When the traffic volume of the network 110 is small as shown in FIG. 6, the data to be transmitted easily flows into the network 110, and therefore the data retention volume in each of the communication buffers 601, 611, 621 is small. However, as shown in FIG. 7, when the traffic volume of the network 110 is large, it is difficult for the data to be transmitted to flow in the network 110.
A large amount of data is accumulated in 01, 611 and 621. Therefore, the communication buffers 601, 611, 6
By measuring the amount of data retained in 21, the traffic amount of the network 110 can be known. That is, when the traffic volume of the network 110 is large, the data volume in the communication buffer 601 is large, and when the traffic volume of the network is small, the data volume in the communication buffer 601 is small.

For example, TCP / I on UNIX system
When data transmission is performed by the P protocol, the command "send" is used. “Send” is a command for communicating the designated data array by TCP. This command has a "return value" that indicates the amount of data that can be transmitted. However, if data cannot be transmitted at all, "-1" is returned. For example, from the transmission side communication unit 106 to the reception side communication unit 1
When the image data is transmitted to 22, the data is collectively transmitted for each screen. When the traffic volume of the network is small, the return value of "send" is the data volume of one screen. However, if the network traffic is heavy, the return value of "send" is "-
1 ”or the amount of data is less than the amount of data for one screen. In this way, whether the network traffic amount is large or small can be known depending on whether the return value of "send" is the data amount for one screen.

As another network load measuring method, a method in which a network analyzer is connected to the network 110, the load is measured by the device, and the load information is transmitted to the image transmitting unit 100 can be considered.

After the network load measurement process is performed in step 210 in this way, step 211
Proceed to.

Step 211: The background image processing unit 103 calculates the image reduction ratio of the background area. The image reduction ratio of the background region is calculated based on the data amount and reduction ratio of the original image of the region of interest and the traffic amount of the network 110.

In order to make the total amount of data flowing through the network 110 constant, a method of greatly reducing the resolution of the image of the background area and a small resolution of the image of the region of interest in accordance with the traffic volume of the network, and a method of reducing the background area There are two types that can reduce the resolution of the image. Also,
There are two methods for calculating the reduction ratio of the background area image. The first method is a method of reducing the image data of only a portion corresponding to the background region excluding the region of interest, and the second method is a method of reducing the image data over the entire region.

Here, a method of calculating the reduction rate of the image data in the background area will be described. The total amount of image data flowing in the network 110 is the sum of the product of the number of pixels Ni ₀ [pixels] of the original image of the plurality of regions of interest i, the reduction ratio Di of the image data of the specified region of interest i, and the background region. It is the sum of the number of pixels Nb ₀ [pixel] of the original image and the product of the reduction ratio Db of the image data of the background area. Then, the total amount of image data flowing through the network 110 is the total number of pixels Nd [pixels] per screen after reduction, which is expressed by the equation (1). However, Σ represents the total sum of the regions of interest i (i = 1 to N).

[0075]

Nd = Σ (Ni ₀ × Di) + Nb ₀ × Db Further, the number of pixels Nb ₀ [pixel] of the original image in the background area is represented by the equation (2).

[0076]

[Formula 2] Nb ₀ = N ₀ −ΣNi _{0 Further} , the total number of pixels Nd [pixels] of the reduced image is defined as the allowable transmission capacity C [bits / second] of the network, the frame rate Fr [frames / second], and 1 pixel. When expressed by using the information amount P [bits] of

[0077]

## EQU00003 ## Nd = C / Fr.P frame rate, the amount of information of one pixel, and the number of pixels of the original image of the entire screen are preset values, so the reduction ratio of the background area image is the original area of the region of interest. It depends on the number of pixels in the image, the reduction ratio of the region of interest image, and the amount of network traffic.

The resolution is controlled as follows so that the total amount of data flowing through the network becomes constant. That is, when the number of pixels of the image of the region of interest is small, the reduction rate of the image data of the region of interest is large, or the network load is small, the spatial resolution of the background region image is not lowered so much. On the other hand, when the number of pixels of the original image of the region of interest is large, the reduction ratio of the region of interest image is small, or the network load is large, the spatial resolution of the background region image is greatly reduced. As a result, moving images can be transmitted without causing a decrease in frame rate due to the influence of network communication capacity and network traffic volume.

For example, consider a case where the reduction ratio of the background area image of the window 300 of FIG. 4 is calculated. Window 3
00 is full color (8 bits for each of RGB), the size of the screen is 320 × 240 pixels, 15 frames / sec, and the communication capacity is 10 megabits / sec. It is assumed that the ROI 301 has a reduction ratio of 1, the ROI 302 has a reduction ratio of 1/2, and the ROI 303 has a reduction ratio of 1/3. The number of pixels of the original image of the region of interest 301 is 12,
971 pixels, the reduction ratio is 1, so the screen is not reduced. The number of pixels of the original image of the region of interest 302 is 3,825 pixels, and the reduction ratio is 1/2, so the number of pixels after reduction is 1,913 pixels, the number of pixels of the original image of the region of interest 303 is 1,375, and the reduction ratio is 1/3, so the number of pixels after reduction is 459. Therefore, the total number of pixels of the region of interest image after being reduced is 1,5433 pixels.
The number of pixels of the original image in the background area is 58,629 (= 3
20 × 240-15343) pixels.

(Background Area Reduction Ratio Calculation Method 1) Here, the first method for calculating the reduction ratio of the image data in the background area will be described.

In this case, since the amount of data flowing through the network is controlled by reducing the resolution of only the background area, the reduction rate of the image data in the background area is expressed by the equation (4).

[0082]

Db = {(ΣNi ₀ × Di) −C / Fr · P} / Nb ₀ Let us consider a case where image data can be transmitted at a maximum transmission capacity of 10 megabits / second.
When screen data is transmitted at 15 frames / sec with this transmission capacity, the amount of data per frame is 667 kilobits (= 10 megabits / sec / 15 frames / sec). When the information amount of one pixel is 24 bits, the number of pixels per frame is 27,777 pixels (= 667 kilobits / 24 bits). Therefore, 27,77
Total number of pixels in region of interest image after reduction from 7 pixels 15,34
The pixel number of 12,434 pixels obtained by subtracting 3 pixels is the pixel number of the background area image after reduction. That is, 0.212 (=
12434/58629) is the reduction ratio of the image data of the background area in this case.

On the other hand, when the traffic volume is large, image data cannot be transmitted at the maximum transmission capacity of the network of 10 megabits / second. Therefore, for example, 7 megabits / second is assigned as the transmission capacity to transmit the image data. 15 frames / with this transmission capacity
When screen data is transmitted at a rate of seconds, the amount of data per frame is 467 kilobits (= 7 megabits / second / 15 frames / second). When the information amount of one pixel is 24 bits, the number of pixels per frame is 1
There are 9,444 pixels (= 467 kilobits / 24 bits). Therefore, the pixel number of 4,101 pixels obtained by subtracting the total pixel number of 15,343 pixels after reduction of the image data of the region of interest from 19,444 pixels becomes the pixel number of the image data of the background region after reduction. Therefore, 0.07 (= 4101/5
8629) is obtained as the reduction rate of the image data in the background area when the traffic amount is large.

(Background Area Reduction Ratio Calculation Method 2) Next, a second method for calculating the reduction ratio of the image data in the background area will be described. In this method, the reduction rate of the image data of the background area is set in advance, and the reduction rate of the image data of the region of interest designated by the user is also set in advance.

The image data of each area is reduced by the provisionally set reduction ratio, and then the reduction ratio of the entire image is calculated so as to further reduce the image data amount of the entire image. Then, with the reduction ratio of the entire image, the image data after being reduced at the reduction ratio temporarily set for each area is further reduced. In this case, the total number of pixels per frame of the reduced image is as follows. In the equation, Di ′ and Db ′ are provisionally set reduction ratios, and Nd ′ is a provisionally set reduction ratio and represents the total number of pixels of the entire image after reduction.

[0086]

Nd = D ₀ × (Σ (Ni ₀ × Di ′) + Nb × Db ′) = D ₀ × Nd ′ where Σ represents the sum of the regions of interest i (i = 1 to N).

Therefore, the reduction ratio D _{0 of the} entire screen is changed from the equation (5) to the equation (6).

[0088]

[Equation 6] D ₀ = Nd / Nd ′ Further, the reduction ratio Di of the region of interest and the reduction ratio Db of the background region.
Are as shown in equations (7) and (8), respectively.

[0089]

## EQU7 ## Di = D ₀ × Di '

[0090]

Db = D ₀ × Db ′ For example, if Db ′ = 1/4, the number of pixels of the background area image is 14,657 pixels (= 58,629 pixels × 1/4).
Becomes Further, the number of pixels in the region of interest is 15, as described above,
There are 343 pixels. Therefore, the total number of pixels of the entire screen after reduction at the provisionally set reduction ratio is 30,000 pixels (= 15,
343 pixels + 14,657 pixels).

A case will be described in which the image data thus reduced is transmitted at 10 megabits / second. The number of pixels allowed per frame of the image to be transmitted, that is, the total number of pixels of the reduced image is 27,
Since it has 777 pixels, the reduction ratio of the entire screen is 0.93.
(= 27,777 / 30,000). That is,
The reduction ratio of the image data in the background area is 0.23 (= 1/4 ×
0.93), and the reduction ratio of the image data of the region of interest 301 to 0.9.
93, the reduction ratio of the image data of the region of interest 302 is 0.47.
(= 0.93 / 2) and the reduction rate of the image data of the region of interest 303 is 0.31 (= 0.93 / 3), the image data can be transmitted at the desired frame rate.

On the other hand, when the amount of network traffic is large, that is, when image data is transmitted at a frame rate of 7 megabits / second, the number of pixels allowed per frame of the image to be transmitted is 19,444 pixels. is there. Therefore, the reduction ratio of the entire screen is 0.65 (=
19,444 / 30,000). Therefore, the reduction ratio of the image data in the background area is set to 0.16 (= 1/4 × 0.6).
5), the reduction ratio of the image data of the region of interest 301 is 0.6
5. Reduce the image data of the region of interest 302 to 0.33
If (= 0.65 / 2) and the reduction rate of the image data of the region of interest 303 is 0.22 (= 0.65 / 3), the image data can be transmitted at the desired frame rate.

In this way, after the background image processing unit 103 has calculated the reduction ratio of the image data of the background area,
Returning to step 200, the image capturing process is performed again, and the above process is repeated.

(Image Data Reduction Processing of Region of Interest) Here, the display processing method when the region of interest is designated will be described in more detail. As described above, in step 201 of FIG. 1, the image data of the region of interest corresponding to the coordinate data of the region of interest designated in step 225 of FIG. 2 is read into the data array. Then, in step 203, the area image cutout unit 104 reduces the image data of the corresponding region of interest using the reduction rate specified in step 225. Next, in step 204, the background image processing unit 103 reduces the image data of the background area at the reduction ratio calculated in step 211.

FIG. 8 shows how the image data in the background area is reduced and enlarged (restored). When the original image 800 is thinned out in accordance with the obtained reduction ratio, a reduced image 801 is formed. The image data of the reduced image 801 is the transmission side communication unit 1
It becomes the data flowing to the reception side communication unit 122 via 06. Enlarging the reduced image 801 by the reciprocal of the reduction ratio produces an enlarged image 802. This enlarged image 802 is an image of the background area displayed on the display 126.
02 has a lower spatial resolution than the original image 800 according to the reduction ratio. Similar reduction and enlargement processing is performed on the image of the region of interest.

The image data of the region of interest reduced by the transmitting side communication unit 106 and the image data of the reduced background region are transmitted to the receiving side communication unit 122 (step 20).
5).

Here, the format of the image data flowing from the transmission side communication section 106 to the reception side communication section 122 via the network 110 will be described with reference to FIG.
FIG. 9A shows a data format of the image 300 flowing through the network when the ROI is not designated by the user, and data 900 representing the size of the image data.
And image data 901 of the entire area. FIG. 9B is a data format of the image 300 flowing through the network when the user specifies the ROI, and data 91 indicating the size of the image data of the ROI 301.
0, coordinate data 911, reduction ratio 912, image data 91
3, data 914 indicating the size of image data of the region of interest 302, coordinate data 915, reduction ratio 916, image data 917, and data 918 indicating the size of image data of the region of interest 303, coordinate data 919, reduction ratio 92
0, image data 921, data 922 indicating the size of image data of the background area, reduction ratio 923, image data 9
It consists of 24. In the figure, the horizontal length of the data format corresponds to the data amount. The image data transmitted from the transmission side communication unit 106 in such a data format is received by the reception side communication unit 122 (step 220). Then, in the image synthesizing unit 124, the reduced image of the region of interest and the image of the background region are enlarged by the reciprocal of the reduction rate that has been processed, respectively, and the enlarged region of interest image and the background region image are extracted from the transmitted region of interest. Synthesis is performed using the coordinate data (step 222). As a result, the combined screen displayed on the display 126 is as shown in FIG. 10 (step 223). As is clear from the figure,
The images of the regions of interest 1001, 1002, 1003 are clear,
Displayed in detail. On the other hand, although the level of detail of the image in the background area 1004 is reduced, there is no particular problem because it is an image portion where the user is less interested.

As described above, according to this embodiment,
By controlling the amount of data for one frame according to the amount of network traffic, it is possible to perform moving image communication in which the transmission speed of an image is reduced and a delay in transmission is unlikely to occur.

Further, the spatial resolution of the entire screen is not lowered, but the spatial resolution of portions other than the portion desired by the user is reduced, so that the portion desired by the user can be seen clearly.

Although the present embodiment has been described with respect to the case where the spatial resolution of the image is changed by thinning out the pixels of the screen to reduce the size, the same applies to the case where the spatial resolution is changed by changing the number of colors of the pixels. realizable.

In the present embodiment, the case where the region of interest is a rectangle has been described, but the region of interest is not limited to a rectangle, and may be a polygon, a circle, a mask pattern for image recognition or the like. Such a shape may be used.

Further, in the present embodiment, when the background area is transmitted, the area in which the region of interest is cut out is transmitted, but the entire screen area in which the region of interest is not cut out may be transmitted.

Further, in this embodiment, the image reduced for each area is directly transmitted from the image transmitting unit 100 to the image receiving unit 120.
Was transmitted to the image transmission unit 100, the image encoding unit,
In addition, an image decoding unit is provided in the image receiving unit 120 to allow MPE
G, H. Image data may be compressed and then transmitted by a moving image compression algorithm such as H.261.

In the present embodiment, the case where the region of interest designating means is located in the image receiving section 120 has been described. However, the place where the designating means is placed is not limited to the image receiving section 120, and image transmission is performed. It is also possible to place it on the side 100.

Further, the system of this embodiment can be applied to a video telephone system, a video conference system, a plant monitoring system, a traffic control system and the like.

(Second Embodiment) A second embodiment of the present invention will be described using the system configuration shown in FIG. The difference between this embodiment and the first embodiment is that the amount of video information is efficiently reduced by changing the temporal resolution instead of the spatial resolution. In this embodiment, the time resolution of the screen is changed by controlling the frame rate, which is the number of screens per unit time.

As shown in FIG. 11, the system includes an image transmitting unit 1100, an image receiving unit 1120, and an image transmitting unit 11.
00 and the image receiving unit 1120 are divided into three parts of the network 110 for exchanging data.

In the image receiving section 1120, the user looks at the image sent via the network 110 and operates the mouse 125 to specify the region of interest and the frame rate of the region. That is, the user sets a high frame rate when the user is interested in the area and sets a low frame rate when the user is not very interested in the area.

Further, the image transmitting unit 1100 detects whether or not there is a temporal change in the background area. If there is a change, the background image is transmitted, and if there is no change, the background area image is not transmitted. Has become.

The ROI image is transmitted at the designated frame rate, synthesized by the image receiving unit 1120, and displayed on the display 126.

The image transmitter 1100 is used by the TV camera 10
1, moving image capturing unit 102, background image change detection unit 11
01, area image cutout unit 1102, image transmission interval calculation unit 1103, transmission side control unit 1104, transmission side communication unit 11
It is configured to include 05.

The moving image capturing unit 102 is the TV camera 10.
The moving image captured in 1 is loaded into the memory as a digital RGB signal. The area image cutout unit 1102 cuts out an image of the region of interest from the image data captured in the memory by the moving image capture unit 102. Background image change detection unit 11
01 detects the presence or absence of temporal change in the background area image.

The transmitting-side communication section 1105 uses the image receiving section 1 at the frame rate specified by the user for the image data of the region of interest.
The image receiving unit 1 transmits the background region image data to the image receiving unit 120 or transmits the background region image data when the background region image changes with time.
The coordinates of the region of interest and the frame rate transmitted from 120 are received, and the amount of network traffic is measured.

The image transmission interval calculation unit 1103 calculates and adjusts the image data transmission interval according to the amount of network traffic.

The transmission side control unit 1104 controls the entire image transmission unit. For example, control of image capture processing, detection of temporal changes in background area images, ROI image cutout processing, network communication processing, network traffic volume measurement processing, transmission interval calculation processing, and termination processing is performed.

The image receiving section 1120 has a receiving side control section 11
21, receiving side communication unit 1122, area determining unit 1123, image combining unit 1124, mouse 125, display 126
Composed of.

The receiving side communication section 1122 is the transmitting side communication section 11
05 receives the image data of the region of interest and all the image data, and transmits the coordinates of the region of interest and its frame rate to the transmission side communication unit 1105.

With the mouse 125, the user interactively specifies a region of interest and its frame rate, and the region determination unit 1
Reference numeral 123 stores and manages the coordinates in the screen of the area designated by the mouse 125 and its frame rate.

The image synthesizing section 1124 is the receiving side communication section 112.
2. The image of the region of interest received in 2 and the current display 126
Is displayed on the display 126.

The reception side control unit 1121 controls the entire image reception unit. For example, control such as network communication processing, region-of-interest designation processing, image composition processing, and termination processing is performed.

Next, the detailed configuration of this system will be described together with the operation. 12 and 13 show flowcharts of the processing procedure of this system.

Step 1200: Moving image capturing section 10
The image captured by the TV camera 101 by 2 is loaded into the memory as a digital RGB signal for one frame.

Step 1201: The image data of the region of interest is read into the array from the image data taken in the memory, and the region of interest image cutting processing is performed by the region image cutting unit 1.
At 102. However, if the user does not specify the region of interest, the entire region of the image is treated as the region of interest.

Step 1202: If the user has not designated the region of interest, the process proceeds to step 1204, and the image data taken in the memory is transmitted to the transmitting side communication unit 1.
Receiving side communication unit 1 from 105 via network 110
To 122. On the other hand, when the region of interest is designated by the user, in step 1203, the transmission unit 1100 causes the background image change detection unit 1101 to detect the temporal change of the background region image, and then the image in step 1205. The data is transmitted from the transmission side communication unit 1105.

Step 1220: Receiving side communication section 1122
The image data is received at.

Step 1221: If the user has not designated the region of interest, the process proceeds to step 1223 to display the image data received by the receiving side communication section 1122 on the display 126. If the region has been designated, the process proceeds to step 1222. After proceeding to the image composition processing in the image composition section 1124, the image data is displayed on the display 126 in step 1223.

Step 1224: The user's end instruction is received by the mouse 125. If there is an end instruction here, the process proceeds to step 1227 to perform the end process, and if there is no instruction, the process proceeds to step 1225 to perform the region of interest specifying process. That is, the user looks at the image displayed on the display 126 and specifies the region of interest and its frame rate using the mouse 125.

If the region of interest is not specified, step 12
If the network communication process is performed in step 20 and there is a designation, the process proceeds to step 1226, and the region-of-interest coordinates and frame rate designated in step 1225 are set to the receiving side communication unit 1.
Sent from 122.

Step 1206: Transmission side communication section 1105
If there is received data in step 1, the process proceeds to step 1207, and the coordinates and frame rate of the region of interest are set to the receiving side communication unit 1122.
Then, the process proceeds to step 1208, and the transmitting side communication unit 11
If there is no received data in 05, the process proceeds to step 1208.

Step 1208: If the region of interest is not designated, the process proceeds to step 1200 and an image capturing process is performed. On the other hand, if the region of interest is designated, the process proceeds to step 1209, and the transmission side communication unit 1105.
At, the network traffic volume measurement processing is performed.

After the measurement process of the traffic volume of the network is performed, the transmission interval calculation process is performed in step 1210. When the amount of network traffic is large, when the image data is sent to the network, a heavy load is applied to the network and a large delay occurs in the image data to be transmitted. Therefore, a waiting time is required until the traffic volume of the network decreases. Therefore, the transmission interval calculation unit 1103 performs a transmission interval calculation process that determines whether or not to provide a waiting time for transmitting image data depending on the amount of network traffic.

Step 1210: After executing the transmission interval calculation process, the process returns to step 1200 and the image capturing process is performed.

(Operation when Region of Interest is Designated) Step 1201: Image data of the region of interest is read into the array using the region of interest coordinates and frame rate designated in step 1225.

FIGS. 14A to 14D show the state of the region of interest image cutout processing. In the region-of-interest image cut-out processing, a region is cut out at each frame rate specified by the user for each region. On the screen 1300, the user designates regions of interest 1301, 1302, 1303. For example, the frame rate of the region of interest 1301 is 24 frames / sec, the frame rate of the region of interest 1302 is 12 frames / sec, and the frames of the region of interest 1303 are frames. It is assumed that the frame rate is set to 8 frames / sec for each area.
Assuming that the maximum transmission frame rate that can be transmitted is 24 frames / second, the ROI image 1301 is cut out every time as shown in (B) to (D) of FIG.
02 is cut out once every two times the region of interest image 1301 is cut out. The ROI image 1303 is cut out once every three times the ROI image 1303 is cut out.

(Detection Process of Background Image Change Detection Unit 1101) To detect the temporal change of the background area image, the difference between the current screen and the past screen is calculated, and it is determined whether or not the difference is below a threshold value. It is performed by determining. For each pixel in the screen, the difference between the RGB gradation values of the background area image of the past screen and the background area image of the current screen is calculated, and if the difference between the pixels in the frame is less than or equal to the threshold value, It is determined that no change has occurred in the background area of the image.

However, when the pixel difference between frames is simply taken, it is easily affected by noise due to the discretization error of the A / D conversion of the image signal in the moving image capturing unit 102, the blur of the TV camera 101, and the like. The change in the background area image cannot be detected accurately.

Therefore, by applying a smoothing filter to the image and taking the difference between the frames, it is possible to determine the temporal change of the background area which is less susceptible to noise.

FIG. 15 shows the smoothing filter 1405.
The pixel set to which the smoothing filter 1405 is applied is
Pixel 1401 whose RGB gradation value is A, similarly pixel 14 of B
02, C pixel 1403, and D pixel 1404.
When the smoothing filter 1405 is applied to this pixel set, the average value (A + B + C + D) / of the sum of the RGB gradation values of the pixels is obtained.
4 pixels 1406, 1407, 1408, 140
9 groups can be formed. By applying the smoothing filter 1405, the frequency in the frame is lowered, the contrast is weakened, and the influence of noise is less likely to occur when detecting the temporal change of the background area image.

However, considering the case where a moving image is drawn at 24 frames / second while detecting the temporal change of the background area image, the maximum time allowed per frame is about 0.04 seconds, and It is difficult to perform the smoothing filter 1405, which often involves arithmetic processing and division processing, within the above time.

Therefore, by thinning out pixels in the screen in advance, the processing time can be shortened. For example, when pixels are decimated so that the background area image is reduced to 1/4 and then a smoothing filter is applied, the number of pixels to be handled is smaller than that when a smoothing filter is applied without decimating pixels. Since it is 1/4, the processing speed is 4 times faster.

By thus thinning out pixels in the frame and then applying the smoothing filter, the temporal change of the background area can be detected at high speed.

After the background image change detection unit 1101 detects the time change of the background area image, step 12
At 04, the image data is transmitted from the transmission side communication unit 1105. Further, in step 1220, the receiving side communication unit 1122
To receive the image data.

Here, the format of image data transmitted from the transmission side communication section 1105 to the reception side communication section 1122 via the network 110 will be described.

FIG. 16A shows a screen 1 which flows through the network when the user does not specify the region of interest.
The data format is 300, and is composed of a size 1501 of image data and image data 1500 of the entire area.

FIGS. 16B to 16D show the data format of the screen 1300 flowing through the network when the user specifies the region of interest, and the background region image does not change with time.

That is, as shown in FIG. 16B, the size 1511 and the coordinate 1 of the image data of the region of interest 1301.
512 and image data 1510, size 1521 of image data of the region of interest 1302, coordinates 1522, and image data 1
520, the size 153 of the image data of the region of interest 1303
1 consisting of 1 and coordinates 1532 and image data 1530,
As shown in FIG. 16C, the size 1511, the coordinates 1512, and the image data 15 of the image data of the region of interest 1301.
10, size 1521 of image data of region of interest 1302
16D, the coordinates 1522 and the image data 1520. As shown in FIG. 16D, the size 1511 of the image data of the region of interest 1301 and the coordinates 1512 and the image data 151.
0, the size 1531 of the image data of the region of interest 1303, the coordinates 1532, and the image data 1530.

FIG. 16 (F) shows the data format of the screen 1300 flowing through the network when the user specifies the region of interest, and the background region image has a temporal change. Size 1
501 and image data 1501 of the entire area.

At step 1222, the receiving side communication section 11
Image composition processing is performed using the image data received in step 22. The image combining process combines the region-of-interest image data included in the transmitted image data with the image data currently displayed on the display 126. Then, in step 1223, the combined image is displayed on the display 126.

At step 1224, an end instruction from the user is accepted, and if there is an end instruction, step 122
In step 7, the termination instruction signal is transmitted from the reception side communication unit 1122 to the transmission side communication unit 1105, and in step 1228, the termination process of the image reception unit 1120 is performed. Furthermore, in step 1205, the transmission side communication unit 1105 receives the termination instruction signal transmitted from the image reception unit 1120, and in step 1211, the termination processing of the image transmission unit 1100 is performed.

As described above, according to this embodiment,
Since only the image of the portion that the user wants to see is transmitted and the images of the other portions are transmitted only when there is a change, the communication data amount can be greatly reduced.

In the present embodiment, the case where the time resolution of the image is changed by changing the frame rate has been described, but the same applies when the time resolution is changed by changing the thinning interval of the fields in the interlace.

Further, in the present embodiment, when the background area is transmitted, the area in which the region of interest is cut out is transmitted, but the entire screen area in which the region of interest is not cut out may be transmitted. Further, in this embodiment, the area image is transmitted from the image transmitting unit to the image receiving unit as it is, but the image transmitting unit is provided with the image encoding unit, and the image receiving unit is provided with the image decoding unit.
G, H. The region image data may be compressed and then transmitted by a moving image compression algorithm such as H.261.

(Third Embodiment) FIG. 17 shows the system configuration of the third embodiment of the present invention.

In this embodiment, when transmitting a moving image using the moving image compression algorithm, the user interactively specifies the region of interest to the user and the other region, and the amplitude resolution ( By changing the step size,
This is an example of increasing the image compression rate and efficiently transmitting the image.
In this example, the image is compressed and transmitted by the known MPEG method.

Now, the MPEG method will be described. M
The compression process of the PEG method is as follows. DCT transform (discrete cosine transform) of the image signal. (DCT conversion) 2. The value after the DCT conversion (DCT coefficient) is quantized. (Quantization) 3. Encode the quantized value. (Encoding) consists of three phases. When the image is compressed by the MPEG method, the DCT conversion, the quantization, and the encoding process are sequentially performed on the image signal for each block including 8 × 8 pixels.

As an example, FIG. 18 shows a process of DCT-transforming image data 2200 consisting of 176 × 144 pixels, quantizing it, and encoding it.

(DCT Transformation) In general, signals having strong signal correlation are concentrated in the low frequency region in the frequency domain, and signals having weak correlation are concentrated in the high frequency region. The image signal generally contains a lot of low frequency components. The DCT transform is a kind of orthogonal transform and has a function of concentrating low frequency components at a specific DCT coefficient. Therefore, the 8 × 8 matrix image signal 2201 is converted into the DCT transform.
Upon conversion, the signal concentrates on certain DCT coefficients.
The DCT coefficient 2302 composed of this 8 × 8 matrix is
The upper left element has a lower frequency component, and the lower right element has a higher frequency component.
The CT coefficient has a larger value, and the lower right DCT coefficient has a smaller value.

(Quantization) Quantization is performed on the transform value (DCT coefficient) after DCT transforming the image signal. DC
Each element of the 8 × 8 matrix 2202 after T conversion (DCT
Coefficient), the frequency component where the signal is concentrated is low D
The CT coefficient is finely quantized, and the DCT coefficient having a high frequency component where signals are not concentrated is roughly quantized. Further, since the human eye is less sensitive to the high frequency component signal than the low frequency component signal, an image that does not feel unnatural to the human eye even if the quantization of the DCT coefficient having a high frequency component is roughened. Can be provided. Thus, a DCT composed of an 8 × 8 matrix
To quantize the coefficients with different quantization step sizes, the DC
A so-called quantization matrix 2203 in which a different quantization step size is described for each element of the T coefficient is prepared.
The DCT coefficient is quantized using this quantization matrix. In practice, the quantization step size when performing quantization is 1 for each element of the quantization matrix 2203, which is called MQUANT.
Multiplying the constants from 31 to 31, MQUANT
Can be specified by the user. When this MQUANT is set to a large value, the quantization step size becomes large and the image resolution drops significantly.

(Encoding) The 8 × 8 matrix 2204 after quantization contains many 0s (runs), and the run becomes larger especially in the lower right element of coarse quantization. Therefore, the quantized matrix is scanned in a zigzag direction from the upper left to the lower right, and the 64 data sets 2205 scanned in the zigzag are encoded as one block.

[0160] In this data set, there are many parts in which runs are continuously continuous, and for that part, the run is not described as it is, but the data set 22
The data is compressed by describing the run length such as 06. This compression technique is called run length encoding, and image information can be significantly compressed by the run length encoding.

As shown in FIG. 17, the system configuration of this embodiment is such that the system includes an image transmitting section 1600, an image receiving section 1620, an image transmitting section 1600 and an image receiving section 1620.
The network 110 is divided into three parts for exchanging data between them.

In the image receiving unit 1620, the user looks at the image sent via the network 110 and operates the mouse 125 to specify the region of interest and the parameter (MQUANT) for determining the resolution of the amplitude of the region.

The smaller the value of this parameter (MQUANT), the finer the amplitude resolution. Therefore, MQUANT is set small when the area is of interest, and MQUANT is set large when the area is not very interested.
A large value is set in advance for MQUANT in the background area. The image is compressed in the image transmitting unit 1600 using MQUANT set for each area. And
The compressed image data flows through the network 110, is decompressed in the image receiving unit 1620, and is displayed on the display 12.
6 is displayed.

The image transmission unit 1600 is used by the TV camera 10
1, an image compression unit 1601, a moving image capturing unit 102, a transmission side control unit 1602, and a transmission side communication unit 1603.

The image compressing unit 1601 compresses the image data captured in the memory by the moving image capturing unit 102 using MQUANT designated for each region of interest by the user. The transmission side communication unit 1603 transmits the compressed image data to the image receiving unit 1620, and receives the coordinates of the region of interest and MQUANT transmitted from the image receiving unit 1620. The transmission side control unit 1602 controls the entire image transmission unit. For example, control of image capture processing, image compression processing, network communication processing, and termination processing is performed.

The image receiving section 1620 is the receiving side control section 16
21, reception side communication unit 1622, area determination unit 1623, image expansion unit 1624, mouse 125, display 126.
Consists of.

The receiving side communication section 1622 receives the compressed image data transmitted from the transmitting side communication section 1603, and sets the coordinates of the region of interest and MQUANT in the transmitting side communication section 1622.
Send to 03. The user interactively specifies the region of interest and MQUANT with the mouse 125, and the region determination unit 1
Reference numeral 623 stores and manages the coordinates and MQUANT in the screen of the area designated by the mouse 125.

The image decompression unit 1624 is the receiving side communication unit 16
The compressed image data received at 22 is decompressed and the image is displayed on the display 126.

The receiving-side control section 1621 controls the entire image receiving section, and performs control such as network communication processing, region-of-interest designation processing, image expansion processing, and termination processing.

Next, the detailed configuration of this embodiment will be described together with the operation. 19 and 20 show flowcharts of the processing procedure of this embodiment.

In step 1700, the moving image capturing unit 102 captures the image captured by the TV camera 101 into the memory as a digital RGB signal for one frame.

At step 1701, the image compression section 160.
In No. 1, the image data taken in the memory is compressed with MQUANT designated by the user for each area. However, when the user does not specify the region of interest, MQUANT of all regions is set to a small value (for example, 1).

At step 1702, the compressed image data is transmitted from the transmission side communication section 1603 to the reception side communication section 1622 via the network 110.

At step 1710, the receiving side communication section 162
The image data compressed in 2 is received.

At step 1711, the image expansion section 162
In 4, the image data received by the receiving side communication unit 1622 is decompressed by using the designated MQUANT. Then, in step 1712, the image expanded is displayed on the display 126.

At step 1713, the mouse 125 receives the end instruction from the user. If there is an end instruction here, an end instruction signal is transmitted from the receiving side communication unit 1622 to the transmitting side communication unit 1603 in step 1716, and the ending process of the image receiving unit 1620 is performed in step 1717.

Further, in step 1703, the end instruction signal transmitted from the image receiving unit 1620 is transmitted to the transmitting side communication unit 1.
The image is received in step 603, and in step 1706 the termination process of the image transmission unit 1600 is performed.

If there is no end instruction in step 1713, the flow advances to step 1714 to perform region-of-interest designation processing. That is, the user looks at the image displayed on the display 126 and uses the mouse 125 to specify the region of interest and its MQUANT. The area of interest and MQUAN
The information of T is stored and managed in the area determination unit 1623.

If the region of interest is not designated in step 1714, the process proceeds to step 1710 and the network communication process is performed. On the other hand, if there is a region of interest specified,
In step 1715, the coordinates of the region of interest designated by the user and MQUANT are transmitted from the reception side communication unit 1622.

At step 1704, the transmitting side communication section 16
If there is no received data in 03, the process returns to step 1700. If there is received data in the transmitting side communication unit 1603, the process proceeds to step 1705 and the coordinates of the region of interest and MQUANT.
Is received by the transmission side communication unit 1622, and then step 17
Return to 00.

As described above, according to this embodiment,
Since the user specifies the amplitude resolution (step size) for each region according to the degree of interest, it is possible to increase the compression rate without lowering the resolution of the portion of interest to the user.

In the present embodiment, the case where the entire image is compressed at once by the moving image compression algorithm has been described, but it is also possible to perform compression separately for each area for communication.

Further, in this embodiment, the MQU in the MPEG is
The method of controlling the image by changing the ANT has been described, but simply controlling the step size (quantization step size) of the RGB values of each pixel to make the area of interest fine, It is also possible to efficiently compress the information amount of the image by quantizing the area without a mark with a rough step size.

(Fourth Embodiment) FIG. 21 shows the entire system configuration of a fourth embodiment of the present invention. This embodiment is an example of correcting the size and position of the region of interest in cooperation with the movement of the camera in the image communication system for transmitting moving images with different resolutions for each region described in each of the above embodiments.

FIG. 21 shows the moving image communication system of the first embodiment to which the characteristic parts of this embodiment are applied. That is, the camera control unit 1801 is added to the image transmission unit 100 of the first embodiment.

In the camera control unit 1801, the TV camera 1
01 zoom control and region of interest coordinates correction.

When the TV camera 101 zooms, the camera control unit 1801 reads the current region-of-interest coordinates from the region image clipping unit 104, corrects the coordinates using the zoom ratio of the camera, and the corrected coordinates. Is sent to the area image cutout unit 104 and the background image processing unit 103.

Here, a method of correcting the coordinates of the region of interest when the region of interest changes due to the zoom of the TV camera 101 will be described. FIG. 22 shows changes in the size and position of the region of interest when the screen is enlarged by the zoom of the TV camera 101. Region of interest 190 on screen 1900
19 is a screen in which the screen 1900 is enlarged by the zoom ratio α, and the region of interest after the enlargement is 1911.

Assuming that the coordinate origin is the focal position (screen 1900
Center) of the region of interest 1901 and the lower left coordinate is (s, t) and the upper right coordinate is (u, v). Further, the lower left coordinates of the region of interest 1911 after the zoom ratio change are (αs, αt), and the upper right coordinates are (αu, αv).

In this system, the coordinate origin is at the lower left of the screen, so the coordinate system must be moved so that the coordinate origin is at the lower left of the screen 1900 from the focus position. In FIG. 23, the coordinate origin of the screen 1900 is moved to the lower left of the screen. The focus position in this coordinate system is (x, y). The coordinates of the region of interest 1901 are (s + x, t + y)-
(U + x, v + y), and the region of interest 191 after zooming
The coordinates of 1 are (αs + x, αt + y)-(αu + x, αv +
y).

As described above, the position and size of the region of interest after zooming is calculated using the zoom ratio of the TV camera 101 by the camera control unit 1801, and the corrected region of interest coordinates are extracted by the region image cutting unit 104 and the background image processing unit. Send to 103.

According to the present embodiment, even if the area designated by the user changes due to the zoom of the camera, the means for automatically correcting the position and size of the area is provided. Need not be specified.

In this embodiment, the case where the area correction is performed by using the zoom ratio of the camera has been described. However, the area correction is not limited to the zoom ratio but is performed by parameters such as the photographing direction of the camera and the focal length of the camera. It is also possible to do so.

(Fifth Embodiment) FIG. 24 shows the entire system configuration of a fifth embodiment of the present invention. In the present embodiment, in a moving image communication system that transmits a moving image with different resolution for each region, when the position of the region of interest changes due to the movement of the target of interest, the amount of movement of the target of interest is obtained to correct the position of the region of interest Is an example of performing.

This embodiment is similar to the image transmitter 10 of the first embodiment.
The image recognition unit 2001 is added to 0.

The image recognition unit 2001 recognizes an object of interest in the area of interest, measures the amount of movement of the object of interest, and corrects the coordinates of the area of interest from the amount of movement.

When the object of interest moves, the image recognition unit 20
In 01, the current region-of-interest coordinates are converted into the region-coordinate cutting unit
04, the target of interest is recognized, the amount of movement of the target of interest is measured, the coordinates of the region of interest are corrected from the amount of movement, and the corrected region of interest coordinates are extracted by the region image cutting unit 104 and the background image processing unit. Send to 103.

Here, a method of correcting the coordinates of the region of interest when the object of interest moves will be described. FIG. 25 shows a car 21
It is shown that the position of the license plate of interest is changing as 01 moves from left to right on the screen. The car 2101 moves from the left side to the right side of the screen over time, and the license plate of interest also moves. The image recognition unit 2101 sequentially scans an object of interest whose position changes from moment to moment and constantly measures the position on the screen.

Then, the region of interest coordinates are corrected based on the amount of movement of the subject of interest. The region of interest coordinates before the car 2201 moves are A1 (10,20) -B1 (70,7).
0). After specifying the region of interest, the user performs a matching process in the region of interest 210 to measure the difference between the position of the target of interest before moving and the position of the target of interest after moving.
3 (hereinafter referred to as a template) is designated. And
In the image recognition unit 2001, when the region of interest moves,
The part 2 in which the template 2103 designated by the user is scanned over the entire screen and the similarity with the template is maximum 2
Search for 105. The part 210 with the highest degree of similarity
5 is a template 2105 after the movement, which is matched with the template 2103 before the movement. The position (260, 80) of the template 2105 after the movement (the coordinates of (C2)) and the position of the template 2103 before the movement (30 , 4
0) (the coordinates of (C1)) is the amount of movement of the region of interest. Then, the region of interest can be corrected by shifting the region of interest coordinates by this amount of movement. The corrected region of interest coordinates are A2 (270,100) -B2 (330,150).

As described above, according to this embodiment,
Even if an object of interest to the user moves, the user has a means to automatically correct the position of the region of interest by scanning the object of interest and measuring the position on the screen, so You do not need to specify the area.

The system of the first to fifth embodiments described above can be applied to a video telephone system, a video conference system, a plant monitoring system, a traffic control system and the like.

[0202]

The present invention has the following effects.

A user can interactively specify a region of interest,
Since it is possible to transmit the image in which the region other than the region of interest is coarse in detail, it is possible to reduce the amount of data transmission.

When there are a plurality of regions of interest, the user can specify the resolution according to the degree of interest in the regions of interest, which has the effect of improving the efficiency of reducing the amount of transmission of video data.

According to the traffic volume of the network,
Since the amount of video data transmission can be controlled, a decrease in frame rate, delay in transmission, and the like are less likely to occur.

Even if the region of interest changes due to the movement of the camera or the movement of the image, it has a means for automatically correcting the position and size of the region, so that the user does not need to specify the region again. .

[Brief description of drawings]

FIG. 1 is a block configuration diagram of a moving image communication system according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing a processing procedure of an image transmission unit of the first embodiment.

FIG. 3 is a flowchart showing a processing procedure of an image receiving unit of the first embodiment.

FIG. 4 is a diagram illustrating a method of designating a region of interest according to the present invention.

5A and 5B are diagrams showing a state of designating a region of interest, respectively.

FIG. 6 is a diagram illustrating an example of measuring a traffic volume of a network, and is a diagram illustrating a state of a communication buffer when the traffic volume is small.

FIG. 7 is a diagram illustrating an example of measuring a traffic amount of a network, and is a diagram illustrating a state of a communication buffer when the traffic amount is large.

FIG. 8 is a diagram illustrating reduction and enlargement of image data according to the first embodiment.

FIG. 9 is a diagram of a transmission format of image data,
(A) shows the case where the ROI is not specified, and (B) shows the case where the ROI is specified.

FIG. 10 is an example diagram of an image displayed on a display according to the first embodiment.

FIG. 11 is a block configuration diagram of a moving image communication system according to a second embodiment of the present invention.

FIG. 12 is a flowchart showing a processing procedure of an image transmitting unit of the second embodiment.

FIG. 13 is a flowchart showing a processing procedure of an image receiving unit of the second embodiment.

14A and 14B are diagrams illustrating a temporal change of a region of interest image cut out from the screen according to the second embodiment, in which FIG. 14A shows a setting state of the region of interest, and FIGS. It is a figure explaining that the cut-out frequency of an image changes according to the difference of the frame rate set for every.

FIG. 15 is a diagram illustrating a smoothing filter according to a second example.

FIG. 16 is a diagram of a transmission format of image data according to the second embodiment, in which (A) shows a case where the ROI is not specified, and (B) to (D) shows a case where the ROI is specified. If there is no temporal change in the background area image, (E)
Indicates the case where the region of interest is designated and the background region image does not change with time.

FIG. 17 is a block configuration diagram of a moving image communication system according to a third embodiment of the present invention.

FIG. 18 is a DCT based on the MPEG method according to the third embodiment.
It is a figure explaining the process of conversion, quantization, and coding.

FIG. 19 is a flowchart showing a processing procedure of an image transmitting unit of the third embodiment.

FIG. 20 is a flowchart showing a processing procedure of an image receiving unit of the second embodiment.

FIG. 21 is a block configuration diagram of a moving image communication system according to a fourth embodiment of the present invention.

FIG. 22 is a coordinate system on the screen when the focal position of the TV camera of the fourth embodiment is the origin.

FIG. 23 is a coordinate system in the screen when the lower left of the screen is the origin in the fourth embodiment.

FIG. 24 is a block diagram of a moving image communication system according to a fifth embodiment of the present invention.

FIG. 25 is a diagram illustrating correction of a region of interest according to movement of a target of interest in the fifth example.

[Explanation of symbols]

 100 image transmission unit 101 TV camera 102 moving image capture unit 103 background image processing unit 104 area image cutout unit 105 transmission side control unit 106 transmission side communication unit 110 network 120 image reception unit 121 reception side control unit 122 reception side communication unit 123 area Deciding section 124 Image synthesizing section 125 Input section (mouse) 126 Image display section (display) 601, 611, 621 Communication buffer 1100 Image transmitting section 1101 Background image change detecting section 1102 Area image cutting section 1103 Transmission interval calculating section 1104 Transmission side Control unit 1105 Transmission side communication unit 1120 Image reception unit 1121 Reception side control unit 1122 Reception side communication unit 1123 Area determination unit 1124 Image combination unit 1124 Image combination unit 1300 Received image screen 1405 Smoothing filter 1600 image Transmission unit 1601 Image compression unit 1602 Transmission side control unit 1603 Transmission side communication unit 1620 Image reception unit 1621 Reception side control unit 1622 Area determination unit 1622 Image decompression unit 1800 Image transmission unit 1801 Camera control unit

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Koichiro Tanigoshi Koichiro Tanikoshi 7-1-1 Omika-cho, Hitachi City, Ibaraki Hitachi Ltd. Hitachi Research Laboratory (72) Inventor Harumi Uchigasaki 7-chome, Omika-cho, Hitachi City, Ibaraki Prefecture No. 1-1 Hitachi Ltd., Hitachi Research Laboratory (72) Inventor Masayasu Futagawa 7, 1-1 Omika-cho, Hitachi City, Hitachi City, Ibaraki Prefecture Hitachi Ltd. Hitachi Research Laboratory (72) Inventor Masato Hotta Hitachi City, Ibaraki Prefecture 7-1-1 Omika-cho, Hitachi, Ltd. Hitachi Research Laboratory

Claims (18)

[Claims] 1. A transmission system for transmitting image data of a moving image to a communication network, and a receiving system for receiving the image data via the communication network and displaying the moving image on a display, the transmission Either one of the system and the receiving system is provided with an area designating unit for designating an arbitrary area in the moving image, and the transmitting system is not designated as the designated area designated by the area designating unit. A moving image communication system comprising image processing means for reducing the image data amount for each designated area. 2. The resolution setting means according to claim 1, wherein resolution setting means for setting at least one of the resolution of the designated area and the resolution of the non-designated area is provided, and the image processing means is set by the resolution setting means. A moving image communication system, wherein the image data amount of the designated area and the non-designated area is reduced based on the resolution. 3. The resolution according to claim 2, wherein the resolution is at least one of spatial resolution and temporal resolution.
A moving image communication system characterized in that 4. The spatial resolution according to claim 3, wherein the spatial resolution is the number of pixels per unit area, the number of colors of a color image, the resolution of the amplitude of a frequency signal obtained by frequency-converting image data, and the pixel data. A moving image communication system, which is at least one of amplitude resolution and a quantization step size when quantizing pixel data. 5. The moving image communication system according to claim 3, wherein the temporal resolution is at least one of a frame rate and the number of interlaces. 6. A transmission system for transmitting image data of a moving image to a communication network, and a reception system for receiving the image data via the communication network and displaying the moving image on a display. The system has area designating means for designating an arbitrary area in the moving image, and resolution setting means for setting the resolution of the designated area designated by the area designating means. The image data amount of the designated area is reduced based on the resolution set by the setting means, and the resolution of the non-designated area not designated by the area designating means is calculated so as to satisfy the allowable transmission capacity of the communication network. A moving image communication system comprising image processing means for reducing the image data amount of the non-designated area based on the resolution. 7. The moving image communication system according to claim 6, wherein the area designating unit is formed so that a user can interactively set the designated area. 8. The image processing means according to claim 6, further comprising means for measuring the current traffic volume of the communication network, and the resolution of the non-designated area according to the current traffic volume measured by the means. A moving image communication system characterized by calculating. 9. The method according to claim 6, wherein the resolution is at least one of spatial resolution and temporal resolution.
A moving image communication system characterized in that 10. The spatial resolution according to claim 9, wherein the spatial resolution is the number of pixels per unit area, the number of colors of a color image, the resolution of the amplitude of a frequency signal obtained by frequency-converting image data, and the pixel data. A moving image communication system, which is at least one of amplitude resolution and a quantization step size when quantizing pixel data. 11. The moving image communication system according to claim 10, wherein the temporal resolution is at least one of a frame rate and the number of interlaces. 12. A transmission system for transmitting image data of a moving image captured by a TV camera to a communication network, and a receiving system for receiving the image data via the communication network and displaying the moving image on a display. An area designating means for designating an arbitrary area in the moving image, and a resolution for setting the resolution of the designated area designated by the area designating means in either the transmitting system or the receiving system. Setting means is provided, the transmission system reduces the image data amount of the designated area based on the resolution set by the resolution setting means, and sets the resolution of a non-designated area not designated by the area designating means to the communication network. The image data of the non-designated area is calculated based on the calculated resolution. Image processing means for reducing the amount, designated area detecting means for detecting movement of the designated area on the screen, and correcting the designated area according to the movement of the designated area detected by the designated area detecting means. A moving image communication system comprising a designated area correction means. 13. The specified area detection means according to claim 12, wherein the specified area detection means is a camera detection means for detecting movement of the TV camera, and the specified area correction means is
A moving image communication system characterized in that the position and size of the designated area are corrected according to the movement of the camera detected by the camera detection means. 14. The specified area detection means according to claim 12, wherein the specified area detection means detects a movement of a specified image in the specified area, and the specified area correction means is detected by the detection means. A moving image communication system, wherein the position and size of the designated area are corrected according to the movement of the designated image. 15. A resolution setting means for setting the resolution of a designated area of a moving image, and an image of the designated area for reducing the image data amount of the designated area based on the resolution set by the resolution setting means. An image of the non-designated area for reducing the image data amount of the non-designated area based on the calculated resolution by calculating the resolution of the non-designated area not specified by the data reducing means so as to satisfy the allowable transmission capacity of the communication network. A moving image transmission system comprising: a data reducing means; and a communication means for transmitting the image data reduced by the image data reducing means of the designated area and the non-designated area to a communication network. 16. The moving image transmission system according to claim 15, further comprising area designating means for designating the designated area by a dialogue with a user. 17. A communication unit for receiving image data of a moving image transmitted from a moving image transmission system via a communication network, and a display for displaying the moving image based on the image data received by the communication unit. And an area designating means for designating an arbitrary area of the moving image displayed on this display and a resolution of the area, and the coordinate data and the resolution of the designated area designated by the area designating means are transmitted by the communication. A moving image receiving system comprising transmitting to the moving image transmitting system via a means. 18. The moving image receiving system according to claim 17, wherein the area designating unit designates the designated area and the resolution thereof through a dialog with a user.