JP7334470B2 - VIDEO PROCESSING DEVICE, VIDEO CONFERENCE SYSTEM, VIDEO PROCESSING METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a video processing device, a video conference system, a video processing method, and a program.

Japanese Patent Laid-Open No. 2002-200003 discloses that, in an image captured by a surveillance camera, the image in the area where the movement or face is not detected is reduced in quality, and the image in the area where the movement or face is detected is converted to an image with the movement or the face. Techniques have been disclosed for improving the image quality of an image of an area that has not been detected. According to this technique, it is possible to reduce the coded data size of a captured image, reduce the load on a transmission line in a network, and improve the visibility of an image in a motion area.

However, in the conventional technology, when an image is divided into a low image quality area and a high image quality area, the difference in image quality at the boundary between the low image quality area and the high image quality area becomes conspicuous, giving the viewer a sense of discomfort. There are issues such as

SUMMARY OF THE INVENTION In order to solve the above-described problems of the prior art, the present invention reduces the amount of video data and makes it possible to make the difference in image quality at the boundary between the low image quality area and the high image quality area inconspicuous. With the goal.

In order to solve the above-described problems, the image processing apparatus of the present invention includes an image acquisition unit that acquires an image, an image analysis unit that analyzes high-frequency components for each region in the image acquired by the image acquisition unit, an image quality adjustment unit that performs image quality adjustment such that at least a part of the area in the image has a higher image quality as the area having more high-frequency components, according to the analysis result of the image analysis unit.

According to the present invention, it is possible to reduce the amount of video data and make the difference in image quality at the boundary between the low image quality area and the high image quality area less noticeable.

1 is a diagram showing the system configuration of a video conference system according to an embodiment of the present invention; FIG. 1 is a diagram showing the appearance of an IWB according to an embodiment of the present invention; FIG. 1 is a diagram showing the hardware configuration of an IWB according to an embodiment of the present invention; FIG. 1 is a diagram showing the functional configuration of an IWB according to an embodiment of the present invention; FIG. 4 is a flowchart showing the procedure of videoconference execution control processing by IWB according to one embodiment of the present invention; 4 is a flow chart showing the procedure of video processing by the video processing unit according to one embodiment of the present invention; FIG. 4 is a diagram showing a specific example of video processing by the video processing unit according to one embodiment of the present invention; FIG. 4 is a diagram showing a specific example of video processing by the video processing unit according to one embodiment of the present invention;

[One embodiment]
An embodiment of the present invention will be described below with reference to the drawings.

(System configuration of video conference system 10)
FIG. 1 is a diagram showing the system configuration of a video conference system 10 according to one embodiment of the invention. As shown in FIG. 1, the videoconferencing system 10 includes a conference server 12, a conference reservation server 14, and a plurality of IWBs (Interactive White Boards) 100. These plurality of devices are connected to the Internet, an intranet, a LAN (Local Area Network) or the like. The videoconferencing system 10 can hold a so-called videoconference between a plurality of bases using these devices.

The conference server 12 is an example of a "server device". The conference server 12 performs various controls related to video conferences by a plurality of IWBs 100 . For example, the conference server 12 monitors the communication connection state between each IWB 100 and the conference server 12, calls each IWB 100, and the like, at the start of the video conference. Also, the conference server 12 performs transfer processing of various data (for example, video data, audio data, drawing data, etc.) between the plurality of IWBs 100 during the video conference.

The conference reservation server 14 manages the reservation status of video conferences. Specifically, the conference reservation server 14 manages conference information input via the network 16 from an external information processing device (for example, a PC (Personal Computer), etc.). The meeting information includes, for example, date and time of meeting, place of meeting, participants, roles, terminals used, and the like. The video conference system 10 conducts video conferences based on conference information managed by the conference reservation server 14 .

The IWB 100 is an example of a "video processing device," an "imaging device," and a "communication terminal." The IWB 100 is a communication terminal installed at each site where the videoconference is held and used by the participants of the videoconference. For example, the IWB 100 can transmit various data (eg, video data, audio data, drawing data, etc.) input in the videoconference to other IWBs 100 via the network 16 and conference server 12 . Also, for example, the IWB 100 presents various data transmitted from other IWBs 100 to the participants of the video conference by outputting them by an output method (for example, display, audio output, etc.) according to the type of data. be able to.

(Configuration of IWB 100)
FIG. 2 is a diagram showing the appearance of IWB 100 according to one embodiment of the present invention. As shown in FIG. 2, the IWB 100 has a camera 101, a touch panel display 102, a microphone 103, and a speaker 104 on the front surface of the main body 100A.

A camera 101 captures an image in front of the IWB 100 . The camera 101 includes, for example, a lens, an image sensor, and an image processing circuit such as a DSP (Digital Signal Processor). An image sensor generates video data (RAW data) by photoelectrically converting light condensed by a lens. As the image sensor, for example, CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor), etc. are used. The video processing circuit performs Bayer conversion, 3A control (AE (automatic exposure control), AF (autofocus), and AWB (automatic white balance)) on video data (RAW data) generated by the image sensor. Image data (YUV data) is generated by performing general image processing. The image processing circuit then outputs the generated image data (YUV data). YUV data represents color information by a combination of a luminance signal (Y), a difference (U) between the luminance signal and the blue component, and a difference (V) between the luminance signal and the red component.

The touch panel display 102 is a device that includes a display and a touch panel. The touch panel display 102 can display various information (for example, video data, drawing data, etc.) on the display. In addition, the touch panel display 102 can input various types of information (eg, characters, graphics, images, etc.) by a touch operation of the operation body 18 (eg, finger, pen, etc.) through the touch panel. As the display, for example, a liquid crystal display, an organic EL display, electronic paper, or the like can be used. As the touch panel, for example, a capacitive touch panel can be used.

The microphone 103 collects sound around the IWB 100, generates sound data (analog data) corresponding to the sound, and converts the sound data (analog data) from analog to digital. It outputs voice data (digital data) corresponding to voice.

The speaker 104 outputs sound corresponding to the audio data by being driven based on the audio data (analog data). For example, the speaker 104 outputs sound collected by the IWB 100 at the other site by being driven based on audio data transmitted from the IWB 100 at the other site.

The IWB 100 configured in this way reduces the amount of data by performing video processing and encoding processing, which will be described later, on the video data acquired from the camera 101, and then displays the video data together with the touch panel display. Various display data (for example, video data, drawing data, etc.) acquired from 102 and audio data acquired from microphone 103 are transmitted to other IWBs 100 via conference server 12, thereby enabling the transmission of these data. It can be shared with other IWBs 100 . In addition, the IWB 100 displays, on the touch panel display 102, display contents based on various display data (for example, video data, drawing data, etc.) transmitted from the other IWB 100, and also displays audio data transmitted from the other IWB 100. This information can be shared with other IWBs 100 by outputting the based voice through the speaker 104 .

For example, in the example shown in FIG. 2, the touch panel display 102 displays a display layout having a plurality of display areas 102A and 102B. The display area 102A is a drawing area in which drawing data drawn by the operating tool 18 is displayed. In the display area 102B, an image of the own site captured by the camera 101 is displayed. Note that the touch panel display 102 can also display drawing data drawn by another IWB 100, video of another site captured by another IWB 100, and the like.

(Hardware configuration of IWB 100)
FIG. 3 is a diagram showing the hardware configuration of the IWB 100 according to one embodiment of the invention. As shown in FIG. 3, the IWB 100 includes the camera 101, the touch panel display 102, the microphone 103, and the speaker 104 described in FIG. , a memory 107 , a communication I/F (Interface) 108 , an operation unit 109 , and a recording device 110 .

The system controller 105 performs various controls of the IWB 100 by executing various programs stored in the auxiliary storage device 106 or memory 107 . For example, the system control 105 includes a CPU, an interface with peripheral units, a data access arbitration function, and the like, and controls various hardware provided in the IWB 100 and executes various functions (see FIG. 4) related to the video conference provided in the IWB 100. .

For example, the system control 105, as a basic function related to video conferencing, transmits video data acquired from the camera 101, drawing data acquired from the touch panel display 102, and audio data acquired from the microphone 103 through the communication I/F 108. to another IWB 100 via the

Also, for example, the system controller 105 can control the image based on the image data acquired from the camera 101 and the rendering content based on the drawing data acquired from the touch panel display 102 (that is, the image data and drawing data of its own site). , is displayed on the touch panel display 102 .

Also, for example, the system controller 105 acquires video data, drawing data, and audio data transmitted from the IWB 100 at another site via the communication I/F 108 . The system controller 105 causes the touch panel display 102 to display the video based on the video data and the drawing content based on the drawing data, and outputs the audio based on the audio data from the speaker 104 .

The auxiliary storage device 106 stores various programs executed by the system controller 105, data necessary for the system controller 105 to execute various programs, and the like. As the auxiliary storage device 106, for example, a non-volatile storage device such as a flash memory or HDD (Hard Disk Drive) is used.

The memory 107 functions as a temporary storage area used when the system control 105 executes various programs. As the memory 107, for example, a volatile storage device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory) is used.

The communication I/F 108 is an interface for connecting to the network 16 and transmitting/receiving various data to/from another IWB 100 via the network 16 . As the communication I/F 108, for example, a wired LAN interface compatible with 10Base-T, 100Base-TX, 1000Base-T, etc., a wireless LAN interface compatible with IEEE802.11a/b/g/n, etc. can be used. can.

The operation unit 109 is operated by the user to perform various inputs. As the operation unit 109, for example, a keyboard, mouse, switches, etc. are used.

The recording device 110 records video data and audio data in the video conference in the memory 107 . Also, the recording device 110 reproduces video data and audio data recorded in the memory 107 .

(Functional configuration of IWB 100)
FIG. 4 is a diagram showing the functional configuration of the IWB 100 according to one embodiment of the invention. As shown in FIG. 4, the IWB 100 includes a main control unit 120, a video acquisition unit 122, a video processing unit 150, an encoding unit 128, a transmission unit 130, a reception unit 132, a decoding unit 134, a display control unit 136, an audio acquisition It has a section 138 , an audio processing section 140 and an audio output section 142 .

The image acquisition unit 122 acquires image data (YUV data) acquired from the camera 101 . The video data acquired by the video acquisition unit 122 is configured by combining a plurality of frame images.

The video processing unit 150 performs video processing on the video data acquired by the video acquisition unit 122 . The image processing unit 150 includes a blocking unit 151 , an image analysis unit 152 , an image quality determination unit 153 , a specific area detection unit 154 and an image quality adjustment unit 155 .

Blocking unit 151 divides a frame image into a plurality of blocks. For example, in the examples shown in FIGS. 7 and 8, the blocking unit 151 divides one frame image into 48 blocks (8×6 blocks). However, in this example, a relatively small number of blocks is used for the sake of clarity of explanation. In the case of 16 pixels, the frame image is divided into 40×23 blocks. Further, for example, when the resolution of the frame image is 1920×1080 pixels (Full HD), and one block is 16×16 pixels, the frame image is divided into 120×68 blocks.

The video analysis unit 152 analyzes high frequency components for each of the plurality of blocks. To "analyze high frequency components" means to quantify the amount of high frequency components. The high-frequency component represents the magnitude of the grayscale difference between adjacent pixels. That is, in a frame image, an area with a small grayscale difference between adjacent pixels is an area with few high-frequency components and a large grayscale difference between adjacent pixels. The region is a region with many high frequency components. Any known method may be used as a high-frequency component analysis method. cosine transform) or the like can be used.

The image quality determination unit 153 determines image quality for each of the plurality of blocks according to the analysis result of the high frequency component. Specifically, the image quality determination unit 153 creates an image quality level map by setting the image quality for each of the plurality of blocks based on the analysis result of the high frequency components by the video analysis unit 152 . At this time, the image quality determination unit 153 sets the image quality for each block based on the analysis result of the high frequency components by the video analysis unit 152 so that the image quality is higher in areas with more high frequency components. For example, the image quality determining unit 153 selects four levels of image quality “A (highest image quality)”, “B (high image quality)”, “C (medium image quality)”, and “D (low image quality)” for each block. set either.

Note that the image quality determination unit 153 can change the image quality setting in the image quality level map once generated as described above. For example, when a face area is detected by the specific area detection unit 154, the image quality determining unit 153 sets the image quality in the image quality level map so that the image quality of the face area is higher than the image quality of other areas other than the face area. You can change the image quality settings. At this time, the image quality determination unit 153 can reduce the amount of data in the area by changing the image quality of the area other than the area around the face area to the lowest image quality (for example, image quality "D").

Further, for example, the image quality determining unit 153 satisfies a predetermined first condition for determining that the network band (an example of “communication resources used for transmission”) is insufficient (for example, communication). If the speed is equal to or less than a predetermined first threshold), the image quality of other areas can be changed to the lowest image quality (for example, image quality "D"), thereby reducing the amount of data in that area. Further, for example, when the image quality determination unit 153 satisfies a predetermined second condition for determining that there is a margin in the network band (for example, when the communication speed is equal to or higher than a predetermined second threshold; however, second threshold≧first threshold), and by changing the image quality of the face region to the highest image quality (for example, image quality “A”), the image quality of the face region can be improved.

Further, for example, when the image quality determination unit 153 changes the image quality of the area other than the area around the speaker area to "D (low image quality)" when creating the image quality level map, the image quality determination unit 153 changes the image quality of the area around the speaker area to "D (low image quality)". It is possible to restore the image quality of areas other than the area of , to the image quality set in the originally created image quality level map.

The specific area detection section 154 detects a specific area in the video data (frame image) acquired by the video acquisition section 122 . Specifically, the specific area detection unit 154 detects, as a specific area, a face area in which a person's face is detected in the video data (frame image) acquired by the video acquisition unit 122 . Any known method may be used as a method for detecting a face area. For example, a method of extracting feature points such as the eyes, nose, and mouth to detect a face area may be used. Further, the specific area detection unit 154 uses any known detection method to identify, as a speaker area, a face area in which the face of a person who is having a conversation is displayed.

The image quality adjustment unit 155 performs image quality adjustment for each pixel on one frame image according to the final image quality level map. For example, if one of "A", "B", "C", and "D" is set for each block in the image quality level map, the image quality adjustment unit 155 determines that the image quality level relationship is " The image quality of each pixel is adjusted so that A”>“B”>“C”>“D”. Any known method may be used as the image quality adjustment method. For example, the image quality adjustment unit 155 maintains the original image quality for blocks set to image quality "A". Further, for example, the image quality adjustment unit 155 applies some known image quality adjustment method (for example, resolution adjustment, contrast adjustment, low-pass filter, frame rate adjustment, etc.) is used to reduce the image quality from the original image quality (image quality "A"). As an example, a low-pass filter is not applied to blocks set with image quality "A", and a 3×3 low-pass filter is applied to blocks set with image quality "B". A 5×5 low-pass filter is applied to blocks set to "C", and a 7×7 low-pass filter is applied to blocks set to "D". As a result, the information amount of the frame image can be appropriately reduced according to the image quality level.

The encoding unit 128 encodes video data after video processing by the video processing unit 150 . Encoding schemes used by the encoding unit 128 include, for example, H.264/AVC, H.264/SVC, H.265, and the like.

The transmission unit 130 transmits the video data encoded by the encoding unit 128 to the other IWB 100 via the network 16 together with the audio data acquired from the microphone 103 (audio data after audio processing by the audio processing unit 140). Send to

Receiving unit 132 receives video data and audio data transmitted from other IWBs 100 via network 16 . The decoding unit 134 decodes the video data received by the receiving unit 132 using a predetermined decoding method. The decoding method used by the decoding unit 134 is a decoding method (for example, H.264/AVC, H.264/SVC, H.265, etc.) corresponding to the encoding method used by the encoding unit 128 .

The display control unit 136 reproduces the video data decoded by the decoding unit 134 so that the touch panel display 102 displays the video based on the video data (that is, the video of the other site). In addition, the display control unit 136 reproduces the video data acquired from the camera 101 so that the touch panel display 102 displays the video based on the video data (that is, the video of the home base). Note that the display control unit 136 can display multiple types of images in a display layout having multiple display areas based on layout setting information set in the IWB 100 . For example, the display control unit 136 can simultaneously display an image of its own site and an image of another site.

The main control unit 120 controls the IWB 100 as a whole. For example, the main control unit 120 controls the initial setting of each module, the setting of the shooting mode of the camera 101, the communication start request to the other IWB 100, the start of the video conference, the end of the video conference, the recording by the recording device 110, and the like. .

The voice acquisition unit 138 acquires voice data from the microphone 103 . The audio processing unit 140 performs various audio processing on the audio data acquired by the audio acquiring unit 138 and the audio data received by the receiving unit 132 . For example, the audio processing unit 140 performs general audio processing such as codec processing and noise cancellation (NC) processing on the audio data received by the receiving unit 132 . Also, for example, the audio processing unit 140 performs general audio processing such as codec processing and echo cancellation (EC) processing on the audio data acquired by the audio acquisition unit 138 .

The audio output unit 142 converts the audio data received by the receiving unit 132 (audio data after audio processing by the audio processing unit 140) into an analog signal and reproduces it, thereby producing audio based on the audio data (that is, other data). base station) is output from the speaker 104 .

Each function of the IWB 100 described above is realized, for example, by the CPU of the system controller 105 executing a program stored in the auxiliary storage device 106 in the IWB 100 . This program may be provided with being installed in the IWB 100 in advance, or may be provided externally and installed in the IWB 100 . In the latter case, the program may be provided by an external storage medium (e.g., USB memory, memory card, CD-ROM, etc.) or provided by downloading from a server on a network (e.g., Internet, etc.). You may do so. Of the functions of the IWB 100 described above, some functions (for example, some or all of the functions of the video processing unit 150, the encoding unit 128, the decoding unit 134, etc.) are performed separately from the system control 105. It may be realized by dedicated processing circuitry provided.

(Procedure of video conference execution control processing by IWB 100)
FIG. 5 is a flow chart showing the procedure of videoconference execution control processing by the IWB 100 according to one embodiment of the present invention.

First, the main control unit 120 performs initial setting of each module to enable imaging by the camera 101 (step S501). Next, the main control unit 120 sets the shooting mode of the camera 101 (step S502). The setting of the photographing mode by the main control unit 120 can include those automatically performed based on the outputs of various sensors and those manually performed by the operator's operation input. Then, the main control unit 120 requests the IWB 100 at the other site to start communication, and starts the video conference (step S503). Note that the main control unit 120 may start the videoconference upon receiving a communication start request from another IWB 100 . Also, the main control unit 120 may start the video and audio recording by the recording device 110 at the same time as the video conference is started.

When the video conference is started, on the one hand, the video acquisition unit 122 acquires video data (YUV data) from the camera 101, and the audio acquisition unit 138 acquires audio data from the microphone 103 (step S504). Then, the video processing unit 150 performs video processing (described in detail in FIG. 6) for the video data acquired in step S504, and the audio processing unit 140 performs various audio processing for the audio data acquired in step S504. Processing is performed (step S505). Furthermore, the encoding unit 128 encodes the video data after the video processing in step S505 (step S506). Then, the transmission unit 130 transmits the video data encoded in step S506 together with the audio data acquired in step S504 to another IWB 100 via the network 16 (step S507).

In parallel with steps S504 to S507, receiving unit 132 receives video data and audio data transmitted from other IWBs 100 via network 16 (step S508). Then, the decoding unit 134 decodes the video data received in step S508. Also, the audio processing unit 140 performs various audio processing on the audio data received in step S508 (step S510). Furthermore, the display control unit 136 causes the touch panel display 102 to display the video based on the video data decoded in step S509, and the audio output unit 142 outputs audio based on the audio data after the audio processing in step S510. , is output from the speaker 104 (step S511). In addition, in step S511, the display control unit 136 can also cause the touch panel display 102 to display an image based on the image data acquired in step S504 (that is, the image of its own site).

Following the transmission process of steps S504 to S507, main control unit 120 determines whether or not the video conference has ended (step S512). Further, following the reception process of steps S508 to S511, the main control unit 120 determines whether or not the video conference has ended (step S513). The end of the videoconference is determined, for example, when the user performs a predetermined end operation in any of the IWBs 100 participating in the videoconference. If it is determined in step S512 that the video conference has not ended (step S512: No), the IWB 100 returns to step S504. That is, the transmission process of steps S504 to S507 is repeatedly executed. If it is determined in step S513 that the video conference has not ended (step S513: No), the IWB 100 returns to step S508. That is, the receiving process of steps S508 to S511 is repeatedly executed. On the other hand, if it is determined in step S512 or step S513 that the video conference has ended (step S512: Yes or step S513: Yes), the IWB 100 ends the series of processes shown in FIG.

(Procedure of video processing by video processing unit 150)
FIG. 6 is a flow chart showing the procedure of image processing by the image processing unit 150 according to one embodiment of the present invention. FIG. 6 shows in detail the video processing procedure of step S505 in the flowchart of FIG.

First, the blocking unit 151 sequentially selects one frame image from among a plurality of frame images forming video data (step S601). Then, the blocking unit 151 divides one frame image selected in step S601 into a plurality of blocks (step S602).

Next, the video analysis unit 152 analyzes the high-frequency components of the one frame image selected in step S601 for each block divided in step S602 (step S603).

Then, the image quality determination unit 153 sets the image quality for each block divided in step S602 for the one frame image selected in step S601 based on the analysis result of the high-frequency component in step S603. A level map is created (step S604).

Next, the specific area detection unit 154 detects a face area, which is an area in which a person's face is displayed, from the one frame image selected in step S601 (step S605). Further, the specific area detection unit 154 detects a speaker area, which is an area in which the face of a person who is having a conversation is displayed, from the face areas detected in step S605 (step S606).

Then, the image quality determining unit 153 changes the image quality level map created in step S604 based on the face area detection result in step S605 and the speaker area detection result in step S606 (step S607). For example, the image quality determination unit 153 changes the image quality of the face area, which is the speaker area, to "A (highest image quality)" for the image quality level map created in step S604, and changes the image quality of the face area, which is not the speaker area, to "B (high quality)". In addition, the image quality determination unit 153 sets the image quality of the area other than the speaker area to " D (low quality)”.

Next, the image quality determination unit 153 determines whether the network band being used for the videoconference has room (step S608). In step S608, if it is determined that there is enough network bandwidth (step S608: Yes), the image quality determination unit 153 changes the image quality level map so as to improve the image quality of some areas (step S609). For example, the image quality determination unit 153 changes the image quality of the face area that is not the speaker area from "B (high image quality)" to "A (best image quality)", and furthermore, changes the image quality of the area that is not the peripheral area of the speaker area. The image quality is returned to the image quality set in the image quality level map initially created in step S604. After that, the video processing unit 150 advances the process to step S612.

On the other hand, if it is determined in step S608 that the network band being used for the video conference does not have enough margin (step S608: No), the image quality determining unit 153 determines whether the network band is insufficient (step S610). In step S610, if it is determined that the network bandwidth is insufficient (step S610: Yes), the image quality determination unit 153 changes the image quality of areas other than the face area to "D (low image quality)" ( step S611). Then, video processing unit 150 advances the process to step S612.

On the other hand, if it is determined in step S610 that the network bandwidth is not insufficient (step S610: No), the video processing unit 150 advances the process to step S612.

In step S612, the image quality adjustment unit 155 performs image quality adjustment for each pixel on the one frame image selected in step S601 according to the final image quality level map.

After that, the video processing unit 150 determines whether or not the above video processing has been performed on all the frame images forming the video data (step S613). When it is determined in step S613 that video processing has not been performed on all frame images (step S613: No), the video processing unit 150 returns the processing to step S601. On the other hand, if it is determined in step S613 that video processing has been performed on all frame images (step S613: Yes), the video processing unit 150 ends the series of processes shown in FIG.

(Specific example of video processing by video processing unit 150)
7 and 8 are diagrams showing a specific example of video processing by the video processing unit 150 according to one embodiment of the present invention. A frame image 700 shown in FIG. 7 represents an example of one frame image to be subjected to video processing by the video processing unit 150 .

First, as shown in FIG. 7A, a frame image 700 is divided into a plurality of blocks by the blocking unit 151. FIG. In the example shown in FIG. 7A, the frame image 700 is divided into 48 blocks (8×6 blocks).

Next, the video analysis unit 152 analyzes the high frequency components of each of the plurality of blocks in the frame image 700 . In the example shown in FIG. 7(a), any one of "0" to "3" is shown as the level of the high-frequency component, which is the analysis result of the high-frequency component, for each block. Here, the magnitude relationship of the levels of high-frequency components is "3">"2">"1">"0".

Next, an image quality level map corresponding to the frame image 700 is created by the image quality determining section 153 . The image quality level map 800 shown in FIG. 7(b) is created by the image quality determination unit 153 based on the analysis results of the high frequency components shown in FIG. 7(a). In the example shown in FIG. 7B, in the image quality level map 800, for each block, the image quality is "A (highest image quality)", "B (high image quality)", "C (middle image quality)", "D ( low image quality)” is set. The image qualities "A", "B", "C" and "D" correspond to the high frequency component levels "3", "2", "1" and "0", respectively.

Next, the specific area detection unit 154 detects a face area, which is an area in which a person's face is displayed, from the frame image 700 . Further, the specific area detection unit 154 detects, from the face areas detected in the frame image 700, a speaker area, which is an area in which the face of a person who is having a conversation is displayed. In the example shown in FIG. 7C, face areas 710 and 712 are detected. Of these, the face area 710 is detected as the speaker area.

Then, the image quality determination unit 153 changes the image quality level map 800 based on the detection results of the face areas 710 and 712 . For example, in the example shown in FIG. 8(a), the image quality determination unit 153 sets the image quality of the face area 710, which is the speaker area, to "A (best image quality)" for the image quality level map 800 shown in FIG. 7(b). , and the image quality of the face area 712 that is not the speaker area is changed to "B (high image quality)". Further, in the example shown in FIG. 8A, in areas other than the face areas 710 and 712 (hereinafter referred to as "background area 720"), the image quality determining unit 153 The image quality of the area other than the peripheral area of the face area 710 is changed to "D (low image quality)" without changing the image quality.

Furthermore, when it is determined that the network band being used for the video conference has room, the image quality determination unit 153 changes the image quality level map 800 so as to increase the image quality of a part of the area.

For example, in the example shown in FIG. 8B, the image quality determination unit 153 changes the image quality of the face area 712 from "B (high image quality)" to "A (highest image quality)" in the image quality level map 800. .

In the example shown in FIG. 8(c), the image quality determining unit 153 sets the image quality of the area other than the surrounding area of the speaker area in the background area 720 to the image quality "D (low image quality)" in the image quality level map 800. , the image quality is returned to the initially set image quality shown in FIG. 7(b).

Conversely, if it is determined that the network bandwidth being used for the videoconference is insufficient, the image quality determination unit 153 determines that the image quality of the background area 720 is reduced in the image quality level map 800, as shown in FIG. 8(d). , is changed to “D (low image quality)”.

The frame image 700 is the final image quality level map 800 created as described above by the image quality adjustment unit 155 (one of the image quality level maps 800 shown in FIG. 7(b) and FIGS. 8(a) to (d)). Image quality adjustment is performed for each pixel based on.

As a result, the frame image 700 has a relatively high image quality set in the face regions 710 and 712 that attract relatively high viewer attention, and the background region 720 that receives relatively low viewer attention. In , a relatively low image quality is set.

However, in the frame image 700, for the background area 720, relatively high image quality is set for areas where image quality deterioration is relatively noticeable (areas with many high frequency components such as blinds) according to the analysis result of the high frequency components. A relatively low image quality is set for areas where image quality deterioration is relatively inconspicuous (areas with few high-frequency components, such as walls and displays). For this reason, frame image 700 is one in which image quality deterioration in background area 720 is less noticeable.

Furthermore, in the frame image 700, the image quality of the background area 720 changes step by block in the spatial direction. Therefore, in the frame image 700, in the background area 720, the difference in image quality at the boundary between the area where relatively high image quality is set and the area where relatively low image quality is set is less noticeable.

Therefore, according to the IWB 100 of this embodiment, it is possible to reduce the amount of video data and make the difference in image quality at the boundary between the low image quality area and the high image quality area less noticeable.

Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications or Change is possible.

For example, in the above embodiment, the IWB 100 (electronic blackboard) is used as an example of the "video processing device" and the "communication terminal", but the present invention is not limited to this. For example, the functions of the IWB 100 described in the above embodiments may be implemented by other information processing devices (for example, smart phones, tablet terminals, laptop computers, etc.) equipped with an imaging device, and other information processing devices without an imaging device may be implemented. It may be implemented by an information processing device (for example, a personal computer, etc.).

Also, for example, in the above embodiment, an example in which the present invention is applied to a video conference system has been described, but the present invention is not limited to this. That is, the present invention can be applied to any application as long as the purpose is to reduce the amount of data by lowering the image quality of a partial area of video data. Moreover, the present invention can also be applied to an information processing apparatus that does not encode or decode video data.

Also, for example, in the above embodiment, the face detection area is used as an example of the "specific area", but the present invention is not limited to this. That is, if the "specific area" is an area in which a subject that is preferably of relatively high image quality (for example, a document showing text or an image, a whiteboard, a person in a surveillance camera, etc.) is projected, It can be any area.

In the above embodiment, various setting values used for each process (for example, the type of subject to be detected in a specific area, the block size and number of blocks used when dividing a frame image, the level of analysis results of high-frequency components, etc.) The number of steps, the number of steps of image quality level, adjustment items and adjustment amounts for image quality adjustment, etc.) may be set to suitable values in advance. For example, it may be possible to arbitrarily set suitable values from a personal computer or the like.

10 video conference system 12 conference server (server device)
14 conference reservation server 16 network 100 IWB (video processing device, imaging device, communication terminal)
101 camera (imaging unit)
102 touch panel display (display device)
103 Microphone 104 Speaker 108 Communication I/F (Communication Unit)
120 main control unit 122 video acquisition unit 128 encoding unit 130 transmission unit 132 reception unit 134 decoding unit 136 display control unit 138 audio acquisition unit 140 audio processing unit 142 audio output unit 150 video processing unit 151 blocking unit 152 video analysis unit 153 Image quality determination unit 154 Specific area detection unit 155 Image quality adjustment unit

JP 2017-163228 A

Claims (10)

an image acquisition unit that acquires an image;
a video analysis unit that analyzes high-frequency components for each region in the video acquired by the video acquisition unit;
a specific area detection unit that detects a specific area that is an area in which a person's face is displayed in the video;
an image quality adjustment unit that adjusts the image quality so that at least a part of the region in the image has a higher image quality as the area with more high frequency components, according to the analysis result by the image analysis unit ;
The image quality adjustment unit
Adjusting the image quality so that the image quality of the specific region is higher than the image quality of other regions other than the specific region,
When a plurality of the specific areas are detected by the specific area detection unit, the specific area in which the face of the person who is not talking is displayed is replaced with the specific area in which the face of the person who is talking is displayed. Make the image quality lower than the area
A video processing device characterized by : further comprising a blocking unit that divides the video into a plurality of blocks;
The video analysis unit
analyzing the high-frequency component for each block in the video;
The image quality adjustment unit
The video processing device according to claim 1, wherein the image quality adjustment is performed for each block in the video. The image quality adjustment unit
The image quality of the area that is the peripheral area of the specific area in the other area is the image quality according to the analysis result, and the image quality of the area that is not the peripheral area of the specific area in the other area is the analysis result. 3. The video processing apparatus according to claim 1 , wherein the image quality adjustment is performed so that the image quality is lower than the image quality corresponding to the image quality. an encoding unit that encodes the video after image quality adjustment by the image quality adjustment unit;
The video processing device according to any one of claims 1 to 3, further comprising a communication unit that transmits the video encoded by the encoding unit to an external device. The image quality adjustment unit
5. The video processing apparatus according to claim 4 , wherein the image quality of the other area is set to the lowest image quality when communication resources used for the transmission are insufficient. The image quality adjustment unit
6. The video processing device according to claim 4 , wherein the image quality of the specific area is set to the highest image quality when there is sufficient communication resource to be used for the transmission. The image quality adjustment unit
7. The video processing device according to any one of claims 4 to 6 , wherein the image quality of the other area is enhanced when there is a margin in the communication resources used for the transmission. a plurality of communication terminals;
a server device that performs various controls related to video conferencing by the plurality of communication terminals;
each of the plurality of communication terminals,
an imaging unit that captures an image;
a video analysis unit that analyzes high-frequency components for each region in the video imaged by the imaging unit;
a specific area detection unit that detects a specific area that is an area in which a person's face is displayed in the video;
an image quality adjustment unit that adjusts the image quality so that at least a part of the region in the image has a higher image quality as the high frequency component increases, according to the analysis result of the image analysis unit;
a communication unit that transmits the image after the image quality adjustment by the image quality adjustment unit to an external device ;
The image quality adjustment unit
Adjusting the image quality so that the image quality of the specific region is higher than the image quality of other regions other than the specific region,
When a plurality of the specific areas are detected by the specific area detection unit, the specific area in which the face of the person who is not talking is displayed is replaced with the specific area in which the face of the person who is talking is displayed. Make the image quality lower than the area
A video conference system characterized by: an image acquisition step of acquiring an image;
a video analysis step of analyzing high-frequency components for each region in the video acquired in the video acquisition step;
a specific area detection step of detecting a specific area that is an area in which a person's face is projected in the video;
an image quality adjustment step of adjusting the image quality so that at least a part of the region in the image has a higher image quality as the high frequency component increases, according to the analysis result in the image analysis step ;
The image quality adjustment step includes:
Adjusting the image quality so that the image quality of the specific region is higher than the image quality of other regions other than the specific region,
When a plurality of the specific regions are detected by the specific region detection step, the specific region showing the face of the person not having a conversation is replaced with the specific region showing the face of the person having a conversation. Make the image quality lower than the area
A video processing method characterized by: the computer,
an image acquisition unit that acquires an image;
a video analysis unit that analyzes high-frequency components for each region in the video acquired by the video acquisition unit;
a specific area detection unit that detects a specific area that is an area in which a person's face is displayed in the video; and
functioning as an image quality adjustment unit that adjusts image quality so that at least a part of the region in the image has a higher image quality as the region with more high frequency components, according to the analysis result by the image analysis unit ;
The image quality adjustment unit
Adjusting the image quality so that the image quality of the specific region is higher than the image quality of other regions other than the specific region,
When a plurality of the specific areas are detected by the specific area detection unit, the specific area in which the face of the person who is not talking is displayed is replaced with the specific area in which the face of the person who is talking is displayed. Make the image quality lower than the area
program.

Images