JP2008306265A - Image processor, image processing method, video conference system, video conference method, program, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent images captured in respective conference rooms from mutually be fed back to be displayed one over another on display devices etc., in the respective conference rooms. <P>SOLUTION: When an image imaged by imaging a member C in a first conference room by a camera in the conference room is displayed in a second conference room, a fist image including the member C, a display device behind it, and a display area 3021b thereof is displayed. When an image imaged by imaging a member A in front of the first image in a second conference room by a camera therein is displayed in the first conference room, a second image obtained by feeding the first image back is displayed in a display area 2021b together with the member A. For the purpose, the first and second images are compared with each other to determine an area 1000, shown by a dotted line, including an image such that the both match each other. The member A is not included in the area 1000. An image obtained by converting all pixels in the area 1000 to gray is displayed in the first conference room together with the member A. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, a video conference system, a video conference method, a program, and a recording medium used for performing a video conference and the like.

Video conferencing systems have been developed in the past. By using the video conferencing system, video / audio can be transmitted and received between two or more remote conference rooms. We were able to hold a meeting.
On the other hand, as a conventional image processing technology, there is a technology for generating a closed contour without a moving region in a moving image for which a background image is not prepared, and separating the moving region and the background region using the closed contour. It has been developed (see, for example, Patent Document 1).
Japanese Patent Laid-Open No. 11-353486 (first page, FIG. 1 etc.)

  However, in the conventional video conference system, depending on the installation location and orientation of the camera that captures the state of the conference in the first conference room, the camera is not only a participant in the first conference room, There may be a case where an image obtained by photographing the state of the second conference room displayed on a display or the like disposed inside, an image of a document, or the like is photographed. Thus, an image obtained by re-taking an image once displayed on a display or the like usually has poor image quality and is difficult to grasp. Therefore, when such an image is displayed in the second conference room, it is annoying and the image displayed in the display in this image is displayed in the conference room in the second conference room. There has been a problem that attention has shifted and it is not possible to concentrate on the meeting, and there is a possibility that an efficient meeting cannot be realized.

In particular, when an image taken in the first meeting room includes an image including a display that displays the state of the second meeting room, the image taken in the first meeting room is When sent to the second conference room at a remote location and displayed on a display or the like, the display or the like of the second conference room contains images or the like taken in the second conference room that are not originally required to be displayed. Feedback will be displayed. As a result, in the second meeting room, the state of the first meeting room and the state of the second meeting room are displayed on the same screen, and the entire displayed image is congested, and the concentration in the meeting There was a problem that it would not come.
Specifically, one of the participants speaking in the second meeting room is also displayed on the display of the second meeting room. As a result, in the second meeting room, where and who There was a problem that it was difficult to instantly determine whether or not he was speaking, and the participants could not grasp the situation and were confused and unable to concentrate on the meeting.
The present invention has been made to solve the above problem, and it is an object of the present invention to prevent images taken in each conference room from being fed back to each other and displayed on a display or the like in each conference room. To do.

An image processing apparatus of the present invention includes an image information storage unit that stores first image information that is information indicating a first image, and a display unit that displays the first image using the first image information. Using an image receiving unit that receives a second image that is an image of an area including the first image displayed by the display unit, and the first image information stored in the image information storage unit, A difference image information constituting unit constituting difference image information which is information indicating a difference image which is an image obtained by removing the first image included in the second image from the second image received by the image receiving unit. And an output unit that outputs the difference image information configured by the difference image information configuration unit.
With this configuration, it is possible to configure and output an image obtained by deleting the image displayed in the shooting area from the shot image. For example, by using this image processing apparatus for a video conference or the like, an image that hinders the conference can be removed from the conference image, and an efficient conference can be realized.

In the image processing apparatus of the present invention, in the image processing apparatus, the difference image information configuration unit may display an image on an area of the second image where the display unit displays the first image. Compared with the first image indicated by the first image information stored in the image information storage unit, an area having an image that matches the first image is determined, and the determined from the second image It is an image processing apparatus which comprises difference image information which shows the difference image which removed the image in an area | region.
With this configuration, it is possible to configure and output an image obtained by deleting the image displayed in the shooting area from the shot image.

In the image processing device of the present invention, in the image processing device, the difference image information configuration unit is an area in which the display unit in the first image or the second image displays the first image. An image processing apparatus that performs the comparison by converting the number of pixels of at least one of the upper images.
With this configuration, it is possible to accurately compare pixels between images, and thus it is possible to accurately delete an image displayed in a shooting area from a shot image.

In the image processing device of the present invention, in the image processing device, the difference image information configuration unit is an area in which the display unit in the first image or the second image displays the first image. It is an image processing apparatus for converting the color of at least one of the above images and performing the comparison.
With this configuration, it is possible to compare pixels between images with high accuracy, so that an image displayed in the shooting region can be deleted with high accuracy from the captured image.

The video conference system of the present invention is a teleconferencing system comprising a first image processing device and a second image processing device, wherein the first image processing device outputs the second image processing device. A first receiving unit that receives first image information that is information indicating a first image; a first image information storage unit that stores first image information received by the first receiving unit; and the first image Using the information, a first display unit that receives the first image that displays the first image and a second image that is an image of a region that includes the first image displayed by the first display unit. And the first image included in the second image from the second image received by the first image receiving unit using the first image information stored in the first image information storage unit. The difference image which is information indicating the difference image which is an image obtained by removing the image of A first difference image information constituting unit constituting a report, and a first output unit outputting the difference image information constituted by the first difference image information constituting unit, wherein the second image processing device A second receiving unit that receives the difference image information output by the processing device, a second image information storage unit that stores the difference image information received by the second receiving unit, and the difference image information, A second display unit that displays a difference image, a second image reception unit that receives a third image that is an image of an area including the difference image displayed by the second display unit, and the second image information storage unit The first image which is an image obtained by removing the difference image included in the third image from the third image received by the second image receiving unit using the difference image information stored in The second image information constituting the first image information And min image information forming unit, a television conference system in which the second difference image information forming unit includes a second output section that outputs the first image information constructed.
With this configuration, it is possible to construct and output an image obtained by deleting the image displayed in the shooting area from an image obtained by capturing the situation of the conference and the like, and removing an image that hinders the conference from the image of the conference. Thus, an efficient meeting can be realized.

In the video conference system according to the present invention, in the video conference system, the first difference image information configuration unit is on an area where the first display unit in the second image displays the first image. The image is compared with the first image indicated by the first image information stored in the first image information storage unit, an area having an image that matches the first image is determined, and the second image information is stored. The difference image information indicating the difference image obtained by removing the image in the determined area from the image is configured, and the second difference image information configuring unit is configured so that the second display unit in the third image is The image on the area displaying the difference image is compared with the difference image indicated by the difference image information stored in the second image information storage unit, and an area having an image that matches the difference image is determined. , From the third image A television conference system that constitutes the first image information indicating the image the first image obtained by removing the boss was region.
With this configuration, it is possible to construct and output an image obtained by deleting the image displayed in the shooting area from an image obtained by capturing the situation of the conference and the like, and removing an image that hinders the conference from the image of the conference. Efficient meetings.

In the video conference system of the present invention, in the video conference system, the first differential image information configuration unit displays the first image in the display unit in the first image or the second image. The comparison is performed by converting the number of pixels of at least one of the images on the area to be processed, and the second difference image information configuration unit displays the difference image on the difference image or the third image in the display unit. This is a video conference system that performs the comparison by converting the number of pixels of at least one of the images on the area to be processed.
With this configuration, it is possible to accurately compare pixels between images, and thus it is possible to accurately delete an image displayed in a shooting area from a shot image.

In the video conference system of the present invention, in the video conference system, the first differential image information configuration unit displays the first image in the display unit in the first image or the second image. The comparison is performed by converting at least one color of the image on the area to be performed, and the second difference image information configuration unit displays the difference image on the display unit in the difference image or the third image. In the video conference system, the comparison is performed by converting at least one color of an image on a region.
With this configuration, it is possible to accurately compare pixels between images, and thus it is possible to accurately delete an image displayed in a shooting area from a shot image.

  According to the present invention, it is possible to configure and output an image obtained by deleting the image displayed in the shooting area from the shot image.

Hereinafter, embodiments of an image processing apparatus and the like will be described with reference to the drawings. In addition, since the component which attached | subjected the same code | symbol in embodiment performs the same operation | movement, re-explanation may be abbreviate | omitted.
FIG. 1 is a block diagram of an image processing apparatus according to this embodiment.
The image processing apparatus 100 includes an image information storage unit 101, a display unit 102, a photographing unit 103, an image reception unit 107, a difference image information configuration unit 104, and an output unit 105.

  The image information storage unit 101 stores first image information that is information indicating the first image. The first image is usually a moving image, but may be a still image. The first image is preferably a color image, but may be a grayscale image. This first image is, for example, an image of a meeting scene. A specific example is an image taken in another conference room and transmitted in real time in a video conference system using the image processing apparatus 100 as a terminal. In addition, the first image may be an image obtained by storing an image of a meeting scene.

  The first image is not limited to an image obtained by capturing a meeting scene, and may be any image. For example, it may be an image of a material used for a meeting or an image for presentation. The first image information may be analog image information or digital image information. For example, in the case of digital image information, MPEG (Moving Picture Experts Group), ITU-TH. H.264 and other algorithms for compressing and encoding image information may be used. Further, uncompressed image information may be output. The image information may be any image quality, resolution, etc. such as an HD image. Further, the first image information may include synchronization information for synchronizing with other image information and the like. Also, synchronization information may be added.

  The image information storage unit 101 can be realized by a predetermined recording medium such as a semiconductor memory, a magnetic disk, or an optical disk. The process in which the first image information is stored in the image information storage unit 101 does not matter. For example, the first image information may be stored in the image information storage unit 101 via a recording medium (not shown), and the first image information transmitted via a communication line (not shown) is stored in the image information storage unit 101. The first image information input via the input device may be stored in the image information storage unit 101. In addition to the first image information, the image information storage unit 101 may store audio information corresponding to the first image information. The storage by the image information storage unit 101 may be temporary storage in the RAM or the like of the first image information transmitted via a communication line or the like, so-called buffering, or long-term storage.

  The display unit 102 displays the first image using the first image information stored in the image information storage unit 101. Specifically, a first image configured using the first image information is displayed on a display device such as a liquid crystal display, a plasma display, or a CRT display. Note that the display unit 102 may output audio information corresponding to the first image information together with the first image. The resolution at which the display unit 102 displays the first image does not matter. In addition, when the first image information includes mask information or transmission information of the first image, the display unit 102 uses the information to display a part of the first image in a transparent manner. Alternatively, a partial area of the first image may be replaced with another image, for example, a background image or a specific color image for display. The display unit 102 may or may not include a display device and a speaker that perform display. The display unit 102 may be realized by hardware, or may be realized by software such as a driver that drives a display device.

  The imaging unit 103 captures a second image that is an image of an area including the first image that the display unit 102 displays using a display device or the like. Specifically, the imaging unit 103 captures an area including the area where the display unit 102 displays the first image as an imaging target area, and acquires second image information that is information indicating the second image. To do. More specifically, the imaging unit 103 captures an image of an area including a display surface of a display device such as a display that displays the first image as the second image. The image photographed by the photographing unit 103 is usually a moving image, but may be one or more still images. Note that a plurality of still images taken at predetermined time intervals that are constant or indefinite are moving images. The second image is usually a color image, but may be a grayscale image.

  The angle formed by the photographing direction of the photographing unit 103 and the display surface of the first image displayed on the display unit 102 is not limited. The imaging unit 103 is a camera that captures images, for example, and captures images using an image sensor such as a CCD or CMOS. Further, when the photographing unit 103 also acquires audio information, the photographing unit 103 may include a microphone, for example. There is no limitation on the data structure, timing, or the like for the imaging unit 103 to output an image. For example, the photographing unit 103 may output the acquired moving image information as one moving image file. Further, the photographing unit 103 may sequentially output image information of each frame constituting the moving image or image information corresponding to each frame, or a predetermined number of frames of images have been acquired. At the time, the image information may be output in units of a predetermined number of frames.

  In addition, the photographing unit 103 may create a moving image in which time position information that is information that can specify an arbitrary temporal position in the moving image is added to the captured moving image. For example, a moving image to which information indicating a time position such as information indicating the time at the time of shooting is added may be created as the time position information. For example, the image capturing unit 103 adds, to the image information of each frame constituting the moving image, information on the time at which the image information was acquired as time position information, specifically, a time code or image information added as a time stamp. May be created. Further, frame identification information for identifying a frame may be added to each frame of the image information.

  The image information created and output by the photographing unit 103 may be analog image information or digital image information. For example, in the case of digital image information, MPEG (Moving Picture Experts Group) or ITU-T H.264 is used. H.264 and other algorithms for compressing and encoding image information may be used. Further, uncompressed image information may be output. The image information may be any image quality, resolution, etc. such as an HD image. Further, the photographing unit 103 may include a microphone and acquire voice information.

  The imaging unit 103 may output device identification information for identifying the imaging unit 103. When the photographing unit 103 is a photographing device such as a camera, the device identification information may be a manufacturing number of the photographing device or an identification number set in the photographing unit 103 in advance. . Further, it may be physical address information on the network such as the IP address of the photographing unit 103. The device identification information may be information that can identify the imaging unit 103 as a result, and may be information that identifies an input terminal such as a port number to which the imaging unit 103 is connected. The device identification information is stored in advance in a memory (not shown) in the photographing unit 103, for example. Moreover, it does not ask how the imaging | photography part 103 transmits identification information. Specifically, the identification information can be output by the same method as the position identification information described above.

  The image receiving unit 107 receives a second image that is an image of an area including the first image displayed on the display unit 102. Here, specifically, the second image information, which is information indicating the second image captured by the imaging unit 103 output from the imaging unit 103, is received. Here, this acceptance may be, for example, a wired or wireless communication line (for example, the Internet, an intranet, a LAN, or the like, or an IEEE 1394 cable, a dedicated cable for transmitting video information, or the like). Information received via a predetermined recording medium (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.) may be received. In the present embodiment, it is assumed that the image reception unit 107 receives information indicating the second image captured by the imaging unit 103 via a dedicated cable or the like. The image receiving unit 107 may or may not include a device (for example, a modem or a network card) for receiving. The image receiving unit 107 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

  The difference image information configuration unit 104 uses the first image information stored in the image information storage unit 101 to select the first image included in the second image from the second image received by the image reception unit 107. Difference image information that is information indicating a difference image that is an image from which the image has been removed is created. Specifically, the difference image information configuration unit 104 determines that the second image received by the image reception unit 107 matches the first image indicated by the first image information stored in the image information storage unit 101. Difference image information that is information indicating a difference image that is an image from which the image to be removed is removed is created. Usually, the process of removing the first image included in the second image from the second image is performed by digital processing. For this reason, when the first image information or the second image information, which is information indicating the second image, is analog information, it is preferable to perform processing after conversion to a digital signal using a digital signal processor or the like. . However, the process of removing the first image included in the second image from the second image may be performed by analog processing.

  The first image included in the second image removed by the difference image information configuration unit 104 is determined by using a generally known pattern matching technique. The image to be removed may not be an image that completely matches the first image indicated by the first image information. Since the first image included in the second image is obtained by photographing the first image displayed on the display or the like by the display unit 102 by the photographing unit 103, the color, the number of pixels, This is because details or the like of the image is changed or lost, and usually does not completely match the first image indicated by the first image information. Therefore, the difference image information configuration unit 104 removes an image that is determined to be substantially the first image included in the second image.

  Specifically, the difference image information configuration unit 104 displays the image on the area where the display unit 102 displays the first image in the second image as the first image information stored in the image information storage unit 101. The difference image which shows the difference image which determined the area | region which has the 1st image in a 2nd image compared with the 1st image which shows, and removed the image in the determined area | region from a 2nd image Create information. When the first image and the second image indicated by the first image information are moving images, the first image and the second image indicated by the first image information stored in the image information storage unit 101 used for comparison are used. An image is usually an image in frame units that constitute each of these moving images, and images of one frame are used for comparison.

  Note that the first image and the second image indicated by the first image information stored in the image information storage unit 101 used for comparison may be each still image constituting each moving image. The first image indicated by the first image information stored in the image information storage unit 101 used for comparison may be the first image displayed by the display unit 102 at the time when the imaging unit 103 captures the image. preferable. However, in particular, when the image processing apparatus 100 is used for a video conference or the like, the first image indicated by the first image information used for comparison is captured by the photographing unit 103 for the second image of one frame. At this time, the image may be one frame image that is temporally changed with respect to the one frame image of the first image displayed. This is because, in an image used for a video conference or the like, an image with a sudden movement is usually not captured and displayed.

“Area in which the display unit 102 displays the first image in the second image” is, for example, an area in which the display unit 102 in the second image displays the first image in advance in the second image. It may be possible to set information for designating an area in which the area is displayed. For example, if this area is a rectangle, coordinate information specifying four vertices of the rectangle, vector information, and the like can be used as the information specifying this area.

  Further, the area for displaying the first image may be detected by image processing or the like. For example, the difference image information configuration unit 104 specifies four frames, such as a display (not shown) frame on which the first image is displayed, and corners of the frame, so that the first image that is an area inside the frame is specified. You may specify the area | region which displays. A predetermined color, pattern, marker, etc. are arranged at the four points of the frame of the display and the corner of the frame so that the four points of the frame of the display and the corner of the frame can be specified. May be. The predetermined color is preferably a color that is not generally used in a meeting scene. The marker may be, for example, an optical marker used in motion capture or the like. The specific method of the area in which the display unit 102 displays the first image in the second image is not limited to these methods, and other methods may be used. In addition, a method for specifying such a predetermined region is already known, and detailed description thereof is omitted.

  When the second image and the area where the display unit 102 displays the first image match, for example, when the shooting area of the shooting unit 103 and the display area of the display unit 102 match, this The process of setting “an area in which the display unit 102 displays the first image in the second image” can be omitted. Alternatively, the “region in which the display unit 102 displays the first image in the second image” may be set to the entire region of the second image. Once the display area has been set, it is usually fixed as long as the area captured by the imaging unit 103 and the position of the display of the display unit 102 are not changed.

  Further, “on the region where the display unit 102 in the second image displays the first image” means not only the first image actually included in the second image but also the display unit 102. This is an area where the display unit 102 displays the first image when objects such as overlapping conference participants are ignored. For example, when the display area of the display used when the display unit 102 displays the second image is a quadrangle, it is assumed that an object that covers a part of the display, such as a user, is captured in the second image. In addition, the area where the display unit 102 displays the first image is a rectangle. However, the shape may not completely match depending on the orientation of the display area of the display device with respect to the photographing unit 103.

  The difference image information configuration unit 104 includes, for example, an image on an area where the display unit 102 displays the first image in the second image specified as described above (hereinafter referred to as a second area image), Corresponding pixels are compared among the pixels constituting the first image indicated by the first image information stored in the image information storage unit 101. Then, as a result of the comparison, the first image indicated by the first image information stored in the image information storage unit 101 indicates the pixels in the second image determined to have a predetermined relationship set in advance. It is determined that the pixel corresponds to. Then, the same determination is sequentially performed on the pixels in the second region image, usually all the pixels, and the region including the pixels corresponding to the first pixel is determined. And the difference image information which shows the difference image which removed the image of this area | region is produced.

  The comparison between the corresponding pixels is not necessarily a one-to-one comparison between the corresponding pixels. For example, one pixel may be compared with a plurality of pixels, or a plurality of pixels may be compared with a plurality of pixels. This is because when the number of pixels and the like of the second region image and the first image indicated by the first image information stored in the image information storage unit 101 are different, the pixels do not correspond one-on-one. For example, when comparing one pixel of one image with a plurality of pixels of the other image corresponding to this pixel, the pixel value of one pixel of one image and the average value or intermediate value of the pixel values of the plurality of pixels You may compare with representative values, such as a value. For example, when comparing a plurality of pixels of one image with a plurality of pixels of the other image corresponding to the plurality of pixels, a representative value such as an average value of pixel values of the plurality of pixels of one image And a representative value such as an average value of pixel values of a plurality of pixels may be compared. For example, comparison between pixels is usually performed by comparing pixel information, for example, information on each channel of RGB, and values of color information such as luminance information.

  Specifically, when the second region image and the first image are in a similar relationship and the total number of pixels of both is the same, information on a predetermined pixel between coordinates at the same position, for example, color information If the difference is within a preset threshold value, it is determined that the pixel matches, and if the difference is greater than the threshold value, it is determined that the pixel does not match. Then, such a process is performed for all the pixels, and an area including matching pixels is determined. The process of comparing pixel information to determine whether or not they match is a known technique.

  In addition, pattern matching may be sequentially performed on regions at the same position in two images to determine matching regions in the second region image, or these methods may be combined. Further, among the second region images, a minute pixel region or the like surrounded by a pixel region that matches the first image is considered as noise and is determined to be a pixel that matches the first image. It may be. There are various processes for comparing the images and determining the matching area, including the above-described method, but since it is a known technique, a detailed description thereof will be omitted.

  When the second region image and the first image indicated by the first image information stored in the image information storage unit 101 are not similar or when the total number of pixels is not the same, the following is performed. You may make it perform a process. First, a correlation function between the shape and size of the image on the area where the display unit 102 displays the first image in the second image and the shape and size of the first image indicated by the first image information is obtained. Then, using this correlation function or its inverse function, the coordinates in the first image corresponding to each pixel of the second region image are calculated, and the first image indicated by the first image information is calculated. The information such as the color information of the pixel at the position indicated by the coordinates and the information such as the color information of the pixel of the corresponding second region image are compared using a threshold value or the like as described above. And the area | region containing the pixel judged to correspond with the 1st image which 1st image information shows in a 2nd area image is determined. And the difference image information which shows the difference image which removed the image of the area | region determined as mentioned above from the image of the 2nd image, specifically each frame which comprises a 2nd image is comprised.

  In addition, the process which determines the area | region containing the pixel which corresponds to the 1st image which the 1st image information memorize | stored in the image information storage part 101 as mentioned above in the 2nd area | region image as mentioned above is 2nd Although it may be performed on images of all frames of the image, it may be performed at a predetermined or irregular timing, for example, every few frames or every few seconds. In particular, when the first image is an image of a video conference or the like, it is considered that there is no abrupt movement or the like as described above, and therefore matches the first image indicated by the first image information once determined. This is because even if the region including the pixels is applied to a process of removing an image that matches the first image from the second image of a plurality of consecutive frames, it is considered that there is no significant deterioration in image quality. The above-described processing for determining a region where the images match between the second region image and the first image indicated by the first image information stored in the image information storage unit 101 is described in these methods. It is not limited and other methods may be used. For example, a known technique such as processing for detecting a moving object by comparing images between two frames such as MPEG can be used.

  Here, after the number of pixels of at least one of the first image and the second region image is converted, the difference image information configuration unit 104 matches the first image indicated by the first image information as described above. A comparison between the first image indicated by the first image information and the second region image may be performed to determine the region in the second region image. For example, by interpolating pixels of at least one of the first image and the second region image so that the total number of pixels of the first image and the second region image indicated by the first image information are the same, The number may be increased or decreased. Further, the number of pixels may be increased or decreased by interpolating pixels so that the shapes of the first image and the second region image are the same. For example, by making the total number of pixels and the shape of the first image and the second region image the same, it becomes easy to compare the pixels of the two images.

  Further, the difference image information configuration unit 104 converts at least one color of the first image or the second region image indicated by the first image information stored in the image information storage unit 101, and then, as described above, The first image indicated by the first image information may be compared with the second region image to determine a region in the second region image that matches the first image indicated by the first image information. Good. The color correction includes, for example, color correction, contrast correction, level correction, sharpness correction, and the like. These color corrections are automatically performed using the color information of a part or the whole area of the first image or the second area image indicated by the first image information.

  Usually, when the first image indicated by the first image information is displayed on the display, the color or the like displayed changes depending on the characteristics of the display. Further, when the displayed first image is photographed by the photographing unit 103, the second region in the second image acquired also by the influence of the ambient light on which the display is arranged and the characteristics of the photographing unit 103. The color of the image is different from the color displayed on the display. Therefore, the colors of the second region image and the first image are often very different. If the colors are different as described above, it is considered that the comparison between the first image and the second region image is not normally performed. In contrast, by performing the color correction as described above, it is possible to compare the first image indicated by the first image information with the second region image with high accuracy. For example, by performing color correction on the second region image, it is possible to eliminate a color cast or the like due to the influence of ambient light in the second region image, and to perform highly accurate comparison.

  Further, gray scale conversion may be performed as color correction on the first image and the second region image indicated by the first image information. By making it grayscale, it is possible to compare the images by eliminating the change in the color of the images during shooting. Further, by making the gray scale in this way, the amount of image data is reduced, and the comparison process can be speeded up. Similarly, when detecting a matching region between the first image and the second region image, only the luminance information of the first image and the second region image indicated by the first image information is compared. May be. In this way, by comparing only the luminance information, it is possible to compare the images while eliminating the change in the color of the images at the time of shooting or the like. In addition, the amount of data used for comparison between images can be reduced, and the processing speed can be increased.

  Note that the removal described here is not limited to the complete removal of the pixels, but it is also considered to be the same as the removal as a result. For example, for a pixel to be removed, mask information or transmission information for setting the pixel to be invisible at the time of display can be set, or a pixel to be removed can be identified when displaying an image. In addition, the color information of the pixel may be set to a predetermined value, for example, a value that becomes a specific blue color. In addition, another image or the like may be arranged instead of the removed pixel in the area where the removed pixel is located. The other images are preferably non-significant still images or moving images that do not hinder viewing. In addition, when displaying a difference image, a device or the like that displays a difference image may appropriately synthesize another image as described above into the removed region.

  The difference image information configuration unit 104 can usually be realized by an MPU, a memory, or the like. The processing procedure of the difference image information configuration unit 104 is usually realized by software, and the software is recorded on a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  The output unit 105 outputs the difference image information configured by the difference image information configuration unit 104. The output here is a concept including transmission to another apparatus and accumulation in a recording medium or the like. The output unit 105 preferably outputs the difference image information in a streaming manner when the image processing apparatus 100 is used in a video conference system or the like that performs a conference in real time. There is no limitation on the timing, method, and the like at which the output unit 105 outputs the difference image information. For example, the difference image information may be output every time the difference image information is acquired, or the difference image information may be output when a predetermined amount of difference image information is acquired. The output unit 105 outputs the difference image information output from the difference image information configuration unit 104 to MPEG, ITU-T H.264, or the like. The compression encoding may be performed using a compression encoding algorithm such as H.264.

  Further, the output unit 105 may output the synchronization information added to the difference image information. Further, the output unit 105 may output the difference image information by adding device identification information or the like of the photographing unit 103 or the like. The output unit 105 may or may not include an output device such as a communication device. The output unit 105 can be implemented by output device driver software, or output device driver software and an output device. In addition, when the output unit 105 performs compression encoding processing or the like, the output unit 105 may include an MPU, a memory, or the like for executing these processing. These processing procedures are usually realized by software, and the software is recorded in a recording medium such as a ROM. However, it may be realized by hardware (dedicated circuit).

  Next, an example of the operation of the image processing apparatus will be described using the flowchart of FIG. Here, information for designating an area in which the display unit 102 displays the first image in the second image captured by the imaging unit 103 in advance, for example, coordinate information of the vertex of the area is stored in a memory (not shown) in advance. The case where it is stored will be described. Further, in order to simplify the description, a case will be described as an example where luminance information between pixels is compared to determine whether or not the pixels match.

  (Step S <b> 201) The display unit 102 determines whether it is time to display the first image indicated by the first image information stored in the image information storage unit 101. For example, when the first image is displayed at 30 frames per second, it is determined whether 1/30 seconds have elapsed since the previous frame image was displayed. To do. If it is time to display, the process proceeds to step S202, and if not, the process proceeds to step S204.

(Step S <b> 202) The display unit 102 reads out and acquires one frame of image information corresponding to the display timing among the first image information stored in the image information storage unit 101. For example, image information indicating the image of the next frame of the image of the frame displayed immediately before is acquired.
(Step S203) The display unit 102 displays the image information acquired in step S202 on a display (not shown) or the like.

(Step S204) The photographing unit 103 determines whether or not it is time to photograph the second image. For example, when the second image is captured at 30 frames per second, it is determined whether 1/30 seconds have elapsed since the image of the immediately preceding frame was captured. to decide. If it is time to shoot, the process proceeds to step S205. If it is not time to shoot, the process returns to step S201.
(Step S205) The imaging unit 103 captures a second image of one frame including the first image displayed on the display or the like by the display unit 102. The image receiving unit 107 receives information indicating the captured second image and temporarily stores it in a memory (not shown) or the like.

  (Step S <b> 206) The difference image information configuration unit 104 acquires information for specifying a region in which the display unit 102 in the second image displays the first image, which is stored in advance in a memory or the like (not shown). Using the information, information indicating the second region image is acquired from the information indicating the second image acquired in step S205. Note that this process may be omitted when the second area image matches the second image, that is, when the shooting area of the shooting unit 103 matches the display area of the display unit 102.

  (Step S207) The difference image information configuration unit 104 corrects the luminance information of the second region image. For example, the brightness value that is the brightness information of each pixel in the second area image is automatically level-corrected. Automatic level correction is a process in which the brightest and darkest pixels in an image are defined as the maximum and minimum luminance values, and the luminance values of halftone pixels are evenly redistributed. Information indicating the corrected second region image is temporarily stored in a memory or the like.

(Step S208) The difference image information configuration unit 104 reads out and acquires the first image information from the first image information storage unit.
(Step S209) The difference image information configuration unit 104 converts the first image information acquired in Step S208 into a memory (not shown) so that the first image becomes an image having the same shape and the total number of pixels as the second region image. For example, the display unit 102 in the second image stored in advance corrects the information using information specifying an area for displaying the first image. The corrected first image information is temporarily stored in a memory or the like.

(Step S210) A region that matches the first image in the second region image by comparing the second region image corrected in Step S207 with the first image indicated by the first image information corrected in Step S209. To decide. Note that the processing in step S210 will be described in detail with reference to FIG.
(Step S211) The difference image information configuration unit 104 creates difference image information indicating a difference image, which is an image obtained by removing the image in the region determined in Step S210 from the second image.
(Step S212) The output unit 105 transmits the difference image information configured in step S211. Then, the process returns to step S201.
In the flowchart of FIG. 2, the process is terminated by powering off or a process termination interrupt.

  Next, the process of step S210 of FIG. 2 will be described using the flowchart of FIG. Here, as an example, it is assumed that the first image and the second area image are color images having a luminance value as luminance information, and the luminance value as luminance information between the first image and the second area image is set. A case where the region in the second region image that matches the first image is determined will be described as an example. In addition, here, the first region image and the first image corrected for the number of pixels are each p pixels in the x-axis direction (p is an integer of 1 or more) and q pixels in the y-axis direction (q is 1 or more). The case of an (integer) rectangular image will be described.

(Step S301) The difference image information construction unit 104 substitutes 1 for a counter m.
(Step S302) The difference image information construction unit 104 substitutes 1 for a counter n.
(Step S303) The difference image information configuration unit 104 acquires the luminance value of the pixel located at the coordinates (n, m) of the second region image. The coordinates (n, m) are coordinates in units of pixels in the image. Here, as an example, the upper left coordinates are (0, 0) and the lower right coordinates are (p, q). .

(Step S304) The difference image information configuration unit 104 acquires the luminance value of the pixel located at the coordinates (n, m) of the first image indicated by the first image information stored in the image information storage unit 101.
(Step S305) The difference image information configuration unit 104 calculates the absolute value of the difference between the luminance value acquired in step S303 and the luminance value acquired in step S304.
(Step S306) The difference image information configuration unit 104 determines whether or not the absolute value of the difference calculated in step S305 is equal to or less than a predetermined threshold value set in advance. If it is less than or equal to the threshold value, the process proceeds to step S307, and if it is greater than the threshold value, the process proceeds to step S308.

(Step S307) The difference image information configuration unit 104 selects (n, m) pixels of the second region image as (n, m) of the first image indicated by the first image information stored in the image information storage unit 101. It is determined that the pixel matches the pixel m). Then, the determination result is temporarily stored in a memory or the like, and the process proceeds to step S309.
(Step S <b> 308) The difference image information configuration unit 104 sets the (n, m) pixels of the second region image to (n, m) of the first image indicated by the first image information stored in the image information storage unit 101. It is determined that the pixel does not match the pixel of m). Then, the determination result is temporarily stored in a memory or the like, and the process proceeds to step S309.

(Step S309) The difference image information configuration unit 104 increments the value of the counter n by 1.
(Step S310) The difference image information configuration unit 104 determines whether or not the value of the counter n is greater than p. If larger than p, the process proceeds to step S311. If not larger, the process returns to step S303.
(Step S311) The difference image information configuration unit 104 increments the value of the counter m by 1.
(Step S312) The difference image information configuration unit 104 determines whether or not the value of the counter m is greater than q. If larger than q, the process proceeds to step S311, and if not larger, the process returns to step S302.

  (Step S313) The difference image information configuration unit 104 determines a region that matches the first image in the second region image, using the determination result in step S307. For example, among the pixels determined to match in step S307, a region that is continuously adjacent is determined, and the coordinate information and the like of the pixel that is the edge of the region is stored in the second region image. You may accumulate | store in memory etc. as information which defines the area | region which corresponds to the 1st image which 1st image information shows. In addition, the coordinate information of all the pixels determined to match in step S307 may be stored in a memory or the like as information defining a region that matches the first image in the second region image. Further, when determining the region, the region including the pixel determined to be matched in step S307 may be corrected using the pixel determined not to match in step S308. Then, the process returns to the upper function.

  In the flowcharts of FIGS. 2 and 3 described above, luminance information between pixels is compared to determine whether or not the pixels of the second region image match the pixels of the first image indicated by the first image information. However, the information on other colors between pixels, for example, RGB channel values may be used to determine whether or not the pixels match each other. As a specific example, a difference in values for each channel of RGB is obtained, and when the absolute values of the differences are all equal to or less than a threshold value set for each channel, it is determined that the pixels match each other. It may be. A known technique can be used for such comparison between pixels. When the RGB channel values are used, the color correction for the second region image is performed for the RGB channel values.

Next, an embodiment of a video conference system using the image processing apparatus will be described.
FIG. 4 is a block diagram of the video conference system according to the present embodiment. The video conference system includes a first video conference terminal 20 and a second video conference terminal 30. The first video conference terminal 20 and the second video conference terminal 30 are communicably connected via, for example, a network or a communication line. The first video conference terminal 20 and the second video conference terminal 30 are installed in each conference room so that, for example, a two-way dialogue can be performed between two separate conference rooms.

Here, as an example, the first video conference terminal 20 is assumed to include six image processing apparatuses substantially similar to the image processing apparatus 100 according to the present invention. Here, it is assumed that the six image processing apparatuses are image processing apparatuses 200a to 200f, respectively, and all have the same configuration. However, the number of image processing apparatuses is not limited to six, and may be one or more.
The image processing apparatuses 200a to 200f include a first image information storage unit 201, a first display unit 202, a first imaging unit 203, a first image reception unit 207, and a first difference image information configuration unit 204, respectively. The first output unit 205 and the first receiving unit 206 are provided.

The first receiving unit 206 receives first image information transmitted from the second video conference terminal 30 via a communication line or the like. The first image information received by the first receiving unit 206 is temporarily stored in the first image information storage unit 201. The first receiving unit 206 can be realized by a device driver of communication means, control software of a device that reads information from a recording medium, or the like.
The first image information storage unit 201, the first display unit 202, the first imaging unit 203, the first image reception unit 207, the first difference image information configuration unit 204, and the first output unit 205 have been described with reference to FIG. Since the image information storage unit 101, the display unit 102, the photographing unit 103, the image receiving unit 107, the difference image information configuration unit 104, and the output unit 105 have the same configuration, detailed description thereof is omitted.
As described above, each of the internal configurations of the image processing apparatuses 200a to 200f has the same configuration as that of the image processing apparatus 100 shown in FIG. 1 except that the first receiving unit 206 is newly provided. Yes.

Here, as an example, the second video conference terminal 30 is assumed to include six image processing apparatuses substantially the same as the image processing apparatus 100. Here, it is assumed that the six image processing apparatuses are image processing apparatuses 300a to 300f, respectively, and all have the same configuration. However, the number of image processing apparatuses is not limited to six, and may be one or more.
The image processing apparatuses 300a to 300f include a second image information storage unit 301, a second display unit 302, a second imaging unit 303, a second image reception unit 307, and a second difference image information configuration unit 304, respectively. , A second output unit 305 and a second receiving unit 306.

The second receiving unit 306 receives the difference image information transmitted from the first video conference terminal 20 via a communication line or the like. The difference image information received by the second receiving unit 306 is temporarily stored in the second image information storage unit 301. The second receiving unit 306 can be realized by a device driver of communication means, control software of a device that reads information from a recording medium, or the like.
The second image information storage unit 301, the second display unit 302, the second imaging unit 303, the second image reception unit 307, the second difference image information configuration unit 304, and the second output unit 305 are described with reference to FIG. Since the image information storage unit 101, the display unit 102, the photographing unit 103, the image reception unit 107, the difference image information configuration unit 104, and the output unit 105 have the same configuration, detailed description thereof is omitted.

However, in the image processing apparatuses 300a to 300f, the first image and the second image are respectively read as the difference image and the third image indicated by the difference image information received by the second reception unit 306. To do. Furthermore, the difference image information output by the second difference image information configuration unit 304 is output from the second output unit 305 to the second video conference terminal 20 as the first image information.
As described above, the internal configurations of the image processing apparatuses 300a to 300f have the same configuration as that of the image processing apparatus 100 illustrated in FIG. 1 except that the second receiving unit 306 is newly provided. Yes.

FIG. 5 is a diagram showing a specific configuration of the video conference system according to the present embodiment. The video conference system includes a first video conference terminal 20 and a second video conference terminal 30. Here, the conference room in which the first video conference terminal 20 is arranged is referred to as the first conference room 2, and the conference room in which the second video conference terminal 30 is arranged is referred to as the second conference room 3.
FIG. 6 is a schematic diagram of the first video conference terminal 20 in the first conference room 2 in the video conference system according to this example.

  The first video conference terminal 20 includes six first display units 202, six first imaging units 203, and one information processing device 2000. Here, each first display unit 202 corresponding to each of the image processing devices 200a to 200f is referred to as a first display unit 202a to 202f, and each first imaging unit 203 corresponding to each of the image processing devices 200a to 200f is referred to as the first display unit 202a to 202f. The imaging units 203a to 203f are assumed. The first display units 202a to 202f are assumed to be displays, and the first photographing units 203a to 203f are assumed to be cameras. As an example, it is assumed that an image captured by this camera is a color image. Further, it is assumed that this color image has luminance information, for example, luminance channel information.

  The information processing apparatus 2000 includes a first image information storage unit 201, a first display unit 202, a first imaging unit 203, a first image reception unit 207, and a first image processing unit 200a to 200f. It is assumed that the one-difference image information configuration unit 204, the first output unit 205, and the first reception unit 206 are all provided. Note that, in FIG. 5, illustrations of connections between the first display units 202a to 202f, the first imaging units 203a to 203f, and the information processing apparatus 2000 are omitted.

  In the first conference room 2 in which the first video conference terminal 20 is installed, for example, a circular table 501 is installed. Six circular conference members 501 surround the circular table 501 and seat six or more conference participants. Seats 502a to 502f are arranged. Participants in the one or more conferences are subjects in any of the imaging units 203a to 202f.

  Each of the first display sections 202a to 202f of the first video conference terminal 20 is a virtual substantially regular octagon 503 having six sides facing the front of each of the seats 502a to 502f, with the circular table 501 being the center. They are placed almost on the center of each side. The first display portions 202a to 202f are arranged toward the substantially central direction of the region 503 so that the image display surface is parallel to each of the six sides of the virtual octagonal region 503.

  The first photographing units 203a to 203f of the first video conference terminal 20 are respectively arranged on the substantially center of the six sides of the above-described substantially regular octagonal region 503 so as to face the seats 502a to 502f. ing. Here, as an example, each of the first photographing units 203a to 203f is arranged above each of the display units 202a to 202f, and the photographing direction, specifically the halo of the camera, is respectively in front of the six seats 502a to 502f. Each is arranged to face. Usually, the front of the seat is in the same direction as the front of the person seated in the seat.

  The second video conference terminal 30 includes six second display units 302, six second imaging units 303, and one information processing device 3000. Here, the second display units 302 corresponding to the image processing devices 300a to 300f are referred to as second display units 302a to 302f, and the second imaging units 303 respectively corresponding to the image processing devices 300a to 300f are set to the second display units 302a to 302f. Assume that the imaging units 303a to 303f. Here, the second display units 302a to 302f are assumed to be displays, and the second photographing units 303a to 303f are assumed to be cameras. Assume that an image captured by this camera is a color image, for example. Further, it is assumed that this color image has luminance information, for example, luminance channel information.

The information processing device 3000 includes a second image information storage unit 301, a second display unit 302, a second imaging unit 303, a second image reception unit 307, a second image information storage unit 301 corresponding to each of the image processing devices 300a to 300f. It is assumed that the difference image information configuration unit 304, the second output unit 305, and the second reception unit 306 are all provided. Note that, in FIG. 5, illustrations of connections between the second display units 302a to 302f, the second photographing units 303a to 303f, and the information processing device 3000 are omitted.
The information processing device 2000 and the information processing device 3000 are connected via a network or the like, and perform communication between the first video conference terminal 20 and the second video conference terminal 30.

  In the second conference room 2 in which the second video conference terminal 30 is installed, for example, a circular table 601 is installed. Six round conference tables 601 surround the circular table 601 and seat six or more conference participants. Seats 602a to 602f are arranged. Participants in the one or more conferences are subjects in any of the imaging units 303a to 302f.

  Each of the second display units 302a to 302f of the second video conference terminal 30 has a virtual substantially regular octagon 603 having six sides opposed to the front of the seats 602a to 602f, with the circular table 601 as the center. They are placed almost on the center of each side. The second display units 302 a to 302 f are arranged toward the substantially central direction of the region 603 so that the image display surface is parallel to each of the six sides of the virtual octagonal region 603.

  Each of the second photographing units 303a to 303f of the second video conference terminal 30 is disposed on the substantially center of the six sides of the above-described substantially regular octagonal region 603 so as to face the seats 502a to 502f. ing. Here, as an example, each of the second photographing units 303a to 303f is arranged above each of the display units 302a to 302f, and the photographing direction, specifically, the halo of the camera, is respectively in front of the six seats 502a to 502f. Each is arranged to face. Usually, the front of the seat is in the same direction as the front of the person seated in the seat.

  Images captured by the first imaging units 203a to 203f are set to be displayed on any one of the second display units 302a to 302f via the information processing device 2000 and the information processing device 3000. Images captured by the second imaging units 303a to 303f are set to be displayed on any one of the first display units 202a to 202f via the information processing device 3000 and the information processing device 2000. It shall be. For example, information that designates one of the image processing apparatuses 300a to 300f as a destination of image information in advance, for example, address information on the network such as an IP address, is stored in the memory of each of the image processing apparatuses 200a to 200f. In addition, when the first output unit 205 of each of the image processing apparatuses 200a to 200f outputs the image information, the information specifying the transmission destination is added to the image information and transmitted. The image information may be transmitted to the image processing apparatuses 300a to 300f.

  Similarly, information that designates one of the image processing devices 200a to 200f as the transmission destination of the image information is stored in advance in the memory or the like of each of the image processing devices 300a to 300f. When the second output unit 305 outputs image information, the image information is transmitted to the predetermined image processing apparatuses 200a to 200f by adding the information specifying the transmission destination to the image information and transmitting the image information. You just have to do it.

  Here, in particular, it is assumed that the first display portion 202b and the first display portion 202f are arranged on the sides of the octagonal region 503 that face each other so that the display surfaces face each other. The first imaging unit 203b is disposed at a position facing the first display unit 202b, and the imaging area of the first imaging unit 203b includes the entire image displayed by the first display unit 202b. It is assumed that the orientation and the angle of view of the first photographing unit 203b are set and fixed. The first imaging unit 203f is disposed at a position facing the first display unit 202f, and the imaging area of the first imaging unit 203f includes the entire image displayed by the first display unit 202f. It is assumed that the orientation, the angle of view, and the like of the first photographing unit 203f are set and fixed.

  Similarly, it is assumed that the second display portion 302b and the second display portion 302f are arranged on the sides of the octagonal region 603 that face each other so that the display surfaces face each other. The second imaging unit 303b is disposed at a position facing the second display unit 302b, and the entire image displayed on the second display unit 302b is included in the imaging area of the second imaging unit 303b. It is assumed that the orientation, the angle of view, and the like of the second imaging unit 303b are set and fixed. The second imaging unit 303f is disposed at a position facing the second display unit 302f, and the imaging area of the second imaging unit 303f includes the entire image displayed by the second display unit 302f. It is assumed that the direction and the angle of view of the second imaging unit 303f are set and fixed.

  Further, the image photographed by the first photographing unit 203b is transmitted to the image processing device 300b, displayed on the second display unit 302b, and the image photographed by the second photographing unit 303b is transmitted to the image processing device 200b. It is assumed that it is set in advance to be displayed on the first display unit 202b. Similarly, an image photographed by the first photographing unit 203f is transmitted to the image processing device 300f, displayed on the second display unit 302f, and an image photographed by the second photographing unit 303f is displayed on the image processing device 200f. It is assumed that it is set in advance to be transmitted and displayed on the first display unit 202f.

  7 illustrates a first display unit 202b, a first display unit 202f, a first imaging unit 203b, a first imaging unit 203f, a second display unit 302b, a second display unit 302f, a second imaging unit 303b, and a second imaging unit. It is a schematic diagram for demonstrating the outline of the arrangement | positioning relationship and connection relation of 303f, and the arrangement | positioning relationship between seats 502b, 502f, 602b, and 602f.

  The areas where the first display unit 202b and the first display unit 202f are displayed in the second image captured by the first imaging unit 203b and the first imaging unit 203f are examined in advance, and each first image Information for specifying these areas is set in advance in a memory or the like. This information is information indicating an area in which the first display unit 202b and the first display unit 202f display the first image in the second image captured by the first imaging unit 203b and the first imaging unit 203f. For example, when the second image captured by the first capturing unit 203b is displayed on another display or the like, it is assumed that an image as shown in FIG. 8 is obtained.

  Here, if the display surface 2021b of the first display unit 202b is displayed as a rectangle in the second image, the display surface of the first display unit 202b in the second image is used using this image. For example, in the situation where there is no person in the first conference room 2, the upper left coordinates (x1, y1) of the display surface of the first display unit 202b in the second image The lower right coordinates (x2, y2) are set in a memory or the like. The unit of this coordinate is a pixel unit in the second image. Similarly, the areas where the second display unit 302b and the second display unit 302f are displayed in the second images captured by the second imaging unit 303b and the second imaging unit 303f are examined in advance, and each second Information for specifying these areas in the image is previously set in a memory or the like.

Next, the operation of the video conference system having the above configuration will be described.
First, as shown in FIG. 7, in the first conference room 2, the participant A is in the seat 502 b and the participant B is in the seat 502 f, and in the second conference room 3, the participant C It is assumed that the participant D sits on the seat 602f and starts a video conference.

  First, when the participant C sitting on the seat 602b is photographed by the photographing unit 303b, one frame of image information including the participant C, the display unit 302b, and the display area 3021b is obtained as shown in FIG. The image information is sequentially transmitted from the image processing apparatus 300b to the image processing apparatus 200b, and is received by the first receiving unit 206 of the image processing apparatus 200b. The received image information is temporarily stored as first image information in the first image information storage unit 201, and then the first image is sequentially displayed on the first display unit 202b using the first image information. . It is assumed that the number of pixels of this image is 1920 × 1080 pixels.

  Next, the first imaging unit 203b captures a second image of one frame. As shown in FIG. 10, the second image captured by the first imaging unit 203b includes an image of the first display unit 202b and its display area 2021b, and an image of the upper body of the participant A seated on the seat 502b. Will be included. The number of pixels of the first image is assumed to be 1920 × 1080 pixels. The first image receiving unit 207 of the image processing apparatus 200b receives information on the second image captured by the first imaging unit 203b and temporarily stores it in a memory (not shown) or the like.

  Here, the first image displayed in the display area 2021b and indicated by the first image information stored in the image information storage unit 101 is a participation image captured by the second imaging unit 303b as described above. Since the image of the person C and the display part 302b and the image of the display area 3021b are displayed, the image in the display part 202b of the second image also includes the image of the participant C and the display area 3021b. Therefore, when the second image captured by the first imaging unit 203b is transmitted as it is from the first output unit 205 to the second conference terminal 30 and displayed on the second display unit 302b, the second display unit 302b The image of the participant C who is present in the second conference room 3 is also fed back and displayed. For this reason, it cannot be said that an appropriate image is displayed for a video conference, and there is a risk of hindering smooth progress of the conference.

  For this reason, in the present embodiment, processing for transmitting an image from which such an image to be fed back is removed to another conference room is performed. That is, the first difference image information configuration unit 204 designates an area in which the first display unit 202b displays the first image in the second image captured by the first imaging unit 203b stored in advance in a memory or the like. Information indicated here, (x1, y1), (x2, y2) are read out, the second image capturing unit 303b captures the image, and the region indicated by this information from the second image received by the first image receiving unit 207 A second region image which is an image inside is cut out and acquired. An example of the acquired image is shown in FIG.

  Next, the first display unit 202b in the second image captured by the acquired first imaging unit 203b performs level correction on the luminance value of the image in the area where the first image is displayed. The first difference image information configuration unit 204 reads the first image information of one frame currently displayed on the first display unit 202b stored in the first image information storage unit 201. Then, the number of pixels is corrected so that the number of pixels of the first image indicated by the read first image information is the same as the number of pixels of the second region image described above. For example, if the number of pixels in a rectangular area surrounded by coordinates (x1, y1) and coordinates (x2, y2) as shown in FIG. 8 in the second image is 1400 × 800 pixels, The number of pixels of the first image having 1920 × 1080 pixels is changed so as to be 1400 × 800 pixels, which is the same as the number of pixels of the second region image. The method of interpolating the number of pixels at the time of this change does not matter.

  Next, the first difference image information configuration unit 204 compares the first image indicated by the first image information with the corrected number of pixels with the second region image with the corrected luminance information, An area having a matching image is determined. Specifically, here, the difference between the luminance values, which are the luminance information, between each pixel constituting the second region image and the pixel located at the same coordinate in the first image with the corrected number of pixels, It is sequentially determined for each pixel whether or not the absolute value of the difference is equal to or less than a predetermined threshold value set in advance. Instead of determining whether or not the absolute value of the difference is equal to or less than a predetermined threshold value, it may be determined whether or not the difference is a value within a predetermined range.

  Next, an area composed of pixels that are equal to or less than the threshold is determined. Here, the second region image shown in FIG. 11 is compared with the image obtained by correcting the number of pixels of the first image shown in FIG. As a result of this comparison, a region 1000 indicated by a dotted line in FIG. 12 is determined as a region having an image that matches the first image in the second region image. . Participant A is not included in this area 1000.

  Next, in the second image, all the pixels in the region 1000 are removed and replaced with gray pixels. Thereby, for example, a difference image in which a region having an image matching the first image in the second region image as shown in FIG. 13 is changed to a gray image is obtained. Specifically, the area excluding the image that matches the first image of the second area image is the area where the first display section 202b displays the first image in the second image. Is an image showing the participant A overlapping the first image displayed. Therefore, by replacing with the gray image in this way, as shown in FIG. 13, in the area where the second display unit 202b in the second image displays the first image, Only the image of the participant A is displayed.

  The first output unit 205 transmits the difference image information indicating the difference image obtained in this way to the image processing device 300b of the second conference terminal 30 via the network. The first output unit 205 adds, for example, address information on the network of the image processing apparatus 300b stored in advance, such as an IP address, to the difference image information, and transmits the difference image information. Thereby, the difference image information is received by the second receiving unit 306 of the image processing apparatus 300b specified by the IP address or the like. The difference image information of one frame as shown in FIG. 13 received by the second receiving unit 306 is temporarily stored in the second image information storage unit 301 and then displayed by the second display unit 302b. As a result, the second display unit 302 displays an image of the participant A as shown in FIG.

  The second imaging unit 303b is a third image that includes the difference image displayed on the second display unit 302b and the image of the upper body of the participant C sitting on the seat 602 in front of the second display unit 302b. Shoot. The second image receiving unit 307 receives information indicating the third image output from the second imaging unit 303b and temporarily stores it in a memory or the like. Then, similarly to the first difference image information configuration unit 204 described above, the second difference image information configuration unit 304 captures images from the third image captured by the second imaging unit 303b and received by the second image reception unit 307. The second display unit 302b removes an image that matches the currently displayed difference image, and configures a first image obtained by converting the removed area into a gray image. The first image is an image obtained by replacing the area in which the image displayed by the second display unit 302b in the third image is captured with a gray image. The image includes the second display unit 302 replaced with a gray image.

  Next, as with the first output unit 205, the second output unit 305 sends the first image information indicating the first image of one frame acquired by the second difference image information configuration unit 304 to the first conference terminal 20. To the image processing apparatus 200b. When the first receiving unit 206 of the image processing apparatus 200b receives the first image information, the first image information is temporarily stored in the first image information storage unit 201 and displayed by the first display unit 202b. . As described above, the same processing as described above is repeated. In addition, the same processing as described above is performed between the first display unit 202f, the first imaging unit 203f, the second display unit 302f, and the second imaging unit 303f.

  As described above, according to the present embodiment, the first display unit 203 and the second difference image information configuration unit 304 use the first display unit from the images captured by the first imaging unit 203 and the second imaging unit 303. By removing the image that matches the image displayed on the second display unit 202 and the second display unit 302, the image displayed on the first display unit 202 and the second display unit 302 is changed to the first photographing unit 203. In addition, it can be prevented that the image is captured and output by the second image capturing unit 303.

  Thereby, for example, the first image capturing unit 203 and the second image capturing unit 303 display the images on the first display unit 202 or the second display unit 302 in the background of the participants from the images of the participants such as the video conference. An image obtained by removing the image of the participant in the conference in another conference room can be transmitted to the video conference terminal in the other conference room. As a result, images taken in each conference room can be prevented from being fed back and displayed on a display or the like in the same conference room. Can be done.

In the above specific example, for the sake of explanation, the difference image information indicating the difference image in units of one frame is transmitted and received between the first video conference terminal 20 and the video conference terminal 30. Difference image information indicating an image may be transmitted and received.
In the video conference system of the above specific example, the first image information storage unit 201, the first display unit 202, and the first photographing provided for each of the image processing devices 200 a to 200 f of the first video conference terminal 20. Unit 203, first image reception unit 207, first difference image information configuration unit 204, first output unit 205, and first reception unit 206, respectively, one first image information storage unit 201, The display unit 202, the first photographing unit 203, the first image receiving unit 207, the first difference image information configuring unit 204, the first output unit 205, and the first receiving unit 206 may be realized. .
In the video conference system of the above specific example, a second image information storage unit 301, a second display unit 302, and a second display unit provided for each of the image processing devices 300a to 300f of the second video conference terminal 30 are provided. The imaging unit 303, the second image reception unit 307, the second difference image information configuration unit 304, the second output unit 305, and the second reception unit 306, respectively, one second image information storage unit 301, The second display unit 302, the second imaging unit 303, the second image reception unit 307, the second difference image information configuration unit 304, the second output unit 305, and the second reception unit 306 may be realized. Good.

  In the above embodiment, the case where the image processing apparatus is a stand-alone has been described. However, the image processing apparatus may be a server apparatus in a server / client system. In the case of the server device, the image receiving unit, the display unit, etc. receive input of information indicating the second image or output the image via the communication line. Therefore, for example, in the specific examples shown in FIGS. 4 and 5, the configuration of the information processing apparatus 2000 is provided in the second video conference terminal 30 so that the first video conference terminal 20 passes through the network or the like. Using the second image transmitted in this manner, a difference image may be created in the second video conference terminal 30 and displayed on the second display unit 302 of the second video conference terminal 30.

In each of the above embodiments, each process (each function) may be realized by centralized processing by a single device (system), or by distributed processing by a plurality of devices. May be.
In each of the above embodiments, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

The software that realizes the image processing apparatus in each of the above embodiments is the following program. That is, this program uses a first image information that is information indicating a first image stored in the computer, a display unit that displays the first image, and a first unit that displays the first image. An image receiving unit that receives a second image that is an image obtained by capturing an area including the image of the image, and a second image received by the image receiving unit using the stored first image information. A difference image information constituting unit constituting difference image information which is information indicating a difference image which is an image obtained by removing the first image included in the second image, and difference image information constituted by the difference image information constituting unit. It is a program for functioning as an output unit for outputting.
In the above program, in a transmission step for transmitting information, a reception step for receiving information, etc., processing performed by hardware, for example, processing performed by a modem or an interface card in the transmission step (only performed by hardware). Not included) is not included.

Further, this program may be executed by being downloaded from a server or the like, and a program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, or the like) is read out. May be executed by
Further, the computer that executes this program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.
Further, in each of the above embodiments, it goes without saying that two or more communication means (such as an information transmission unit) existing in one apparatus may be physically realized by one medium.
The present invention is not limited to the above-described embodiments, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, the image processing apparatus according to the present invention is suitable as an image processing apparatus that processes an image of a region including a displayed image, and is useful as an image processing apparatus used in a video conference or the like. is there.

1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. It is a flowchart explaining operation | movement of the image processing apparatus. It is a flowchart explaining operation | movement of the image processing apparatus. It is a block diagram of the video conference system. It is a block diagram of the video conference system. It is a schematic block diagram of the video conference system. It is a typical block diagram of the video conference system. It is a block diagram which shows the example of a display of the video conference system. It is a block diagram which shows the example of a display of the video conference system. It is a block diagram which shows the example of a display of the video conference system. It is a block diagram which shows the example of a display of the video conference system. It is a block diagram which shows the example of a display of the video conference system. It is a block diagram which shows the example of a display of the video conference system.

Explanation of symbols

100, 200, 300 Image processing apparatus 103, 203, 303 Imaging unit 101 Image information storage unit 102, 105, 17 Output unit 102, 12, 202, 2021, 203, 302 Display unit 104 Differential image information configuration unit 107 Image reception unit 200, 200a to 200f, 300, 300a to 300f Image processing device 201 First image information storage unit 202, 202a to 202f First display unit 203, 203a to 203f First imaging unit 204 First difference image information configuration unit 205 First Output unit 206 First reception unit 207 First image reception unit 301 Second image information storage unit 302, 302a to 302f Second display unit 303, 303a to 303f Second imaging unit 304 Second difference image information configuration unit 305 Second output Unit 306 second receiving unit 307 second image receiving unit 502a to 502f 602a~602f seat 2000 and 3000 the information processing apparatus

Claims (12)

An image information storage unit that stores first image information that is information indicating the first image;
A display unit for displaying the first image using the first image information;
An image receiving unit that receives a second image that is an image of an area including the first image displayed by the display unit;
An image obtained by removing the first image included in the second image from the second image received by the image receiving unit using the first image information stored in the image information storage unit. A difference image information constituting unit constituting difference image information which is information indicating a difference image;
An image processing apparatus comprising: an output unit configured to output difference image information configured by the difference image information configuration unit. The difference image information configuration unit includes:
In the second image, the display unit compares the image on the area where the first image is displayed with the first image indicated by the first image information stored in the image information storage unit. Determining a region having an image that matches the first image, and constructing difference image information indicating a difference image obtained by removing an image in the determined region from the second image. Item 6. The image processing apparatus according to Item 1. The difference image information configuration unit includes:
The comparison is performed by converting the number of pixels of at least one of the images on an area where the display unit in the first image or the second image displays the first image. 2. The image processing apparatus according to 2. The difference image information configuration unit includes:
3. The comparison is performed by converting at least one color of an image on an area where the first image is displayed by the display unit in the first image or the second image. The image processing apparatus according to claim 3. A video conference system comprising a first image processing device and a second image processing device,
The first image processing apparatus includes:
A first receiving unit that receives first image information that is information indicating a first image output by the second image processing device;
A first image information storage unit for storing first image information received by the first reception unit;
A first display unit that displays the first image using the first image information;
A first image receiving unit that receives a second image that is an image of a region including the first image displayed by the first display unit;
Using the first image information stored in the first image information storage unit, the first image included in the second image is selected from the second image received by the first image reception unit. A first difference image information constituting unit constituting difference image information which is information indicating a difference image which is a removed image;
A first output unit that outputs the difference image information configured by the first difference image information configuration unit, and the second image processing device includes:
A second receiver that receives the difference image information output by the second image processing device;
A second image information storage unit for storing the difference image information received by the second reception unit;
A second display unit that displays the difference image using the difference image information;
A second image receiving unit that receives a third image that is an image of an area including the difference image displayed by the second display unit;
An image obtained by removing the difference image included in the third image from the third image received by the second image receiving unit using the difference image information stored in the second image information storage unit. A second difference image information constituting unit constituting first image information which is information indicating the first image, and
A video conference system, comprising: a second output unit configured to output first image information configured by the second difference image information configuration unit. The first difference image information configuration unit includes:
The image on the area where the first display unit in the second image displays the first image,
Compared with the first image indicated by the first image information stored in the first image information storage unit, an area having an image that matches the first image is determined, and the second image Configuring the difference image information indicating the difference image from which the image in the determined region has been removed;
The second difference image information configuration unit includes:
Compare the image on the area where the second display unit in the third image displays the difference image with the difference image indicated by the difference image information stored in the second image information storage unit, A region having an image that matches the difference image is determined, and the first image information indicating the first image obtained by removing the image in the determined region from the third image is configured. The video conference system according to claim 5. The first difference image information configuration unit includes:
The display unit in the first image or the second image converts the number of pixels of at least one of the images on the area where the first image is displayed, and performs the comparison,
The second difference image information configuration unit includes:
7. The television according to claim 6, wherein the display unit in the difference image or the third image converts the number of pixels of at least one of the images on an area where the difference image is displayed, and performs the comparison. Conference system. The first difference image information configuration unit includes:
The display unit in the first image or the second image converts the color of at least one of the images on the area where the first image is displayed, and performs the comparison.
The second difference image information configuration unit includes:
8. The comparison is performed by converting at least one color of an image on an area where the difference image is displayed by the display unit in the difference image or the third image. The described video conference system. An image of an image processing apparatus including an image information storage unit that stores first image information that is information indicating a first image, a display unit, an image reception unit, a difference image information configuration unit, and an output unit A processing method,
A display step in which the display unit displays the first image using the first image information;
An image receiving step in which the image receiving unit receives a second image that is an image of an area including the first image displayed in the display step;
The difference image information configuration unit uses the first image information stored in the image information storage unit, and from the second image received in the image reception step, the second image included in the second image. A difference image information constituting step constituting difference image information which is information indicating a difference image which is an image obtained by removing one image;
An image processing method comprising: an output step in which the output unit outputs difference image information configured in the difference image information configuration step. 1st receiving part, 1st display part, 1st image reception part, 1st difference image information composition part, 1st output part, 2nd receiving part, 2nd image information storage part, 2nd display part, 2nd image reception A video conference method to be processed in a video conference system having a first difference image information configuration unit, and a second output unit,
A first receiving step in which the first receiving unit receives first image information which is information indicating a first image output by the second output unit;
A first display step in which the first display unit displays the first image using the first image information;
A first image receiving step in which the first image receiving unit receives a second image that is an image of a region including the first image displayed in the first display step;
The first difference image information configuration unit uses the first image information to remove the first image included in the second image from the second image received in the first image reception step. A first difference image information constituting step constituting difference image information which is information indicating a difference image which is an image;
A first output step in which the first output unit outputs the difference image information configured in the first difference image information configuration step;
A second receiving step in which the second receiving unit receives the difference image information output in the first output step;
A second display step in which the second display unit displays the difference image using the difference image information;
A second image receiving step in which the second image receiving unit receives a third image that is an image of an area including the difference image displayed in the second display step;
The second difference image information configuration unit is an image obtained by removing the difference image included in the third image from the third image received in the second image reception step using the difference image information. A second difference image information constituting step constituting first image information which is information indicating the first image;
The video conference method, wherein the second output unit includes a second output step of outputting the first image information configured in the second difference image information configuration step. Computer
A display unit for displaying the first image using first image information which is information indicating the first image stored;
An image receiving unit that receives a second image that is an image of an area including the first image displayed by the display unit;
Information indicating a differential image that is an image obtained by removing the first image included in the second image from the second image received by the image receiving unit using the stored first image information. A difference image information constituting unit constituting difference image information,
The program for functioning as an output part which outputs the difference image information which the said difference image information structure part comprised. The computer-readable recording medium which recorded the program of Claim 11.

Images